# AI agent incident runbook

Template from https://vericto.com/docs/incident-playbook. Fill in every `____`, keep a copy
where the on-call team can reach it without the dashboard, and review it after every drill
and every incident.

Last reviewed: ____ by ____

## 1. Who decides

| Role | Person | Backup | How to reach them |
| --- | --- | --- | --- |
| Incident commander (decides to block and to unblock) | ____ | ____ | ____ |
| Vericto owner or admin on call (carries out the block) | ____ | ____ | ____ |
| Database owner (rotates credentials) | ____ | ____ | ____ |
| Agent owner (fixes the agent) | ____ | ____ | ____ |
| Security / compliance contact | ____ | ____ | ____ |

- Anyone who sees an agent misbehaving may ask for a block; the incident commander decides.
- Do not wait for a root cause to block: blocking is reversible, unblocking needs the incident commander.
- Escalate to ____ if the incident commander cannot be reached within ____ minutes.

## 2. Agent inventory

| Agent | Vericto API key name | Database user | Databases | Principal (registry) | Owner |
| --- | --- | --- | --- | --- | --- |
| ____ | ____ | ____ | ____ | ____ | ____ |
| ____ | ____ | ____ | ____ | ____ | ____ |

Proxy rule sync interval of each production database with agents: ____ s (recommended: 30 s).

## 3. Block the agent

- [ ] Note the time (T0): ____
- [ ] Dashboard → Incident response → Block an agent. Pick the API key, the database user (one database or every database), the principal, or "Block all agents". Reason: ____ (ticket ____).
- [ ] Confirm the block is listed as active, and read the coverage and warnings (monitor mode, proxies that cannot apply it).
- [ ] If the credential is compromised: revoke the API key (irreversible) and rotate the database user's password: ____
- [ ] Revoke any OIDC trust policy or MCP connection the agent used: ____
- [ ] Tell the agent owner and the stakeholders: ____

## 4. Verify that it took effect

- [ ] The incident panel shows every proxy as synced since the block.
- [ ] First rejected statement on each path, from the panel or the Audit Trail:

| Path | First rejection (time) | Seconds after T0 |
| --- | --- | --- |
| Runtime API / SDKs | ____ | ____ |
| MCP check_sql / CLI | ____ | ____ |
| TCP proxy (database ____) | ____ | ____ |

- [ ] The admin audit log shows `agent.blocked` with the actor, reason, target and time.
- [ ] The alert reached the channels: ____

## 5. Unblock

- [ ] The incident commander approved the unblock: ____ (time ____)
- [ ] Dashboard → Incident response → Unblock, with the reason: ____
- [ ] The agent works again on each path: ____

## 6. Drill log

Pass criterion: API paths under 5 s; TCP proxy within the sync interval plus 10 s.

| Date | Target | T0 | Runtime API | MCP / CLI | TCP proxy | Unblocked at | Pass / fail | Run by |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ____ | ____ | ____ | ____ | ____ | ____ | ____ | ____ | ____ |

## 7. Post-incident review

- [ ] Incident report downloaded from the incident panel (block record, audit entries, propagation, first rejections).
- [ ] Audit Trail exported for the incident window: ____ to ____
- [ ] Admin audit log entries for the block and the unblock saved.
- [ ] CLI and MCP check reports for the agent's key saved.
- [ ] Timeline written: detection ____, decision ____, block ____, first rejection ____, unblock ____
- [ ] Root cause and fix: ____
- [ ] Follow-ups (allowlist, sensitive columns, rules, sync interval, this runbook): ____