Incident playbook for AI agents.

Cut an agent's access within minutes, and write down who decides. How to block, revoke credentials, verify and time the block, unblock, and what to keep for the review.

Overview

When an AI agent does something it should not (a loop of DELETEs, reads outside its task, an injected instruction), the first job is to stop it, not to understand why. Vericto has a per-agent kill switch: with one click, every statement of that agent is refused on every path where Vericto evaluates it, with a message that tells it not to retry and to ask a person.

This page is the playbook for control 10 of the production agent-access checklist: "Kill an agent's access within minutes, and know who decides". At the end there is a runbook template to fill in with your team's names.

Who decides

A named role decides, not "the team". Write it in your runbook before you need it:

  • Incident commander: ____ (role, name and backup). Decides to block and decides to unblock.
  • Who carries out the block in Vericto: the workspace owner or an admin. They are the only roles that can block and unblock; members and viewers see the blocks but cannot change them.
  • Database owner: ____. Rotates the database user's password or disables the role when the credential is compromised.
  • Escalation: if the incident commander cannot be reached within ____ minutes, ____ decides.

Blocking is reversible and deletes nothing, so the working rule is block first, ask later. Unblocking needs the incident commander's decision.

How to block an agent

In the dashboard, open Incident response → Block an agent, pick the target, write the reason (required, for example the ticket number) and confirm. The target is how Vericto recognises the agent on each path:

  • An API key: what the agent uses with the Runtime API, the SDKs, the MCP check_sql and the CLI.
  • A database user, on one database or on every database of the workspace: the agent's sessions through the TCP proxy.
  • A principal from the registry: its API key and its exact database user, on the principal's databases, with one block.
  • Block all agents (the workspace emergency switch): every API key of the workspace and every database user on every TCP proxy. On the proxy it also blocks the people who connect through it: on a connection, Vericto cannot tell an agent from a person.

The block does not change the agent's allowlist or its rules: when you unblock, everything is as it was. Every block is recorded in the audit log and announced on the workspace's channels.

What happens on each path, and when

  • Runtime API, SDKs, MCP check_sql and CLI: immediately. The next call gets BLOCKED with rule VERICTO-105 and a message that names the kill switch and who set it, and tells the agent not to retry and to ask a person. Nothing is evaluated and nothing is charged to your quota.
  • TCP proxy: at its next rule sync. The block arrives in GET /sync/rules as an agent-access allowlist that allows no table for that database user, so it works with the proxies you already run (vericto-proxy 4.16.x and the earlier releases with agent access allowlists), with nothing to upgrade. The statement is refused with SQLSTATE 42501 on PostgreSQL or error 1142 on MySQL, and rule VERICTO-087.

The time to the proxy is its sync interval: 300 seconds by default, configurable down to 30 with VERICTO_RULES_SYNC_INTERVAL_SECS on the proxy or in the database's proxy settings in the dashboard. For production databases with agents we recommend 30 seconds: the block then arrives within 30 seconds.

Proxy limits

  • Statements that read no table (for example SELECT 1, the usual session SETs, BEGIN and COMMIT) still pass: the allowlist governs tables. Any statement that reads or writes a table, and all DDL, is refused.
  • Open sessions are not closed, but their next statement that touches a table is refused.
  • While the workspace is in monitor mode or an observe window, the proxy only flags the blocked agent's statements, without refusing them. The dashboard warns you when you block. The API paths block anyway.
  • A proxy without VERICTO_DATABASE_ID only receives "on every database" blocks and the workspace switch.
  • Vericto does not revoke database credentials: if the incident warrants it, rotate the database user's password (see below).

How to revoke credentials

The block stops the agent in Vericto. If the credential is compromised, revoke it as well:

  • The agent's API key: from the same incident panel or in Dashboard → API keys. It is irreversible and applies immediately on the API paths; the agent will need a new key. It does not stop proxy sessions, which authenticate with database credentials.
  • The database user: rotate its password or disable the role in the database (ALTER ROLE … NOLOGIN on PostgreSQL, ALTER USER … ACCOUNT LOCK on MySQL).
  • Anything else the agent uses: the OIDC trust policy it gets keys from in CI, and any MCP connection involved.

Verify and time it

  • The incident panel shows, for each block, which proxies synced after the block and the first rejected statement on each path, with the seconds since the block.
  • The Audit Trail shows the blocked statements (VERICTO-105 on the API paths, VERICTO-087 on the proxy), and each event's detail names the kill switch that refused it.
  • The audit log (Settings → Audit log) records agent.blocked and agent.unblocked with the actor, the reason, the target and the time.
  • Alerts reach the dashboard bell, the owner's and admins' email and the workspace's Slack, Teams and webhook channels (see alerts).

Run a drill

A playbook nobody tested is a playbook nobody knows works. Run a drill at least every quarter, and whenever you change how your agents connect:

  1. Pick a test agent: an API key and a database user that production does not use.
  2. Run a statement in a loop on each path (the Runtime API or an SDK, check_sql or the CLI, and a connection through the proxy), one per second.
  3. Note the time (T0) and block the agent with the reason "drill".
  4. Note the first rejected statement on each path. The incident panel shows it with the seconds since T0.
  5. Unblock, check that the agent works again and record the result in the runbook's drill log.

A reasonable pass criterion: under 5 seconds on the API paths and, on the proxy, no more than its sync interval plus 10 seconds.

How to unblock

With the incident commander's decision, the owner or an admin opens the block in Incident response and picks Unblock, with a required reason. It applies immediately on the API paths and at the next sync on the proxies. The unblock is recorded in the audit log and announced on the same channels as the block. A revoked API key does not come back: the agent needs a new one.

What to export for the review

  • The incident report, downloaded from the panel: the block, its audit entries, the propagation to the proxies and the first rejections per path.
  • The Audit Trail export for the incident window.
  • The audit log entries for the block and the unblock.
  • The check reports of the CLI and the MCP for the agent's key.
  • The filled-in runbook, with the timeline: detection, decision, block, first rejection and unblock.

Runbook template

Download the Markdown template, fill it in with your team's names and keep it where on-call can read it without the dashboard: template in English · template in Spanish. You can also copy it from here:

Template in English 
# AI agent incident runbook

Template from https://vericto.com/docs/incident-playbook. Fill in every `____`, keep a copy
where the on-call team can reach it without the dashboard, and review it after every drill
and every incident.

Last reviewed: ____ by ____

## 1. Who decides

| Role | Person | Backup | How to reach them |
| --- | --- | --- | --- |
| Incident commander (decides to block and to unblock) | ____ | ____ | ____ |
| Vericto owner or admin on call (carries out the block) | ____ | ____ | ____ |
| Database owner (rotates credentials) | ____ | ____ | ____ |
| Agent owner (fixes the agent) | ____ | ____ | ____ |
| Security / compliance contact | ____ | ____ | ____ |

- Anyone who sees an agent misbehaving may ask for a block; the incident commander decides.
- Do not wait for a root cause to block: blocking is reversible, unblocking needs the incident commander.
- Escalate to ____ if the incident commander cannot be reached within ____ minutes.

## 2. Agent inventory

| Agent | Vericto API key name | Database user | Databases | Principal (registry) | Owner |
| --- | --- | --- | --- | --- | --- |
| ____ | ____ | ____ | ____ | ____ | ____ |
| ____ | ____ | ____ | ____ | ____ | ____ |

Proxy rule sync interval of each production database with agents: ____ s (recommended: 30 s).

## 3. Block the agent

- [ ] Note the time (T0): ____
- [ ] Dashboard → Incident response → Block an agent. Pick the API key, the database user (one database or every database), the principal, or "Block all agents". Reason: ____ (ticket ____).
- [ ] Confirm the block is listed as active, and read the coverage and warnings (monitor mode, proxies that cannot apply it).
- [ ] If the credential is compromised: revoke the API key (irreversible) and rotate the database user's password: ____
- [ ] Revoke any OIDC trust policy or MCP connection the agent used: ____
- [ ] Tell the agent owner and the stakeholders: ____

## 4. Verify that it took effect

- [ ] The incident panel shows every proxy as synced since the block.
- [ ] First rejected statement on each path, from the panel or the Audit Trail:

| Path | First rejection (time) | Seconds after T0 |
| --- | --- | --- |
| Runtime API / SDKs | ____ | ____ |
| MCP check_sql / CLI | ____ | ____ |
| TCP proxy (database ____) | ____ | ____ |

- [ ] The admin audit log shows `agent.blocked` with the actor, reason, target and time.
- [ ] The alert reached the channels: ____

## 5. Unblock

- [ ] The incident commander approved the unblock: ____ (time ____)
- [ ] Dashboard → Incident response → Unblock, with the reason: ____
- [ ] The agent works again on each path: ____

## 6. Drill log

Pass criterion: API paths under 5 s; TCP proxy within the sync interval plus 10 s.

| Date | Target | T0 | Runtime API | MCP / CLI | TCP proxy | Unblocked at | Pass / fail | Run by |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ____ | ____ | ____ | ____ | ____ | ____ | ____ | ____ | ____ |

## 7. Post-incident review

- [ ] Incident report downloaded from the incident panel (block record, audit entries, propagation, first rejections).
- [ ] Audit Trail exported for the incident window: ____ to ____
- [ ] Admin audit log entries for the block and the unblock saved.
- [ ] CLI and MCP check reports for the agent's key saved.
- [ ] Timeline written: detection ____, decision ____, block ____, first rejection ____, unblock ____
- [ ] Root cause and fix: ____
- [ ] Follow-ups (allowlist, sensitive columns, rules, sync interval, this runbook): ____
Template in Spanish 
# Runbook de incidentes con agentes de IA

Plantilla de https://vericto.com/es/docs/incident-playbook. Completa cada `____`, guarda una
copia donde el equipo de guardia pueda leerla sin el dashboard y revísala después de cada
simulacro y de cada incidente.

Última revisión: ____ por ____

## 1. Quién decide

| Rol | Persona | Suplente | Cómo contactarla |
| --- | --- | --- | --- |
| Responsable del incidente (decide bloquear y desbloquear) | ____ | ____ | ____ |
| Owner o admin de Vericto de guardia (ejecuta el bloqueo) | ____ | ____ | ____ |
| Responsable de la base de datos (rota credenciales) | ____ | ____ | ____ |
| Responsable del agente (corrige el agente) | ____ | ____ | ____ |
| Contacto de seguridad / compliance | ____ | ____ | ____ |

- Cualquiera que vea a un agente comportarse mal puede pedir un bloqueo; decide el responsable del incidente.
- No esperes a conocer la causa para bloquear: bloquear es reversible, y desbloquear necesita al responsable del incidente.
- Escala a ____ si no se puede contactar al responsable del incidente en ____ minutos.

## 2. Inventario de agentes

| Agente | Nombre de la API key en Vericto | Usuario de base de datos | Bases de datos | Principal (registro) | Responsable |
| --- | --- | --- | --- | --- | --- |
| ____ | ____ | ____ | ____ | ____ | ____ |
| ____ | ____ | ____ | ____ | ____ | ____ |

Intervalo de sincronización de reglas del proxy de cada base de producción con agentes: ____ s (recomendado: 30 s).

## 3. Bloquear al agente

- [ ] Anota la hora (T0): ____
- [ ] Dashboard → Respuesta a incidentes → Bloquear un agente. Elige la API key, el usuario de base de datos (en una base o en todas), el principal o "Bloquear todos los agentes". Motivo: ____ (ticket ____).
- [ ] Confirma que el bloqueo figura como activo y lee la cobertura y los avisos (modo monitor, proxies que no pueden aplicarlo).
- [ ] Si la credencial está comprometida: revoca la API key (irreversible) y rota la contraseña del usuario de base de datos: ____
- [ ] Revoca toda política de confianza OIDC o conexión MCP que usara el agente: ____
- [ ] Avisa al responsable del agente y a los interesados: ____

## 4. Verificar que se aplicó

- [ ] El panel de incidentes muestra cada proxy como sincronizado después del bloqueo.
- [ ] Primera sentencia rechazada en cada vía, según el panel o el Audit Trail:

| Vía | Primer rechazo (hora) | Segundos desde T0 |
| --- | --- | --- |
| API de runtime / SDK | ____ | ____ |
| MCP check_sql / CLI | ____ | ____ |
| Proxy TCP (base ____) | ____ | ____ |

- [ ] El registro de auditoría muestra `agent.blocked` con el actor, el motivo, el objetivo y la hora.
- [ ] La alerta llegó a los canales: ____

## 5. Desbloquear

- [ ] El responsable del incidente aprobó el desbloqueo: ____ (hora ____)
- [ ] Dashboard → Respuesta a incidentes → Desbloquear, con el motivo: ____
- [ ] El agente vuelve a funcionar en cada vía: ____

## 6. Registro de simulacros

Criterio de aprobación: vías de API en menos de 5 s; proxy TCP dentro del intervalo de sincronización más 10 s.

| Fecha | Objetivo | T0 | API de runtime | MCP / CLI | Proxy TCP | Desbloqueado a las | Aprobado / no | Lo ejecutó |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ____ | ____ | ____ | ____ | ____ | ____ | ____ | ____ | ____ |

## 7. Revisión posterior al incidente

- [ ] Reporte del incidente descargado del panel (bloqueo, entradas de auditoría, propagación, primeros rechazos).
- [ ] Audit Trail exportado para la ventana del incidente: ____ a ____
- [ ] Entradas del registro de auditoría del bloqueo y del desbloqueo guardadas.
- [ ] Reportes de checks de la CLI y del MCP de la key del agente guardados.
- [ ] Cronología escrita: detección ____, decisión ____, bloqueo ____, primer rechazo ____, desbloqueo ____
- [ ] Causa raíz y corrección: ____
- [ ] Pendientes (allowlist, columnas sensibles, reglas, intervalo de sincronización, este runbook): ____