AI Agent Incident Response Runbook: Detect, Contain, Recover, and Learn
An AI agent incident response runbook must handle more than a bad answer. It must stop ongoing actions, revoke delegated authority, preserve the context that shaped the run, trace every side effect, restore trustworthy state, and prove that the agent is safe before it returns to production.
AI Agent Incident Response Runbook: The Quick Answer
An AI agent incident is any event in which an agent, its tools, its identity, its context, or its supporting components cause or could cause unauthorized, unsafe, deceptive, destructive, or materially incorrect action. The trigger may be malicious prompt injection, a compromised connector, excessive permissions, a hallucinated plan, poisoned memory, a bad deployment, a stale policy, or an ordinary software defect. The response should be based on impact and blast radius, not on whether the model “meant” to do it.
The operational sequence is: declare, stop, preserve, contain, scope, correct, verify, restore, and learn. First, establish incident command and halt new agent actions. Next, preserve traces, prompts, retrieved context, memory state, tool requests, approvals, credentials, model and policy versions, and downstream system records. Then contain the incident by revoking authority and isolating affected integrations without deleting evidence. Only after the team understands the blast radius should it reverse harmful side effects, repair state, validate controls in a safe environment, and gradually restore service.
This runbook is a focused companion to the broader AI agent infrastructure stack. The pillar explains the layers enterprises need; this article turns the lifecycle and incident-operations layer into a procedure that engineering, security, SRE, product, legal, and business owners can rehearse before a real failure occurs.
Why AI Agent Incident Response Is Different
Traditional incident response already provides the backbone: preparation, detection, response, recovery, communication, and improvement. NIST SP 800-61 Rev. 3 treats incident response as part of organization-wide cybersecurity risk management, not as a late-stage security task. That foundation still applies. AI agents add several complications that change what responders must capture and what “contained” actually means.
An agent may use email, code, databases, browsers, ticketing, cloud consoles, payment systems, or MCP servers. Stopping the model endpoint does not automatically revoke tokens, cancel queued jobs, or undo actions already accepted downstream.
The decisive cause may live in a retrieved document, hidden web instruction, memory item, tool result, system prompt, or approval message. A normal application log can show the API call while missing the information that shaped the agent’s choice.
A replay with the same visible prompt may not reproduce the same chain. Responders need the exact model version, parameters, tool schemas, context order, policy decisions, and environment state rather than relying on a fresh rerun.
Agent incidents also cross organizational boundaries quickly. A wrong answer is usually a quality event. A wrong answer that triggers a refund, sends an email, changes source code, modifies access, or publishes data becomes an operational or security incident. If the agent acts on behalf of a person, responders must distinguish the human requester, the agent identity, the service identity, the credential used, the policy that allowed the action, and the downstream system that accepted it.
The OWASP Top 10 for Agentic Applications makes this shift concrete. Risks such as goal hijack, tool misuse, identity and privilege abuse, agentic supply-chain compromise, unexpected code execution, and rogue behavior concern what an agent can do, not only what it can say. An effective response plan therefore follows the action path across the whole system.
Four questions define the incident
- What authority was exercised? Identify every token, role, connector, delegated permission, approval exception, and write-capable tool involved.
- What state changed? List messages sent, records modified, files created or deleted, deployments triggered, money moved, access changed, memory written, and queues populated.
- What influenced the decision? Preserve prompts, retrieved chunks, memory, tool responses, peer-agent messages, policies, and human instructions in exact order.
- What can still happen? Find scheduled tasks, retries, callbacks, queued tool calls, active sessions, copied credentials, and other agents that may consume contaminated memory or data.
Classify Severity by Impact, Authority, and Persistence
A severity label should accelerate decisions, not create debate. Do not downgrade an incident because the initiating prompt looked harmless or because no attacker has been confirmed. Severity should reflect actual or plausible harm, the authority the agent possessed, the number of systems or people affected, the persistence of the change, and the team’s ability to reverse it.
| Level | Typical condition | Examples | Immediate posture |
|---|---|---|---|
| SEV-1 Critical | Active material harm, privileged compromise, regulated-data exposure, destructive action, external spread, or loss of control. | Agent changes production access, leaks secrets, sends harmful external instructions, deploys malicious code, moves funds, or continues acting after stop commands. | Disable execution paths, revoke credentials, activate security and executive incident command, preserve evidence, notify required stakeholders. |
| SEV-2 High | Serious unauthorized action or broad blast radius, but containment is available and catastrophic harm is not confirmed. | Incorrect bulk updates, unauthorized customer communication, compromised tool, poisoned shared memory, or repeated high-risk policy bypass. | Pause affected agents and queues, isolate tools and memory, revoke scoped tokens, begin rapid impact analysis. |
| SEV-3 Moderate | Limited incorrect action, bounded exposure, or control failure with low immediate impact. | One incorrect ticket update, failed approval enforcement in staging, or a read-only agent accessing an unnecessary document. | Stop the affected workflow, preserve the run, correct state, test the control, and track follow-up work. |
| SEV-4 Low | No external side effect and no sensitive exposure; problem is caught before action. | Evaluation detects unsafe tool selection, simulated prompt injection succeeds, or an agent proposes but cannot execute a disallowed action. | Record as a near miss, fix and test before release, update evaluation cases and threat models. |
Use the highest applicable level. A small number of actions can still be critical if they affect privileged access, safety, finance, protected data, public communications, or irreversible state. Conversely, a noisy failure with thousands of blocked attempts may remain moderate if deterministic controls prevented side effects and evidence confirms that no authority was exercised.
The First 15 Minutes: Stop Action Without Destroying Evidence
The first response objective is not to discover the elegant root cause. It is to stop additional harm while keeping the evidence needed to understand and reverse what happened. Teams often lose time by restarting the agent, editing prompts in place, clearing memory, rotating everything indiscriminately, or deleting suspicious data. Those actions may change the evidence and make the final blast radius unknowable.
- Declare the incident and assign one incident commander. Open a dedicated channel and timestamp every decision. Name technical, security, product, and communications leads as needed.
- Freeze new agent execution. Pause schedulers, queues, webhooks, background runs, and automatic retries for the affected agent or workflow. If scope is uncertain, fail closed for high-risk actions.
- Block write-capable tools first. Disable send, publish, delete, deploy, transfer, grant-access, and mutation operations. Preserve read-only investigation paths only if they cannot extend harm.
- Revoke or suspend delegated credentials. Target the affected agent identity, session, token, connector, and downstream grants. Do not assume disabling a user interface cancels active tokens.
- Snapshot evidence. Export immutable copies of traces, messages, retrieved context, memory, tool arguments and results, policy decisions, approvals, model configuration, registry metadata, and downstream audit logs.
- Mark contaminated state. Quarantine memory, retrieval indexes, generated artifacts, caches, and peer-agent messages that may carry malicious or incorrect instructions.
- Identify ongoing side effects. Search for scheduled jobs, pending approvals, delayed callbacks, outbound messages, open browser sessions, code pipelines, and transactions initiated by the run.
- Protect people and customers. If the agent produced unsafe instructions, exposed data, or sent unauthorized communication, start the appropriate safety, privacy, legal, and customer response in parallel.
The emergency stop mechanism should have been designed before the incident. It needs to work independently of the agent’s planning loop, model provider, and prompt state. The practical patterns in the agentic AI stop conditions guide are especially useful here: hard step limits, time limits, cost limits, repeated-failure detection, risk thresholds, and escalation rules reduce the distance between detection and containment.
The response path is sequential, but communication, evidence preservation, and impact protection continue throughout the incident.
The Complete AI Agent Incident Response Runbook
Phase 0: Prepare before an incident
A runbook is credible only if the team can execute it. Every production agent should have a named business owner, technical owner, on-call contact, data classification, tool list, identity, permission scope, approved autonomy level, emergency stop method, evidence location, retention period, and recovery procedure. Record those fields in an operational registry; the AI agent registry checklist provides a practical structure.
Preparation also means making important actions reversible. Use staged deletion, delayed external sends, idempotency keys, transaction boundaries, approval gates, versioned artifacts, backups, and canary releases. Logs must connect one run identifier across model calls, retrieved documents, memory reads and writes, policy checks, human approvals, tool calls, downstream responses, and final business outcomes. Run tabletop exercises at least as often as the agent’s authority or architecture changes.
Phase 1: Detect and declare
Detection can come from policy denials, anomaly rules, tool-call patterns, user reports, downstream audit logs, evaluation failures, cost spikes, data-loss alerts, or human observation. Treat these signals as evidence to correlate, not as isolated dashboards. A sudden rise in tool errors may be ordinary reliability trouble; the same rise combined with unusual resource access and repeated retries may indicate goal hijack, privilege abuse, or a compromised connector.
Declare an incident when a credible signal suggests unauthorized action, material incorrect action, control bypass, sensitive exposure, loss of agent control, or contamination that could spread. Create a stable incident ID and attach every artifact to it. Record the initial reporter, time detected, suspected start time, affected agent IDs, owners, environments, current actions, and provisional severity. Avoid causal language until evidence supports it.
Phase 2: Contain authority and execution
Containment should move from the narrowest reliable control to broader shutdown only when necessary. Pause the affected run, then its queue, then related workflows. Disable high-risk tools before removing all observability. Revoke the agent’s credential and active sessions; if a shared credential was used, replace it and investigate every consumer. Block compromised connectors or MCP servers at the gateway. Quarantine contaminated memory and retrieval sources. If peer agents received instructions or artifacts from the affected agent, treat them as exposed until verified.
Containment is incomplete until downstream systems confirm it. A revoked token does not retract an email, cancel a cloud job, stop a deployment already accepted, or reverse a database write. Query each connected system for actions associated with the incident’s identity, time window, run ID, request ID, or idempotency key. Where the action trail is missing, expand the uncertainty boundary rather than assuming safety.
Phase 3: Scope impact and preserve evidence
Build a chronological action ledger. Start before the first suspicious model call because the trigger may have entered through earlier memory, retrieval, or a tool result. Continue after the visible incident because retries, delayed jobs, and other agents may act later. Each ledger row should answer: who or what initiated the action, what context was available, which policy applied, which tool was called, what resource changed, whether a human approved it, what the downstream system returned, and whether the effect is reversible.
CISA’s AI Cybersecurity Collaboration Playbook highlights the value of recording affected model information, lifecycle phase, platforms, APIs, libraries, access, impacted users, categories of harm, and external systems. Adapt those fields to your internal evidence manifest even when an event does not require external reporting.
Phase 4: Eradicate the cause, not only the symptom
Root cause may sit in any layer: malicious input, prompt construction, retrieval, memory, model behavior, tool schema, authorization, orchestration, approval logic, connector implementation, deployment configuration, or human operating procedure. Use a fault tree rather than choosing “the model hallucinated” as a catch-all explanation. Ask why the unsafe plan was possible, why the action was permitted, why controls did not block it, why monitoring did not detect it earlier, and why the blast radius was as large as it was.
Eradication can require removing poisoned content, patching a connector, narrowing permissions, rotating secrets, changing tool descriptions, adding deterministic validation, repairing approval binding, updating stop conditions, or retraining operators. A prompt edit alone is rarely sufficient for a high-impact incident. Prompts are probabilistic guidance; containment for consequential actions should rely on enforceable policy, scoped identity, approval, sandboxing, and downstream controls.
Phase 5: Recover through controlled re-entry
Restore from a known-good state. Repair or reverse downstream changes first, then rebuild agent state from verified inputs. Test the corrected workflow with recorded incident cases plus nearby variations. Use a sandbox or shadow environment, then read-only mode, then low-risk tools, then a small canary population. The same principles used in the AI agent canary rollout checklist apply: define success, rollback, exposure, and observation windows before each expansion.
Recovery requires independent evidence. A successful agent message is not proof. Verify downstream state directly, check that revoked credentials remain invalid, confirm the contaminated memory is excluded, exercise the emergency stop, and compare traces against policy expectations. For code or configuration changes, use the normal agent deployment pipeline with evaluations, review, tracing, and rollback rather than making a special unreviewed production edit.
Phase 6: Communicate, learn, and prevent recurrence
Write the post-incident review around system conditions and decisions, not blame. Document customer and business impact, timeline, detection source, authority exercised, contributing factors, control successes, control failures, recovery evidence, and remaining uncertainty. Assign every corrective action an owner, deadline, verification method, and risk if delayed. Add the incident and near misses to regression evaluations, threat models, tabletop exercises, and the agent registry.
Communication should match impact. Security, privacy, legal, compliance, vendors, customers, and regulators may need different facts at different times. Maintain one verified incident record so public or customer statements do not drift from technical evidence. If sharing externally, protect sensitive details while preserving the information needed for other defenders to learn.
Build an AI Agent Incident Evidence Manifest
An ordinary request log is not enough. Responders need a manifest that joins intent, context, authority, action, and effect. Store it in a tamper-resistant location with access controls and retention appropriate to the data. Redact secrets from analyst views, but preserve a secure way to prove which credential or identity was used.
- Identity: agent ID, version, owner, invoking user or service, delegated identity, session, tenant, environment, and active roles.
- Model and policy: provider, model identifier, model snapshot when available, parameters, system instructions, policy bundle, guardrail version, and evaluation release.
- Context: user input, system messages, retrieved chunks and source identifiers, memory reads, memory writes, peer-agent messages, and tool results in order.
- Decision trail: plan steps, policy checks, risk score, stop-condition state, approval request, approver, approval scope, and expiration.
- Actions: tool name and version, schema, arguments, target resource, response, retries, timestamps, request IDs, idempotency keys, and error handling.
- Effects: downstream audit events, records changed, messages sent, data exposed, code or configuration deployed, money moved, access granted, and pending jobs.
- Recovery: containment actions, credential revocations, reversals, restored versions, verification queries, test results, canary metrics, and final authorization to resume.
Preserve both raw and normalized forms. Raw artifacts support forensic accuracy; normalized fields support fast search and correlation. Hash or otherwise integrity-protect exports. Record who collected each artifact and when. If privacy rules limit retention of prompts or personal data, design selective capture, encryption, and controlled access before an incident rather than improvising after one.
The AI agent observability guide explains how traces connect model calls, tools, approvals, and outcomes. Incident readiness adds two requirements: evidence must remain available when the production system is paused, and investigators must be able to prove that the records were not silently altered by the affected agent or a compromised component.
Map the blast radius and preserve the evidence chain in parallel; waiting to finish one before starting the other creates avoidable blind spots.
Choose the Right Containment Control
The fastest shutdown is not always the safest investigation step. Pick controls according to the action path and the risk of continued execution. A model endpoint block stops new reasoning but may leave queued jobs and live tokens. Revoking a token stops one identity but may break unrelated services if credentials were shared. Deleting memory may remove the trigger but destroy evidence. The table below shows what each control does and what it can miss.
| Control | Stops | Does not automatically stop | Evidence caution |
|---|---|---|---|
| Pause agent runtime | New planning and tool dispatch from that runtime | Accepted jobs, callbacks, copied tokens, peer agents | Snapshot volatile run state first when safe |
| Disable tool or connector | Calls through that integration path | Direct API access or another connector to the same system | Preserve tool configuration and schema version |
| Revoke agent credential | Actions requiring that identity | Cached sessions, shared credentials, already authorized transactions | Record token ID and effective revocation time, never the secret value |
| Stop queues and retries | Deferred and repeated execution | External jobs already dequeued or scheduled elsewhere | Export pending message IDs and payload metadata |
| Quarantine memory or retrieval source | Further consumption of suspected content | Copies already present in prompts, caches, or peer-agent memory | Snapshot before removal; preserve source provenance |
| Block network egress | Most external communication | Local destructive action or access inside the boundary | Keep necessary forensic access on a separate trusted path |
| Global kill switch | Broad agent execution | Irreversible completed actions | Use for active severe harm; document collateral operational impact |
Containment should be layered. For an agent that is sending unauthorized email, pause the runtime, disable the send tool, revoke the mailbox token, cancel pending sends, and query the mail provider’s audit log. For an agent that modified code, suspend its repository identity, block pushes and merges, freeze the deployment queue, preserve commits and review events, and compare production state with the last trusted release. For poisoned memory, stop readers and writers, snapshot the store, trace every consumer, and rebuild from verified sources.
Recovery Gates Before the Agent Returns
Do not restore an agent because the immediate error disappeared. Recovery should pass explicit gates owned by different functions. High-risk systems need separation between the person who implemented the fix and the person who authorizes production re-entry.
| Gate | Evidence required | Owner |
|---|---|---|
| Cause understood | Fault tree identifies initiating event, enabling conditions, failed controls, and uncertainty | Engineering + security |
| Authority corrected | Permissions, credentials, tool scope, approvals, and session behavior verified independently | Identity/security owner |
| State repaired | Downstream changes reversed or accepted; contaminated memory and artifacts removed from active use | System and business owners |
| Regression coverage added | Incident case and variations fail before the fix and pass after it in an isolated environment | Evaluation owner |
| Observability proven | Traces and alerts capture the triggering pattern, action path, denial, and outcome | SRE/observability |
| Emergency stop tested | Stop, revoke, queue pause, and tool disablement work within the required response time | Platform operations |
| Canary succeeds | Defined volume, duration, risk scope, success metrics, and rollback conditions pass | Release owner |
| Business approval | Residual risk and customer impact accepted by the accountable owner | Business owner |
Recovery should be progressive. Start with replay against preserved cases in a sandbox. Move to shadow mode where the agent proposes actions but cannot execute them. Next allow read-only tools, then low-risk reversible writes, then limited production traffic. Observe behavior across normal, adversarial, ambiguous, and degraded conditions. If any gate fails, return to containment or eradication rather than lowering the bar.
The NIST AI RMF Generative AI Profile connects post-deployment monitoring with incident response, recovery, decommissioning, and change management. That is the correct mindset: recovery is not a single restart. It is a governed transition from untrusted state to demonstrated control.
Interactive AI Agent Incident Triage Helper
Use this lightweight helper during preparation or an early incident call. It is not a substitute for your organization’s severity policy, legal obligations, or safety procedures. It highlights when broad containment and cross-functional escalation are prudent.
The score deliberately favors uncertainty. Unknown scope plus active authority is more dangerous than a well-understood defect that deterministic controls blocked. If an agent can still act, containment takes priority over precise scoring.
Run a Tabletop Exercise Before Production Needs It
A tabletop reveals whether the runbook maps to real controls. Choose a plausible scenario: a support agent retrieves a malicious instruction from a customer ticket, uses an overbroad mailbox token, sends unauthorized messages, writes a misleading summary into shared memory, and schedules follow-up tasks. At the start of the exercise, give participants only the first alert. Release new facts every ten minutes.
Questions the exercise must answer
Who can declare the incident? Which on-call rotation receives the signal? Where is the incident channel? Who has final authority to disable the agent?
Can the team pause the runtime, stop queues, disable one tool, revoke one agent credential, quarantine memory, and confirm the change downstream?
Can responders retrieve exact context, model and policy versions, approval state, tool arguments, downstream audit events, and every related run without using the affected agent?
Who decides whether to contact recipients, customers, vendors, legal counsel, privacy teams, or regulators? Which facts must be verified first?
Who restores data, tests the correction, approves canary scope, watches metrics, and decides the agent may resume?
How are new controls, evaluations, registry fields, training, and architecture changes assigned and verified after the exercise?
Measure the drill. Useful metrics include time to declare, time to stop new tool actions, time to revoke authority, percentage of downstream effects traced, percentage of evidence fields available, time to produce a customer-impact estimate, and time to demonstrate a safe recovery path. Do not optimize only for shutdown speed. A team that stops the runtime in two minutes but cannot find queued actions or poisoned shared memory is not incident-ready.
Repeat the exercise after major model, prompt, tool, identity, memory, orchestration, or ownership changes. Rotate participants so response does not depend on one expert. Turn every failed drill step into an owned engineering or process improvement and verify it in the next exercise.
Common Incident-Response Mistakes
Treating every event as “the model hallucinated”
This label hides the control chain. Even if the model produced an incorrect plan, the system chose the available tools, authority, approval rules, and retry behavior. Root cause should explain why the output became an action and why the action created harm.
Clearing memory before preserving it
Deleting contaminated memory may prevent spread, but it also removes the trigger, provenance, and list of exposed consumers. Snapshot first when safe, revoke access, then rebuild active state from verified sources.
Rotating one key and declaring containment
Agents often hold multiple sessions and delegated paths. Inspect service accounts, user delegation, connector tokens, browser sessions, cached credentials, webhooks, queues, and peer agents. Confirm revocation at every downstream system.
Trusting a clean replay
A new run may use different model behavior, context order, retrieval results, time-dependent data, or environment state. Replay is a test, not proof of what happened. Preserve the original trace and state.
Restoring full autonomy immediately
Jumping from shutdown to normal production makes the first users part of the test. Use shadow, read-only, reversible, and canary stages with clear rollback conditions.
Writing a postmortem without changing controls
A polished timeline does not reduce recurrence. Convert lessons into enforceable permission changes, approval binding, tool validation, monitoring, stop conditions, evaluations, registry updates, and rehearsed response procedures.
Final Recommendation: Make Incident Readiness a Property of the Stack
The safest AI agent is not one expected to behave perfectly. It is one whose authority is bounded, actions are observable, evidence survives failure, side effects are reversible, and recovery is governed. Incident response cannot be bolted on after an agent has already gained broad permissions and become embedded in critical workflows.
Start with three deliverables. First, create a registry entry for every production agent with owner, identity, tools, data, stop method, and recovery path. Second, implement an evidence manifest that joins model context to downstream effects. Third, rehearse one realistic incident until the team can stop actions, revoke authority, preserve evidence, trace blast radius, and restore service without relying on the affected agent.
The broader infrastructure matters because no single control is enough. Identity limits authority. Policy and approval constrain high-risk action. Observability reconstructs the run. Registries reveal ownership and dependencies. Durable execution exposes queues and retries. Evaluation tests the fix. Release engineering makes recovery gradual. Together, these layers turn an AI agent from an opaque autonomous process into an operable production system.
FAQ: AI Agent Incident Response
What counts as an AI agent incident?
An AI agent incident is an event in which an agent or its supporting system causes or could cause unauthorized, unsafe, destructive, deceptive, or materially incorrect action. It includes security compromise, reliability failure, policy bypass, excessive permissions, poisoned context or memory, harmful external communication, and failed human-control mechanisms.
Should the first step be shutting down the model?
The first objective is stopping harm. Pausing the agent runtime may be necessary, but it is often insufficient. Teams may also need to stop queues and retries, disable write-capable tools, revoke credentials, cancel downstream jobs, quarantine memory, and preserve evidence. Choose the containment path that actually interrupts authority and action.
What evidence should teams preserve?
Preserve agent and user identity, model and policy versions, prompts, retrieved context, memory reads and writes, tool schemas, tool arguments and results, approval records, credentials used, trace IDs, retries, downstream audit events, state changes, queued work, containment actions, and recovery verification.
How do you contain an agent without destroying evidence?
Prefer pausing execution, revoking access, disabling connectors, and quarantining state over deleting logs or memory. Snapshot volatile state when safe, export immutable records, record each containment timestamp, and keep investigation access separate from the affected agent’s authority.
Who should own an AI agent incident?
Use one incident commander with clear authority. Technical response usually spans platform engineering, security, SRE, the agent’s product owner, identity administrators, data owners, and affected business teams. Privacy, legal, compliance, communications, safety, and vendor contacts should join when the impact requires them.
When can an AI agent return to production?
Return only after the cause and blast radius are understood, affected state is repaired, authority is corrected, incident cases pass regression tests, observability and emergency stop controls are verified, a limited canary succeeds, and the accountable business owner accepts residual risk.
Can the affected agent help investigate its own incident?
It may provide leads in a contained environment, but it should not be trusted as the sole investigator or remediator. Use independent logs, downstream system records, deterministic queries, human review, and separately trusted tools. Do not restore the same permissions merely to let the agent explain itself.
How is AI agent incident response different from normal cyber incident response?
The core lifecycle is the same, but responders must also preserve probabilistic context, memory, retrieval, model and prompt versions, tool-selection paths, approvals, delegated identity, peer-agent messages, and business side effects. Replaying the visible prompt may not reproduce the original run.
How often should teams test the runbook?
Test before production and after material changes to models, prompts, policies, tools, connectors, identity, memory, orchestration, autonomy, ownership, or deployment. High-impact agents should have recurring exercises based on risk and change frequency, with measurable follow-up actions.
