AI Incidents Are Becoming Operational Crises. We Need to Treat Them That Way.

For the last few years, most organizations have talked about AI risk in the language of governance.

Is the model accurate? Is it fair? Is the data approved? Are we complying with emerging regulation? Those are important questions, but they are not the only questions. The more urgent question is what happens when something goes wrong?

What happens when an AI agent takes an action it was never meant to take? What happens when a model leaks sensitive data? What happens when a hallucinated answer creates legal exposure, or an automated decision affects a customer, employee, patient, or partner in a way that cannot easily be undone?

And perhaps most practically: where does the organization go to coordinate the response? That is the shift now underway. AI risk is becoming an operational resilience issue, not just a governance issue.

AI is moving from experimentation into the working machinery of the enterprise. It is being embedded into customer support, software development, financial operations, healthcare workflows, hiring, claims processing, supply chains, and internal automation. As AI becomes more connected to the business, AI failures become business incidents.

The OECD AI Incidents Monitor tracked 596 AI Incidents in January 2026 alone, 200% growth year over year. Efforts like the OECD AI Incidents Monitor and AI Incident Database documents negative or harmful outcomes involving AI systems so the industry can learn from experience in the same way aviation and cybersecurity have done for years.

That comparison is important. Mature industries ask how to prevent failure AND how to respond when failure happens anyway.

AI Incidents Do Not Behave Like Traditional Software Bugs

A traditional software bug usually has a relatively clear boundary. Something breaks, engineers investigate, and the team reproduces the issue, patches the code, and ships a fix.

However, AI incidents are messier. They can be probabilistic, intermittent, and emerge from the interaction between a model, a prompt, a retrieval system, a plug-in, an agent, a user, and a downstream business process. Sometimes the AI system is operating exactly as designed, but the design is incomplete for the environment in which it is being used. That makes the response more challenging. 

A hallucination in a consumer chatbot is one kind of problem. A hallucination inside a legal, financial, clinical, or HR workflow is another. A biased output in testing is serious. A biased decision process running at scale in production is something else entirely. An AI assistant that drafts an email is one thing, but an agent that can change permissions, issue refunds, update records, trigger workflows, or execute code is a very different category of risk.

The MIT AI Risk Repository captures a broad range of AI risks, including false or misleading information, privacy and security failures, discrimination, misuse, and system safety issues. OWASP’s Top 10 for LLM Applications similarly highlights risks such as prompt injection, sensitive information disclosure, insecure output handling, and excessive agency. These are practical failure modes, rather than abstract technical concerns. 

If an AI agent has too much authority, it may take actions no human intended. If a prompt injection succeeds, the system may disclose information or follow hostile instructions. If sensitive data leaks through a model or retrieval system, the response has legal, regulatory, customer, and reputational implications. That is why the language of AI governance can sometimes become too passive. Governance tells us what should be true. Incident response tells us what to do when reality is moving faster than the policy.

The Response Will Be Cross-Functional

One of the biggest lessons from cybersecurity is that incidents rarely stay inside the security team. At the beginning, an event may look technical. Very quickly, it involves legal, communications, business leadership, compliance, customer teams, outside counsel, insurers, forensic specialists, and sometimes the board. 

AI incidents will follow the same pattern.

Imagine a model that exposes sensitive customer information. Security and privacy teams need to understand what happened. Legal needs to assess obligations. Communications may need to prepare for customers, regulators, or the media. Engineering may need to disable or roll back a system. Business leaders may need to weigh continuity against containment.

Or imagine an AI agent that begins taking unintended actions across enterprise systems. The technical team may be able to shut it down, but the organization still needs to know what it did, who was affected, what decisions were made, whether contractual obligations were triggered, and how the same failure will be prevented in the future. That cannot be solved by the AI team alone.

Organizations should build cross-functional muscle memory before they need it. That means clear escalation triggers, clear roles, clear decision rights, clear communications paths, clear documentation, and practice.

In a crisis, coordination is infrastructure, not just a soft skill.

The AI Being Investigated Should Not Control the Response

There is another issue organizations need to think about much more carefully. If the AI system under investigation can access the same communications, documents, workflows, or automation being used to coordinate the response, the organization has a problem.

In cybersecurity, this is a familiar principle. If ransomware has compromised the corporate network, you do not coordinate the response on systems the attacker may be able to read, disrupt, or manipulate. You move out-of-band. You separate the incident from the response.

The same logic applies to AI.

An AI system may not be malicious in the human sense, but if it can see the response plan, summarize the response meeting, influence the workflow, recommend the next step, or operate inside the same environment being used to contain it, then the organization has not truly isolated the response.

This becomes even more important with agentic AI. Google’s Secure AI Framework highlights risks such as prompt injection, data poisoning, and rogue actions, and maps them to controls across the AI lifecycle. That is the right framing. As AI systems become more capable of acting across tools, data, and workflows, organizations need to think about model safety AND about operational separation.

Think of it like investigating a fire. You would not want the sprinkler controls tied to the same malfunctioning system you are trying to diagnose.

Prepare, Practice, Respond, Report

A useful framework for AI incident readiness is the same one that has matured in cybersecurity: prepare, practice, respond, report.

Prepare means defining incident types before they happen, like bias, hallucination, data leakage, model drift, prompt injection, agentic runaway, unauthorized tool use, and third-party model failure. Each requires different stakeholders and different decisions.

A good playbook should not be a 200-page document sitting in a folder. Nobody opens page 137 during a crisis. A good playbook is role-based, accessible, and actionable. Legal knows what legal needs to do. Engineering knows what engineering needs to do. Communications knows when to engage. The board knows when management will escalate.

Practice means running tabletop exercises. Not once a year as a checkbox, but often enough to build muscle memory. The first time the board discusses an AI incident should not be during a real AI incident. The first time legal, engineering, privacy, security, and communications work through an AI failure together should not be when customers are already asking questions.

Respond means coordinating the live event with discipline. Who is in the room? What facts are known? What facts are still uncertain? What decisions were made? Who approved them? What changed between hour 12 and hour 48?

Report means recognizing that AI regulation is becoming more concrete. The EU AI Act includes serious incident reporting obligations for providers of certain high-risk AI systems. The details will vary by jurisdiction, industry, and use case, but the direction is clear. AI incidents will increasingly require a defensible record of what happened, what was known, what actions were taken, and when.

AI Can Help, But It Cannot Replace Judgment

There is a temptation to think AI incident response should be fully automated. I think that is the wrong framing.

AI can help tremendously. It can summarize facts. It can identify missing information. It can compare an incident to prior patterns. It can draft after-action reports. It can help map regulatory obligations. It can reduce administrative burden when people are under pressure.

But in a serious incident, humans remain indispensable.

Someone has to decide whether the facts are sufficient. Someone has to weigh customer impact. Someone has to decide whether to pause a system. Someone has to determine whether the organization has crossed a reporting threshold. Someone has to communicate with accountability and empathy.

The right role for AI in incident response is not to replace the crisis team. It is to give the crisis team better context, faster.

NIST’s AI Risk Management Framework is useful because it frames AI risk management around four functions: govern, map, measure, and manage. For incident response, I would add one practical extension: rehearse.

A plan that has never been tested is not really a plan. It is a theory.

Boards Need a Playbook, Too

AI risk is becoming a board-level topic, but board involvement cannot stop at oversight slides. Boards need to understand their role before a crisis occurs.

When will the board be informed? What decisions require board input? What information will management provide? How will materiality, customer impact, legal exposure, regulatory obligations, and operational disruption be assessed?

Many organizations have security playbooks, privacy playbooks, communications playbooks, and legal playbooks. Far fewer have a board playbook for AI incidents. That gap will become more visible as AI systems move into regulated, revenue-generating, and customer-facing workflows. The board’s role is to help the organization make better decisions under pressure, not become more technical.

Trustworthy AI Requires Operational Resilience

There is a lot of conversation about trustworthy AI. That is the right aspiration, but trust is not created by principles alone. Trust is created when organizations can show how they prepare, how they detect issues, how they respond, how they communicate, how they document decisions, and how they improve.

Cybersecurity went through this same evolution. Organizations spent years investing in prevention, and they should continue to do so. But mature organizations eventually learned that prevention is not enough. You also need resilience. AI is entering that same phase.

We should absolutely build safer models, stronger controls, better evaluations, better red-teaming, and better governance, but we should also accept that incidents will happen. Models will fail, agents will behave unexpectedly, data will leak, humans will misuse systems, vendors will make mistakes, and regulations will evolve.

The question is whether the organization can respond with speed, coordination, judgment, and accountability when an incident occurs. That is how AI moves from experimentation to dependable infrastructure, and that is how resilience becomes culture.