AI Agents Rarely Crash. They Fail Quietly, and Research Shows Why

Large language model agents now answer support tickets, query databases and plan trips. When they go wrong, they usually return a confident answer and a success code. A growing body of research explains why these failures are hard to see, and how engineering teams are learning to measure them.

A Failure That Looks Like Success

Traditional software tends to fail loudly. A function throws an exception, a server returns an error code, and a monitoring dashboard turns red. An AI agent behaves differently. It is a loop: a language model reads a prompt, decides to call a tool such as a search API or a database, reads the result, and decides what to do next. Each step can go wrong while every network request in the chain still succeeds.

Consider a common case. A weather tool fails and returns the text “service unavailable” inside an otherwise normal response. The model reads that text, apologises politely, or worse, writes a plausible forecast from its training data. The user receives a fluent answer. The server logs show a successful request with normal latency. Nothing in a conventional monitoring stack records that anything went wrong.

Model providers describe the same gap in their own documentation. Google’s Gemini API guide notes that the model “doesn’t execute the function itself”. It only proposes a function name and arguments, and the developer’s application runs the call. The application is therefore the only part of the system that sees the real result. Anthropic’s engineering guide on building agents makes the related point that agents need “ground truth” from the environment at each step, such as tool results, to judge their own progress. When a tool reports a failure as ordinary text, that ground truth is wrong, and the agent has no way to notice.

This is not a new kind of problem, only a sharper version of an old one. In 2015, Google engineers led by D. Sculley described “hidden technical debt” in machine learning systems. They argued that most of the risk in a production ML system sits outside the model code, in data dependencies, configuration and glue code. Agents add a new layer to that list. Prompts and tool descriptions are configuration that can change a system’s behaviour without any change to its code.

Small Edits, Large Effects

How much can a small prompt change matter? In a paper presented at ICLR 2024, Melanie Sclar of the University of Washington and colleagues changed only the formatting of few-shot prompts: separators, spacing and capitalisation. The meaning stayed the same. Performance on the same task still varied by up to 76 accuracy points on LLaMA-2-13B. The sensitivity did not disappear with larger models, more examples or instruction tuning.

The authors recommend reporting a range of performance across plausible prompt formats rather than one number. For engineering teams, the practical lesson is simpler. A prompt edit is a code change. It needs a version history, and it needs a test before it reaches users.

The same applies to tool descriptions, which the model reads as part of its prompt. Anthropic’s guide advises that tool definitions deserve “just as much prompt engineering attention” as the main prompt. It describes a coding agent that kept making mistakes with relative file paths. When the tool was changed to require absolute paths, the model used it without errors. The model did not change. Only the description it read did.

One Success Proves Little

A second line of research looks at consistency. The τ-bench benchmark, published by Shunyu Yao and colleagues at Sierra and presented at ICLR 2025, simulates customer-service conversations in which an agent must use tools and follow business rules. The authors introduced a metric called pass^k: the probability that an agent solves the same task correctly in all k attempts.

The results were sobering. A GPT-4o agent solved about 61% of retail tasks on a single attempt. When it had to succeed on the same task eight times in a row, the rate fell to about 25%. A demo that works once says little about how a system behaves on the thousandth run.

Where Multi-Agent Systems Break

Many teams now chain several agents together, with one planning, another executing and a third checking the work. A 2025 study from UC Berkeley, “Why Do Multi-Agent LLM Systems Fail?”, analysed more than 1,600 annotated execution traces across seven popular multi-agent frameworks. The authors, led by Mert Cemri, built a taxonomy called MAST with 14 failure modes in three groups: system design issues, misalignment between agents, and task verification.

Their central finding is that many failures come from how the system is designed, not only from the limits of the underlying model. Agents ignore their assigned role, lose track of the conversation, or declare a task finished without checking the result. These failures leave no error behind. They are visible only if someone can read what each agent actually did. Unite.AI’s coverage of the multi-agent paradox makes a related point from architecture: adding agents can amplify an early error rather than cancel it out.

Making the Invisible Visible

The engineering response borrows from distributed systems, where requests have long been followed across many services with a technique called tracing. Applied to agents, each user request becomes a trace, and each model call or tool call becomes a “span” inside it. A span records its input, its output, the tokens used and how long it took. The industry standards body for this work, OpenTelemetry, is drafting shared conventions for generative AI spans. As of 2026, those conventions are still marked as in development, so attribute names may still change.

Several open-source projects such as Langfuse, Arize Phoenix and AcruxCore already apply this idea. Their source code is public, so teams can inspect how a trace is stored and run the software on their own servers. They record model calls and tool calls as spans of a single trace and let teams attach quality scores to them. The choice of tool matters less than one detail: a tool span must record what the tool actually returned, not only whether the network request succeeded. Without that, the weather-tool failure described above stays invisible.

Industry surveys suggest that tracing is now common, but testing lags behind it. LangChain’s State of Agent Engineering report surveyed 1,340 practitioners in late 2025. Quality was the most cited barrier to putting agents into production, named by 32% of respondents. 89% of organisations reported some form of observability for their agents, and 62% had detailed tracing of individual steps and tool calls. Only 52.4% ran offline evaluations on test sets, and 37.3% evaluated live traffic. Most teams can see what their agents did. Fewer check whether it was correct.

Grading Answers at Scale

Traces show what happened. They do not say whether the answer was good. Reading every conversation by hand does not scale, so teams increasingly use a second model as a grader, an approach known as LLM-as-a-judge. A NeurIPS 2023 paper by Lianmin Zheng and colleagues found that strong models such as GPT-4 agreed with human preferences more than 80% of the time, about the same rate at which humans agree with each other.

The same paper documented the method’s weaknesses. Judge models prefer the answer shown first, prefer longer answers, and can favour text written by themselves. Judge scores are therefore estimates. They are useful for volume, but they need to be checked against a small set of human-labelled examples, with answer order randomised.

From Practice to Policy

This measurement work is also becoming a governance expectation. The US National Institute of Standards and Technology published its AI Risk Management Framework in January 2023. One of its four core functions, “Measure”, asks organisations to quantify, track and document AI risks over time. For an agent, traces and evaluation scores are the evidence that function asks for.

What Teams Can Do Now

This echoes the argument made on this platform that reliability is the real test of agentic AI. The research above adds the measurement detail underneath that argument: reliability is not a property you observe once, it is a range you have to keep measuring.

The research points to a short set of habits, and the major model providers recommend many of the same ones. OpenAI’s practical guide to building agents tells teams to “set up evals to establish a performance baseline” before optimising, and to hand control to a human when an agent exceeds a set number of failed attempts or faces a high-risk action. Anthropic recommends “extensive testing in sandboxed environments” before agents act on their own.

In practice, that means a few concrete steps. Keep prompts and tool descriptions under version control, as code. Test each prompt change against a fixed dataset, and expect a range of results rather than one number. Run each test case several times, because one pass hides inconsistency. Record tool outputs inside traces, so a failed tool cannot hide behind a success code. Treat automated judge scores as estimates, and check them against people.

None of these habits require a breakthrough in model capability. They require treating an AI agent as what it is: a probabilistic system that must be measured, because it will rarely announce its own failures.