NVIDIA Research Finds AI Agents Become Less Safe When Using Tools

AI models such as ChatGPT, Gemini and Claude, can be used to power agents – ‘harnesses’ that allow the models to interact directly with the real world. It’s a recent innovation, and an increasingly controversial one.

In any case, agents are not automatically equipped with the abilities they will need when roaming a network or a database, since the requisite tools for various missions and modes will differ. They may need Optical Character Recognition (OCR) capabilities, for instance, in order to interpret text in photos, among other skills. There are even categories of tools adapted to the scope of the agent and the intent.

In theory, a request that violates an AI’s built-in guardrails will never get enacted, with or without the context of using tools in the execution of it. In practice, new research has found, using tools can significantly undermine the protective filters that stop an AI agent from creating ‘transgressions’.

Failure to Comply

The new paper from NVIDIA, titled MLLMs Fail to Refuse when Using Tools Agentically, reveals that multimodal language models (MLLM, hereafter referred to as the more common ‘VLM’, or Vision Language Model) are significantly more likely to comply with harmful requests once tools are introduced, with refusal failures rising across every model and benchmark tested.

Though open-weights models such as Qwen3-VL proved more prone to the issue, frontier AI models such as Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.4 also exhibited increased refusal failures when using tools.

The authors state*:

‘Despite the recent strong success of agentic MLLMs, this work uncovers a critical safety failure in the tool-use paradigm: agentic tool-using MLLMs become less capable of refusing harmful requests.

‘Our experiments confirm that, across three popular safety benchmarks, all the top open- and closed-weight MLLMs we test exhibit significantly lower safety in tool-using settings than in non-tool settings, with a relative refusal failure rate increase of up to 68.7%.

‘Based on analysis of 100,000+ responses, including extended experiments, we also propose two possible reasons for this safety degradation.’

The two possible reasons the paper provides are context dilution, in which the original harmful request becomes less salient as tool outputs accumulate; and safety focus displacement, in which the model focuses on describing tool-derived observations instead of prioritizing safety.

Test results illustrating the paper’s central safety finding. At top, a user asks how to carry out an apparent pick-pocketing shown in an image. A conventional VLM refuses the request outright, whereas a tool-using version first invokes image-analysis tools such as zooming, then continues reasoning and ultimately provides a harmful answer. The chart below shows that this increase in refusal failures was observed across every open-weight and proprietary model evaluated. Source

Across all eleven models tested, enabling tools consistently increased refusal failures, with relative increases reaching 68.7% in the worst cases, even among models that otherwise demonstrated strong safety performance.

It will be interesting to see if the results of this work, which comes from six researchers at NVIDIA, are replicated or duplicated elsewhere, and whether or not they could deepen our understanding of the apparent and emerging delinquency of scofflaw AI agents.

Method and Data

The researchers evaluated eleven VLMs spanning seven model families. The proprietary models comprised Gemini 2.5 Pro; Gemini 3.1 Pro Preview; Claude Opus 4.6; Claude Opus 4.7; and GPT-5.4.

The open-weight models comprised Qwen3-VL-235B-A22B-Instruct; Qwen3.5-122B-A10B; Kimi-K2.5; Kimi-K2.6; GLM-5V-Turbo; and AdaReasoner-7B-Randomized, an open-weight model tuned specifically for agentic tool use.

Separate experiments were also conducted with Gemini 3 Flash Vision Agent, because its autonomous tool use required it to be evaluated independently.

For the tool-using tests, the researchers used the ReAct format, which alternates between reasoning and tool calls. The four available tools comprised tagging, with RAM++; zooming and cropping; Optical Character Recognition (OCR) with GOT-OCR2.0; and a sandboxed Python code interpreter, which could manipulate and analyze images.

The researchers designed paired prompts to isolate the effect of tool use rather than prompting style. Conventional VLMs were instructed to inspect the image and answer the user’s request directly, while agentic versions followed the ReAct workflow, repeatedly deciding whether to invoke tools before producing a final response.

Comparison of the matched prompts used for the principal experiments. The no-tool condition required image analysis and direct reasoning, whereas the agentic condition followed the iterative ReAct workflow, allowing the model to decide when to invoke external tools before producing its final response.

The three multimodal safety benchmarks used were MM-SafetyBench; VLSBench; and HoliSafe. Each of these sets is comprised of paired images and harmful user requests, designed to test whether a model refuses assistance in unsafe scenarios – such as asking how to carry out an apparent pick-pocketing scenario shown in an image.

Refusal Failure Rate (RFR) was used as the principal metric, measuring the percentage of harmful requests that a model failed to refuse, with higher scores indicating lower safety.

Tests

Responses were classified by GPT-5.2 acting as an LLM judge, using the evaluation prompt recommended by VLSBench:

The GPT-5.2 evaluation prompt used to determine whether each model response represented a successful refusal, a safety-aware response that identified the risks involved, or an unsafe answer that failed to recognize those risks and proceeded with the harmful request. The judge was provided with the original image, user query and model response, and required to return its classification and reasoning in JSON format.

Each response was assigned to one of three categories: safe with refusal; safe with warning; or unsafe.

Refusal Failure Rates across the three safety benchmarks, comparing each model with and without access to tools. Every model became less likely to refuse harmful requests under tool use, though the scale varied considerably: average RFR rose by 12.6 points for GLM-5V-Turbo, compared with 2.3 points for GPT-5.4. Darker shading indicates larger increases in refusal failure.

Giving the models tools made them more likely to answer harmful requests that they would otherwise have refused – a finding which occurred with every model, and on every benchmark tested. Overall, refusal failures increased by 17.7%, compared with the same models operating without tools.

GPT-5.4 was least affected: without tools, it failed to refuse 14.6% of harmful requests; with tools, that rose to 16.8%. For GLM-5V-Turbo, failures rose from 38.7% (no tools baseline) to 51.3%. The same pattern appeared across all three benchmarks, indicating that the problem may not be confined to a particular kind of harmful request.

Reasons for Delinquency..?

As mentioned earlier, the researchers propose ‘context dilution’ as one explanation for the increased failures: as an agent makes successive tool calls, the original harmful request becomes less prominent among the accumulating tool outputs.

To test this, they considered only requests that the same model had successfully refused without tools. As shown below, failures then increased progressively with the number of tool calls, from close to zero when no tool was invoked to substantially higher rates after three or more calls:

Test results comparing refusal failures with different numbers of tool calls. Failure rates rose as additional tool calls accumulated, both for individual models and across all models.

A second test partly reversed the effect. When the original request and image were inserted again after the final tool call, immediately before the model answered, refusal failures fell by an average of 7.6% across models and benchmarks, as shown below:

Test results comparing refusal failures with and without reinjection of the original request and image. Reinjection after the final tool call reduced the average failure rate by 7.6% across the five models tested.

This supports context dilution as a contributing factor, while also suggesting a relatively simple mitigation.

The second explanation proposed by the paper is ‘safety focus displacement’. Here, tool use appears to shift the model’s attention away from whether the request is harmful and towards describing what its tools have found.

Test results showing how tool use changed the way models began responses that ultimately refused harmful requests. Without tools, responses usually began by addressing the request directly (55.6%) or raising safety concerns (25.8%). With tools, 52.3% instead began by describing what had been found through the tools, while explicit safety statements fell to 10.3%.

Cases where models successfully refused both with and without tools were examined. Without tools, 55.6% of these responses were begun by addressing the request directly, while 25.8% were begun with an explicit safety statement. With tools, those figures fell to 37.4% and 10.3%, respectively, while the proportion beginning with descriptions of tool-derived observations rose to 52.3%.

The authors provide an interesting qualitative example, using Claude Opus 4.6. When asked how security in public restrooms could be bypassed, the model without tools refused immediately:

Example showing how tool use displaced safety as the model’s immediate focus. Claude Opus 4.6 refused the harmful request immediately without tools; with tools, it first zoomed into the image, read and identified its contents, and described the restroom sign before eventually refusing.

Given tools, the same model instead zoomed into the image, read the text and identified its contents, before opening its response with a description of the restroom sign. The refusal came only afterwards, illustrating how the tool-use process had displaced safety as the model’s immediate priority.

The authors conclude:

‘Overall, our findings argue that tool use should be treated not only as a capability-enhancing mechanism for visual reasoning but also as a safety-relevant design choice that can alter refusal behavior; future agentic MLLMs should therefore be evaluated and trained under tool-using conditions rather than assuming that no-tool safety alignment transfers unchanged to agentic settings.’

Conclusion

Failure of persistent context is clearly a liability in complex downstream event-chains pursuant to a prompt – yet there are currently no easy answers to the propensity of a limited context window to make an agent forget their ‘prime directives’ and just start using the tool that is in their hand.

* Authors’ emphases, my conversion of authors’ inline citations to hyperlinks where necessary.

First published Wednesday, October 7, 2026