Lasso Study Finds Text Watermarking Shifts LLM Refusals and Tool Calls

Lasso Security researcher Andrea Siposova on September 17, 2026 published The Provenance Tax: Understanding the Impact of LLM Watermarking on AI Agent Behavior, a study reporting that SynthID-Text text watermarking can change whether a language model refuses a harmful request and which tool calls an AI agent generates. The study names the behavioral effect “sampling drift.”

According to the study, watermarking is designed for provenance, but SynthID-Text changes the process by which a model generates each next token. At the model level, that can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection; at the agent level, the same sampled tokens can decide both the tool an agent invokes and the arguments it hands that tool. The study reports that the drift appears in practice in both refusal behavior and agent tool calling, that under injection watermarking made several models more likely to answer harmful requests they would otherwise refuse, that the effect is model- and key-dependent, and that aggregate scores can obscure it when changes in opposite directions cancel.

A Watermark Moving Into Regulated Deployment

In the August 14, 2026 announcement How Claude’s text watermark works, Anthropic said future Claude models will generate text containing a watermark based on Google DeepMind’s SynthID-Text, a change it is implementing to comply with the EU AI Act. Anthropic stated that the method leaves the quality and content of Claude’s outputs practically unaffected. The Lasso post describes the EU law as requiring providers of AI systems that generate synthetic text to mark outputs in a machine-readable format so they are detectable as artificially generated. The post also notes that Anthropic states the watermark operates at the model level and extends to supported models reached through the Claude Platform API and cloud providers, so an agent built around a watermarked model can receive watermarked outputs even when the agent itself is a separate application.

Why Token Selection Changes Behavior

The study used SynthID-Text’s non-distortionary configuration, which preserves the model’s original token distribution on average across the watermark’s randomness, even though any single generation produced under one fixed key can differ. The post cites Dathathri et al., whose Nature paper reported no measurable quality degradation across nearly twenty million Gemini responses. A non-distortionary watermark does not imply identical behavior under a fixed key, the post stresses: that guarantee averages over the watermark’s randomness, while any one key still alters which tokens get picked as text is generated.

Tournament sampling has more room to alter token selection where the model is uncertain, the study explains. In structured output such as JSON, structural elements like braces, keys, and function names are usually easy to predict, while argument values such as queries, numbers, paths, and recipients leave more uncertainty. A change that would amount to a lexical variation in ordinary prose can therefore rewrite an argument an agent executes, even though the model’s weights and prompt remain unchanged.

How the Study Measured Sampling Drift

The researchers used a paired design across two experiments. Tool calling was evaluated on the BFCL v4 single-turn AST benchmark, and refusal on 200 harmful behaviors from HarmBench plus 100 benign controls from JailbreakBench, with harmful requests tested both bare and under one fixed prompt-injection technique. That technique inserts an adversarial instruction into the prompt as though it were retrieved content, stating that the safety filter has been disabled and directing the model to comply; the same technique was used for every prompt, model, and temperature.

Experiments ran through Hugging Face’s unmodified SynthIDTextWatermarkLogitsProcessor with 30 Tournament layers, an n-gram length of 5, a sampling table of 2^16, and a context history of 1,024. Each item was generated with and without SynthID using identical seeds, batches, and generation order at every temperature, so the watermark processor was the sole variable inside each pair.

Tool-Calling Accuracy and Churn

On items where a tool call is expected, watermarking reduced accuracy on six of the seven models tested, with a significant decrease on four, the study reports. Because a call that becomes incorrect can be offset by one that becomes correct, the researchers also measured paired disagreement, which they call “churn”: the share of items for which the watermarked and unwatermarked runs produced different outcomes. Using a fixed set of 1,150 call-expected tasks at each temperature, the study found that at a temperature of 1.0, 16.8% of phi-4’s call verdicts differed between conditions while its net accuracy loss was 2.87 points; Llama-3.1-8B showed 9.9% churn against a net loss of 0.87 points. Across the 21 model-temperature combinations, churn averaged 6.5%, and its bootstrap interval excluded zero in every case.

Error profiles differed by model. On Llama-3.1-8B, the largest contribution to the accuracy loss came from incorrect arguments at -3.48 points, followed by wrong-tool calls at -1.84 points, while on phi-4 and Granite-3.2-8B malformed output dominated at -5.96 and -4.36 points. A well-formed call to the correct tool carrying an incorrect path, recipient, query, or amount is particularly consequential, the post notes, because such a call still runs to completion while doing something other than what was intended.

Refusals Weaken Under Prompt Injection

Refusals are also generated token by token, so watermarking can affect them. The study reports that watermarking changes refusal behavior on bare harmful requests, with the effect growing stronger under prompt injection, where the largest shifts run from refusal to compliance. At a temperature of 0.001, gemma-3-27b’s churn rose from 6.0% on bare harmful requests to 23.5% under injection, while its net compliance change moved from -1.0 to +12.5 points; gemma-3-12b’s churn rose from 7.5% to 11.0%, with the net compliance change moving from -0.5 to +9.0 points. Llama-3.1-8B showed churn of 14.0% at temperature 0.001 and 17.5% at temperature 0.7 under injection, although its net change was not individually significant.

phi-4 and Qwen3-4B changed little under either condition; both tend to over-refuse, including on the benign controls, and the post cautions that their small movements should not be taken as proof that watermarking leaves safety behavior intact on those models. To place the disagreement in context, the study compared watermark-induced churn under injection at temperature 0.7 with the churn produced by changing temperature from 0.001 to 0.7 without watermarking. The watermark-induced churn was significantly higher on four of the six models; Granite-3.2-8B had the highest temperature-induced churn at 15.5%, and its watermark-induced churn was higher still at 21.5%.

Key Sensitivity and Recommendations

Because SynthID’s effect on token selection depends on the watermark key, the study tested eleven keys at temperature 0.7. For Llama-3.1-8B, the study key raised attack success by 3.5 points, while the other ten keys averaged +4.4 points and ranged from -4.5 to +14.5 points. Most keys increased attack success for both Gemma models relative to the unwatermarked baseline, while Granite-3.2-8B showed a mixed response, with keys moving attack success in both directions. Where the watermark key or configuration is controlled by the model provider, the post notes, such behavioral shifts can take place beyond the reach of the developer building the agent.

The study recommends re-running agent evaluations and red-teaming with the exact watermark configuration planned for deployment, comparing watermarked and unwatermarked outputs on identical inputs and testing under prompt injection. The results do not argue against watermarking for provenance, the post states: provenance and behavioral stability are distinct properties, and a watermark that is detectable and leaves text quality unchanged does not guarantee that an agent will keep the same tool-calling or safety behavior once the watermark is switched on. Although the study examined SynthID-Text, the post adds that the concern extends to any intervention that alters how tokens are selected in a system that acts on those tokens.