Even Leading Language Models Are Prone to ‘Interpretive’ Transcription

New research from Amazon finds that the smarter the model, the more likely it will ‘get creative’ when asked to perform Optical Character Recognition.

Optical Character Recognition (OCR) is considered to be a largely-solved problem. If you scan some text or present a photo of the text to the OCR system, it performs elaborate individuation of each letter, making the best guess that it can on a per-letter basis:

Top left, the zoning of characteristic loci in a segmented identified letter (source); main image: segmentation result for an institutional repository (Source).

OCR interprets characters individually; those characters, in close enough proximity, may or may not constitute words or other linguistic constructs. OCR doesn’t care; by the time those issues arise, it is out the door and on to the next job, because spell-checking is not within its remit.

One of the most popular OCR systems is the venerable Tesseract, a strictly algorithmic interpreter, widely used in a variety of open source applications and workflows. With a wide range of ‘flat’ algorithmic OCR add-ons available across all operating systems, the ability to convert image-text back to editable text has now reached the status of a free commodity – even if no system is without flaws or foibles.

Additionally, in order to stay relevant, organizations and commercial products traditionally competing in the algorithmic OCR space have adapted their legacy systems into AI-aided document management systems. However, the core technology – spitting out estimated groups of segmented and identified letters – remains unchanged.

Word Up

Along with a growing number of tasks, OCR is being increasingly handled by the new generation of Vision Language Models (VLMs); confronted with non-editable or rasterized text, agentic versions of these latest systems can adroitly read and transliterate as necessary.

However, according to a new research paper from Amazon.com, those who believe that AI will replicate exactly the same function as OCR may be in for a surprise: the new study has found that your favorite LLM is likely to ‘go the extra mile’ in transcription, whether you want it to or not; exceeding per-letter context into per-word context – and correcting any ‘errors’ it finds, in ways that could have undesirable consequences for areas such as legal, medical and historical transliteration.

The authors assert:

‘Vision Language Models (VLMs) are increasingly used in place of traditional OCR pipelines for document understanding.

‘[We] show they do not always act as faithful transcribers: when text is imperfect, they often tend to rewrite it into a more plausible form – a behavior that clean-text OCR benchmarks cannot detect.’

The authors introduce a voluminous new curated dataset and benchmark titled FaithC4, which they tested across a selection of 15 FOSS and closed-source models, including versions of ChatGPT and Google Gemini.

They found that general-purpose AI models were far more likely than traditional OCR systems to replace imperfect words with more ‘plausible’ ones, sometimes rewriting nearly 65% of scrambled words, while small local errors also caused mistakes elsewhere in otherwise accurate text.

Short words were especially vulnerable, and explicit instructions not to make corrections reduced the problem, though it did not eliminate it.

The paper states*:

‘While the rewriting behavior can be beneficial for recovering some genuine errors, it may be problematic for some domains that may require literal transcription, such as legal documents, medical records, and historical manuscripts.

‘Despite its importance, this rewriting behavior has been largely overlooked in VLM research. Existing OCR benchmarks focus on clean text recognition and are therefore insufficient for investigating how models can handle imperfect inputs.’

The central message is that more advanced AI systems struggle not to correct errors which algorithmic OCR platforms can easily be configured to deliver ‘raw’; in many cases, human interpretation is essential to resolve the issue; but if an incorrect ‘correction’ has already been applied, the problem will tend to glide under the radar of most checking systems.

The new paper is titled Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models, and comes from five researchers at Amazon.com.

Data and Method

To populate their benchmark dataset, the authors drew on the Colossal Cleaned Crawled Corpus – a.k.a. C4 – a hyperscale web-crawled collection from 2020. From this corpus, 1,455 single-page documents were selected across English, Chinese and Korean languages, with each passage sized to fit on a single rendered page.

The final benchmark contained 500 English documents, 500 Chinese documents and 455 Korean documents, providing a multilingual test-bed for measuring whether AI systems copied text faithfully, or in some way rewrote it.

To test whether AI models would faithfully copy what they saw, controlled errors were introduced into the documents before they were rendered as images, with around 8% of eligible words altered, each containing at least four characters. Untouched versions of these were used as ground truth.

Three perturbation variants were then applied: scramble shuffled the internal letters of words while preserving the first and last letters; random replaced every character with a random alternative from the same writing system; and visual substituted visually similar characters that remained plausible to the eye, while changing the underlying text.

The Chinese variant omitted the scramble condition because its writing system lacks space-delimited words, instead using fixed four-character windows for the other perturbations; additionally, Korean’s shorter average word length meant fewer words qualified for modification.

Each document was then rendered as a PDF page, and passed to the competing transcription systems.

Tests

Tests for the study covered both closed-source general-purpose models, namely GPT-5.4-mini; Gemini-3-Flash; and Gemini-2.5-Flash; and open source models, comprising the general-purpose models Qwen3.5-4B; Qwen3-VL-4B; InternVL3.5-4B; Gemma4-E4B; and Gemma4-E2B.

Additionally, OCR-specialized models were included, comprising olmOCR-2-7B; DeepSeek-OCR-2; MinerU2.5-Pro; PaddleOCR-VL-1.5; and LightOnOCR-2-1B. Finally, the traditional OCR engines Tesseract and docTR were also tested.

The three groups rely on language to different degrees: general-purpose AI models draw heavily on broad language knowledge acquired during pretraining; OCR-specialized models place greater emphasis on faithful document transcription through task-specific training; and traditional OCR systems rely on character recognition, rather than linguistic inference.

Two metrics were adopted, beginning with Word Error Rate (WER), the standard OCR metric for determining how many complete words a system transcribed incorrectly. Rather than simply counting wrong characters, WER records substitutions, deletions and insertions relative to the original text, making it sensitive both to incorrect words and to words that have been omitted or added.

For Chinese, where words are not separated by spaces, the same approach was applied at the character level, as Character Error Rate (CER).

To capture subtler mistakes, the metric Edit Distance Similarity (EDS) was also used. EDS compares the predicted text with the original at the character level, and – unlike WER – can distinguish between a word that is completely wrong, and one that differs by only one or two characters. Naturally this makes it useful for detecting near-misses, as well as partial rewrites that might otherwise appear similar.

As shown in the graph below, traditional OCR systems proved the most resistant to all three perturbation types, with little loss of accuracy because they primarily recognize individual characters rather than attempting to infer words from context:

Changes in Word Error Rate (left) and Edit Distance Similarity (right) across English, Chinese and Korean after applying the three text perturbation methods. Traditional OCR systems showed the smallest performance degradation overall, OCR-specialized models occupied the middle ground, while general-purpose AI models were typically the most affected by corrupted text. Source

OCR-specialized AI models occupied the middle ground; MinerU and LightOn remained relatively robust; and olmOCR and DeepSeek-OCR showed greater sensitivity to corrupted text, indicating that document-specific training reduced, but did not eliminate, the tendency to reinterpret what was being read.

General-purpose AI models performed worst overall, exhibiting the largest increase in transcription errors when confronted with perturbed text. Most suffered substantially greater degradation than either OCR-specialized models or traditional OCR – though Gemini-3-Flash stood out as a notable exception, performing much more like a conventional OCR system.

Overall, the ranking closely matched the degree to which each model relies on language knowledge, with stronger language priors corresponding to a greater tendency to rewrite rather than simply transcribe text.

A breakdown of error types, shown in the table below, indicate that traditional OCR systems mostly produced conventional transcription errors, such as substitutions, insertions and deletions:

Results from the English ‘Scramble’ perturbation test, showing how each transcription system distributed its errors across exact rewrites, near-miss transcriptions (‘Close’), more substantial transcription errors (‘Moderate’) and outright incorrect outputs (‘Wrong’). Models are ordered by their rate of rewriting corrupted words.

Conversely, general-purpose AI models were far more likely to replace corrupted words with plausible alternatives while preserving the surrounding sentence, providing strong evidence that they were not merely misreading damaged text, but actively rewriting it using linguistic context.

The study also found that transcription errors did not remain confined to corrupted words:

Results from the error propagation analysis, showing per-word error rates on perturbed and unperturbed words after each perturbation type. Altering only around 4.7% of words caused transcription errors to spread into surrounding, unmodified text across many models.

As shown in the results table above and the graph below, altering only around 4.7% of the words in an English document substantially increased error rates in surrounding, unmodified text:

Further results from the error propagation analysis, showing the increase in transcription errors on both perturbed and unperturbed words after each perturbation type. General-purpose AI models exhibited the greatest error amplification, with mistakes frequently spreading beyond the originally corrupted words.

General-purpose VLMs were the most affected, with error rates on untouched words rising by as much as tenfold over baseline. Conversely, traditional OCR systems showed only modest increases. Models that rewrote the highest proportion of corrupted words also exhibited the strongest error propagation, suggesting that both behaviors stem from the same underlying reliance on linguistic context.

In practice, this means that a small amount of damaged text can compromise the accuracy of an otherwise intact document, making clean OCR benchmark scores an unreliable guide to real-world transcription fidelity.

Finally, the authors also examined whether word length influenced rewriting behavior, finding that shorter words were substantially more vulnerable than longer ones.

Because they contain fewer visual cues, brief words appear to encourage VLMs to rely more heavily on contextual language knowledge – increasing the likelihood that damaged text will be silently replaced with a plausible alternative, instead of being transcribed literally (which might cause a useful alert for human oversight).

The authors also found that while explicitly instructing models not to correct errors reduced this tendency, it does not eliminate it. This would seem to indicate that interpretive transcription is an inherent characteristic of current VLMs, rather than a behavior that can be reliably suppressed through prompting alone.

The authors conclude:

‘For applications that require literal transcription, such as legal document processing, historical manuscript digitization, and forensics, traditional OCR systems or carefully selected OCR-specialized VLMs remain the safest choice, while general-purpose VLMs introduce systematic rewriting that clean-text benchmarks do not capture.’

Conclusion

This study reiterates what is becoming a repetitive strand in LLM literature: that though trained VLM models are tremendously powerful, they are also resistant to critical instructions – in this case Read and parse the letters you find in the image.

The more advanced models just can’t do it, it seems – as if they considered it ‘scut work’, or else are irrationally compelled to conclude a ‘half-finished task’.

* My conversion of the authors’ inline citations to hyperlinks.

First published Monday, July 27, 2026