I had a 27B vision model running locally (IQ4_XS quantized, 15GB resident) and needed to decide whether it was worth using for OCR. The comparison was macOS's built-in Vision framework (VNRecognizeTextRequest) — a dedicated text-recognition engine, free, zero memory footprint.
My expectation going in: the specialist wins on character accuracy, and the general-purpose model is reserved for when you need semantic understanding. Slow and expensive, use sparingly.
That expectation was wrong, and it was wrong in a way that would have been invisible in production.
The usual mistake in an OCR comparison is measuring against real documents, where you don't have ground truth. Then you can't distinguish plausible output from correct output — and plausible output is exactly what both engines produce when they fail.
So I rendered a 1100×720 test image with the answer fixed in advance:
lines of prose
A0-1lO9-B8 — digit 1 next to lowercase l, capital O next todigit 0
-Then I looked at it. The first render was wrong — a label came out as tofu boxes (□□), because the monospace font had no Korean glyphs. If ground truth is broken at the moment you fix it, every measurement afterwards is void. That check costs thirty seconds and it's the whole experiment.
| Local 27B VLM | Apple Vision (dedicated) | |
|---|---|---|
| Character errors | 2 | 8 |
| Reading order | preserved | destroyed |
| Table cells dropped | 0 | 2 (the - cells) |
| Wall clock | 82.8s (cold) | 0.27s |
Vision's eight errors: IQ4_XS→I04_XS, cloud→cLoud (twice), tok/s→tok/5, two characters inside a code string, em dash —→-, arrow →→->.
300× faster. On accuracy alone, four times the error rate on a document of this size is arguably a fine trade.
Accuracy alone is not what decided it.
Vision returned the table decomposed by column. Three model names in a row, then three memory figures, then the speed and status columns appended at the end of the document.
Which means: you cannot recover which speed belongs to which model from the output. The row associations are gone. Not garbled — gone. The characters are all there, correctly grouped, in a well-formed sequence, and the relation between them has evaporated.
The VLM kept the rows. Next to that, 2 errors versus 8 is a rounding difference.
What this actually says is narrower than "the general model is better":
"A dedicated tool beats a general one" depends entirely on where you cut the task.
Vision is dedicated to character recognition. It is not dedicated to document understanding. My task needed the second and I was picking tools by the first one's benchmark. The specialist was genuinely better at the thing it specializes in — I had just mislabeled what I needed.
The obvious architecture is: run the cheap engine, detect failure, escalate to the expensive one. Almost everyone reaches for this.
It requires failure to be detectable.
Column-shredded output is syntactically perfect. It has plausible text, plausible structure, no error signal of any kind. Downstream, it is indistinguishable from a correct read. The information didn't get corrupted — it got dropped, and dropped information leaves no residue to detect.
This generalizes past OCR. Any escalation ladder — cheap model then expensive model, cache then origin, heuristic then solver — is only sound when the cheap tier's failure mode is observable at the boundary. If the cheap tier can fail by silently discarding a relation rather than producing a wrong value, "cheap first" isn't an optimization. It's an undetected data loss path with a cost saving attached.
The adversarial string A0-1lO9-B8:
| Attempt | Output |
|---|---|
| VLM, full image | A0-1109-B8 — 2 misreads |
| VLM, that line at 4× with an explicit "distinguish 0/O and 1/l" instruction | A0-1l09-B8 — recovered l, still lost O |
| Vision, on the enlarged crop | A0-1109-B8 — 2 misreads, unchanged |
I inspected the enlarged image myself. The font renders digit 0 with a slash through it and capital O as a plain oval. The two glyphs are visibly different. This isn't image ambiguity that more pixels would resolve — it's both engines genuinely misreading a distinguishable character, and resolution doesn't touch it.
So:
No OCR engine can be trusted on strings where homoglyphs change the meaning — codes, IDs, hashes, addresses, license keys.
That's not a tool-selection problem. It's a property of the entire tool class, which means the remedy isn't a better engine. It's human confirmation or a checksum. If you're about to build an OCR path for identifiers, build the checksum first.
Vision as the first pass; escalate to the VLM only for documents where reading order carries meaning — tables, forms.
With one correction to that rule, from the paragraph above: since column-shredding isn't detectable downstream, "escalate on failure" doesn't work for tables. If the corpus is mostly tables, go to the VLM first and eat the 300×. Cheap-first is only valid when failure is visible.
waste.
the conclusion.
On raw character accuracy in this measurement the general 27B model was ahead, 2 errors to 8, but that was not the decisive difference. The dedicated engine returned the table decomposed by column, so which speed belonged to which model was unrecoverable from the output. A specialist beats a generalist only within the cut of the task it actually specializes in.
When the cheap tier can fail by silently dropping a relation rather than producing a wrong value. Column-shredded OCR output is syntactically perfect and indistinguishable downstream from a correct read, so there is nothing for the escalation trigger to detect. Cheap-first is only an optimization when failure is observable at the boundary.
No engine in this class can. Both the dedicated engine and the vision model misread capital O as digit zero even at 4x magnification with an explicit instruction to distinguish them, on a font that draws the two glyphs visibly differently. That is a property of the tool class, so the remedy is a checksum or human confirmation rather than a better engine.