The FineBooks project, a collaboration between Hugging Face and EleutherAI, tested 14 open-source OCR tools against over 2,000 historical book pages. The results show that while modern models produce text accurate enough for training AI, they are not yet precise enough for scholarly work.
Why bad text hurts AI training
Training open-source language models on public-domain books often involves processing text extracted years ago by optical character recognition software. That old text is full of errors. The Talkie project calculated the impact: a language model trained on OCR text learned at only 30 percent the efficiency of one trained on human transcriptions of the same books.
EleutherAI and its partners released Common Pile last year. This dataset, the largest openly licensed training corpus to date, contains about 300,000 public-domain books. Much of the text comes from older OCR runs. The FineBooks authors argue that reprocessing those books with better models is one of the most effective ways to improve open AI training datasets.
The team chose the Biodiversity Heritage Library (BHL) as its first target. It holds more than 300,000 digitized natural history documents totaling over 64 million pages. The library offers its collection as a bulk download via AWS.
To measure quality, the team needed pages with known correct transcriptions. They used work from the IMPACT project and BHL-Europe. Between 2011 and 2012, experts transcribed six BHL volumes in English, French, German, and Latin with an error rate of about one character per 2,000. That data is available under a CC-BY license in a GitHub repository and forms the basis of the new ground-truth dataset.
Small models beat larger rivals
All 14 models are freely available and run on local hardware without an API key. The metric is Character Error Rate (CER), the share of incorrectly recognized characters. The leaderboard splits results into a “diplomatic” variant that counts modernization of archaic characters like the long s (ſ) as an error and a “reading” variant that tolerates such changes.
The leading dots.mocr model uses just 3 billion parameters. Qwen3.5-9B scores lower despite being nearly three times as large. OvisOCR2 takes second place with only 0.9 billion parameters at 46 cents per thousand pages. Model size and OCR quality do not correlate for historical documents.
| Model | Size | Accuracy | Cost/1,000 pages |
|---|---|---|---|
| dots.mocr | 3B | 97.6% | $1.94 |
| OvisOCR2 | 0.9B | 96.9% | $0.46 |
| PaddleOCR-VL-1.6 | 1B | 96.1% | $0.34 |
| olmOCR-2 | 8.3B | 95.7% | $0.45 |
| LightOnOCR-2 | 1B | 95.1% | $0.37 |
| Qwen3.5-9B | 9.7B | 94.9% | $0.89 |
| DeepSeek-OCR | 3.3B | 93.8% | $0.37 |
The evaluation covers only Antiqua typefaces in four languages. It does not account for Fraktur, non-Latin scripts, or handwriting. FineBooks is limited to single-column book pages. The team plans to reprocess about 200,000 public-domain BHL documents with one of the top models and release the text as an open dataset. New models are added to the leaderboard on a rolling basis, and the evaluation framework is openly available.
Good enough for AI training, too inaccurate for scholarship
The FineBooks authors judge results by intended use. For training language models, the top performers work well enough. The models produce far fewer errors than the older pipelines. Reprocessing a collection the size of BHL is realistic at the measured costs.
Libraries face a different problem. Their systems rely on ALTO XML, a format with word-level coordinates. The new models output Markdown or plain text without word positions, so they cannot plug into existing library infrastructure.
For scholarly transcriptions, accuracy still falls short, but not because the models misread characters. They silently modernize them, swapping the long “s” or ligatures for modern equivalents. Targeted fine-tuning could fix this, the team says.
