<Phi2AI/>
Post-OCR Correction: The Last Mile of Digitization
Heritage

Post-OCR Correction: The Last Mile of Digitization

#OCR-Correction#Datasets#HuggingFace#ELTeC#NLP
Data Engineering // Cleaning

  • The “99% Trap”: Standard OCR accuracy isn’t enough; a single typo breaks semantic search and LLM reasoning chains.

  • Custom Corpus: Created sagicc/postocr-sr, a parallel dataset aligning noisy raw text with human-corrected ground truth.

  • Script Agnosticism: The model handles mixed Cyrillic and Latin scripts, resolving “hallucinations” caused by OCR engine and scan quality.

The “99% Trap” in Digital Archives

In our work with Poljoindustrija Newspapers, we noted the importance of OCR (Optical Character Recognition). But any one working with archival text knows the “99% Trap.” Even a 99% accuracy rate means that in a standard novel, there are thousands of errors.

For a human reader, a typo like “Tnink” (need “Think”) is confusing. For an LLM, it is noise. It breaks tokenization, confuses semantic embeddings, and degrades the model’s ability to reason about the text.

To fix this, we rely on manual corrections, which is hard and long process. To improve and speed up the process we need Automated Post-OCR Correction.

The First Step: Sagicc/postocr-sr

We are releasing a “Work in Progress” dataset designed specifically to train models that can “heal” broken text. Hosted on Hugging Face, postocr-sr is a parallel corpus aligning raw, noisy OCR output with perfect, human-corrected ground truth.

Dataset Specs

  • Repository: Sagicc/postocr-sr
  • Size: 104 aligned document pairs (Novels from 1840–1920)
  • Format: JSONL (Pairs of raw_ocr_text and corrected_text)
  • License: CC-BY-4.0

The Engineering Challenge: “Hyphens & Hallucinations”

This dataset is derived from the ELTeC (European Literary Text Collection) project, specifically the Serbian subset. It provides a unique training ground for solving specific Serbian NLP challenges:

  1. Hyphenation Removal: Old prints often break words at the end of a line (e.g., “sku- pilo” instead of “skupilo”). Standard rules fail here; the model must understand morphology to rejoin them correctly.
  2. Script Agnosticism: The dataset contains examples in both Cyrillic and Latin scripts, ensuring that correction models don’t fail when switching alphabets, which is a common issue in Serbian digitization.
  3. Archaic Noise: It captures the specific “noise signature” of 19th-century fonts, allowing models to learn the difference between a speck of dust on the paper and a diacritical mark.

Methodology: From Distant Reading to Deep Learning

This work builds on the shoulders of the JeRTeH society and the University of Belgrade. By converting their XML/TEI academic standards into machine-learning-ready JSON/Parquet formats, we are bridging the gap between “Digital Humanities” and “Generative AI.”

The goal is to fine-tune a small AI model that can take a noisy string as input and output clean, vector-ready text.

Work in Progress

This is an active research dataset. The “Raw” column contains real-world noise, and the “Corrected” column represents the gold standard. We invite the community to test different architectures on this data.

Artifact Resources

Back to Artifacts