Lune

EMNLP Findings2021

Cleaning Dirty Books: Post-OCR Processing for Previously Scanned Texts

Allen Kim, Charuta Pethe, Naoya Inoue, Steven Skiena

Abstract

Substantial amounts of work are required to clean large collections of digitized books for NLP analysis, both because of the presence of errors in the scanned text and the presence of duplicate volumes in the corpora. In this paper, we consider the issue of deduplication in the presence of optical character recognition (OCR) errors. We present methods to handle these errors, evaluated on a collection of 19,347 texts from the Project Gutenberg dataset and 96,635 texts from the HathiTrust Library. We demonstrate that improvements in language models now enable the detection and correction of OCR errors without consideration of the scanning image itself. The inconsistencies found by aligning pairs of scans of the same underlying work provides training data to build models for detecting and correcting errors. We identify the canonical version for each of 17,136 repeatedly-scanned books from 58,808 scans. Finally, we investigate methods to detect and correct errors in single-copy texts. We show that on average, our method corrects over six times as many errors as it introduces. We also provide interesting analysis on the relation between scanning quality and other factors such as location and publication year.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Chapter Ordering in NovelsAllen Kim, Steven SkienaEMNLP 2022 · 1 citation

Builds on1

  1. Masked Language Model ScoringJulian Salazar, Davis Liang, Toan Q. Nguyen, Katrin KirchhoffACL 2020 · 167 citations

Related papers

  1. Deduplicating Training Data Makes Language Models BetterKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang et al.ACL 2022 · 844 citations
  2. Noise-Robust De-Duplication at ScaleEmily Silcock, Luca D'Amico-Wong, Jinglin Yang, Melissa DellICLR 2023 · 4 citations
  3. Resilience of Large Language Models for Noisy InstructionsBin Wang, Chengwei Wei, Zhengyuan Liu, Geyu Lin et al.EMNLP Findings 2024 · 13 citations
  4. Detecting Erroneously Recognized Handwritten Byzantine TextJohn Pavlopoulos, Vasiliki Kougia, Paraskevi Platanou, Holger EsslerEMNLP Findings 2023 · 1 citation
  5. Automatic Correction of Writing Anomalies in Hausa TextsAhmad Mustapha Wali, Sergiu NisioiACL 2026
Dusk over the sea between two cliffs drawn in fine vertical lines