Despite remarkable advances in artificial intelligence capable of complex software development and even solving problems in advanced physics, a surprisingly stubborn challenge remains: the Portable Document Format, or PDF. Developed by Adobe in the early 1990s, the PDF was designed to preserve the precise visual appearance of documents across different platforms. However, this very design—focused on layout rather than underlying data structure—has created a significant hurdle for AI systems attempting to extract meaningful information.
The core issue lies in how PDFs are constructed. Unlike formats that store text as logically ordered characters, PDFs primarily consist of character codes, coordinates, and rendering instructions. Essentially, they are blueprints for how to display text and images, not the text and images themselves. This makes it difficult for AI to understand the content’s semantic meaning. State-of-the-art models, when presented with a PDF, often resort to summarizing the document rather than accurately extracting specific data points, frequently misinterpreting footnotes as part of the main text, or even fabricating information entirely—a phenomenon known as “hallucination.”
The problem isn’t merely academic. The increasing availability of crucial information locked within PDFs is driving a need for better extraction methods. Last November, the House Oversight Committee released 20,000 pages of documents from the estate of Jeffrey Epstein, followed by over three million pages from the Department of Justice. These files, largely in PDF format, proved difficult to search effectively, even after the Department of Justice applied Optical Character Recognition (OCR) technology. According to Luke Igel, cofounder of the AI video editing startup Kino, the OCR was insufficient, rendering the files largely unsearchable without painstaking manual effort. “There was no interface the government put out that allowed you to actually see any sort of summary of things like flights, things like calendar events, things like text messages. There was no real index. You just had to get lucky and hope that the document ID that you were looking at contains what you’re looking for,” Igel said.
The challenge extends far beyond legal documents. Countless organizations—businesses, governments, and research institutions—rely on PDFs to store vast amounts of data. Approximately 80–90 percent of organizational data is estimated to be unstructured, much of it trapped within formats like PDF that resist automated analysis. This presents a significant bottleneck for data analysis and machine learning initiatives.
Several companies are now attempting to overcome these limitations. One approach, exemplified by companies like Reducto, involves segmenting PDF pages into their constituent components—headers, tables, charts, and body text—before routing each component to specialized parsing models. This strategy borrows from computer vision techniques used in the development of self-driving vehicles, where identifying and classifying objects within an image is crucial for navigation. By breaking down the PDF into smaller, more manageable parts, these systems can improve accuracy and reduce errors.
The potential benefits of unlocking PDF data are substantial. Researchers at Hugging Face recently discovered approximately 1.3 billion PDFs residing within Common Crawl, a publicly available web archive. The Allen Institute for AI has highlighted that PDFs could provide trillions of novel, high-quality training tokens—the building blocks of AI models—from sources like government reports, textbooks, and academic papers. As AI developers increasingly seek large datasets to train their models, the information contained within PDFs represents a valuable, largely untapped resource.
However, the inherent complexity of PDFs means that a perfect solution remains elusive. PDFs often contain two-column layouts, complex tables, charts, and scanned documents with poor image quality, all of which further complicate the extraction process. Many PDFs are essentially images of text, requiring Optical Character Recognition (OCR) software to convert those images into machine-readable data. The accuracy of OCR can be significantly affected by the quality of the original image, particularly in older documents or those containing handwriting.
The pursuit of reliable PDF data extraction is not simply a technical challenge; it’s a crucial step towards making information more accessible and usable. As AI continues to evolve, the ability to effectively process and analyze the vast amount of data locked within PDFs will become increasingly important for a wide range of applications, from legal discovery and government transparency to scientific research and business intelligence.
