LLM Hallucinations: Why Recall Failures Matter More Than Missing Knowledge

Knowledge Profiling Versus Traditional Accuracy Metrics

To map the gap between what models store and what they can retrieve, the researchers propose shifting evaluation focus from question-level accuracy to fact-level profiling. Rather than evaluating whether a model successfully handles a single standalone prompt, fact-level profiling examines one specific data point across a variety of settings. This determines whether the fact is stored in the model parameters, whether it can be queried from different directions, and what computational effort is required to retrieve it.

The framework distinguishes between whether a fact is parametrically encoded and whether it is known. A model encodes a fact if it can reproduce it when primed with its original training context. A model knows a fact if it can reliably answer questions about it across varied phrasings.

Encoding and recall failures are indistinguishable under accuracy metrics, yet they imply different limitations and solutions.

Google Research and Technion researchers

The researchers note that encoding failures call for pre-training interventions such as scaling model size, while recall failures suggest post-training interventions that improve how models utilize existing encodings.

Categorizing Knowledge Profiles Across Models

Using a sample fact about the band Oasis playing their first gig at the Boardwalk club, the study categorizes knowledge into five distinct profiles. Direct recall occurs when the model encodes the fact and accesses it without extra compute. An encoding failure represents empty shelves where the model neither encodes nor knows the fact. A recall failure represents lost keys, where the model has the fact encoded but cannot access it to answer direct questions.

Other categories include recall with thinking, where the fact is only successfully recalled when the model uses inference-time computation like Chain-of-Thought, and inference without encoding, where the model reasons across other data points without explicitly encoding the specific fact. Evaluating 13 models on over 4 million responses using the WikiProfile benchmark, the researchers found that frontier models fail to directly recall 26-34% of those encoded facts without thinking.

Whenever outputs turn out to be inaccurate, the standard response is scaling up by training a larger model or incorporating additional data. Both are expensive, and if the facts are already encoded, neither helps.

Nitay Calderon, Research Scientist at Google

Implications for Enterprise Developers and RAG Pipelines

The findings require developers to rethink how they approach factuality and pipeline architecture rather than defaulting to Retrieval-Augmented Generation or massive model scaling. Calderon notes that teams frequently deploy RAG to solve facts the model can already answer from memory, adding unnecessary latency and per-call costs.

While inference-time thinking recovers 40-65% of the encoded facts that models initially fail to directly recall, turning it on globally wastes compute budgets because only a minority of facts actually require thinking. Developers can also deploy generate-then-verify pipelines to catch mistakes since models are better at recognizing facts than generating them from scratch. The WikiProfile benchmark is available on Hugging Face for enterprise data engineering teams to diagnose whether their systems suffer from missing data or missing keys.