OpenAI Research Links AI Hallucinations to Inadequate Evaluation Incentives

OpenAI Research Identifies Evaluation Incentives as Key Driver of AI Hallucinations

Despite significant progress in large language models (LLMs) such as GPT-5 and ChatGPT, hallucinations—defined by OpenAI as “plausible but false statements”—remain a persistent challenge. A new research paper by OpenAI investigates why these AI systems continue to generate confidently incorrect information and explores potential remedies.

The researchers illustrate the problem with a striking example: when queried about the title of Adam Tauman Kalai’s Ph.D. dissertation, a widely used chatbot produced three distinct answers, all incorrect. Similarly, questions about Kalai’s birthdate yielded three different, all incorrect, responses. Kalai is one of the paper’s authors, underscoring the direct nature of the test.

OpenAI attributes hallucinations partly to the pretraining process, which emphasizes predicting the next word in a sequence without associating statements with true or false labels. Models learn from fluent language patterns but lack grounding in factual accuracy. While predictable patterns like spelling errors diminish with scale, arbitrary, low-frequency facts—such as personal birthdates—cannot be reliably predicted, leading to hallucinations.

Incentives in Model Evaluation Encourage Guessing

Crucially, the paper shifts focus from pretraining to how LLMs are evaluated. Current evaluation frameworks reward models solely based on accuracy—correct versus incorrect answers—without accounting for uncertainty. This creates an incentive for models to guess answers rather than admit uncertainty, as guessing can yield partial credit or lucky correct answers, whereas abstaining results in zero.

The researchers liken this to multiple-choice testing scenarios where random guessing is rational because it offers a chance of scoring, while leaving questions blank guarantees no points. This incentive structure encourages confident but incorrect responses, exacerbating hallucination issues.

Proposed Reforms: Aligning Evaluation with Uncertainty

To address this, OpenAI proposes revising evaluation metrics to penalize confident errors more heavily than expressions of uncertainty. They suggest adopting scoring methods similar to standardized tests that impose negative marks for wrong answers and partial credit for unanswered questions, thereby discouraging blind guessing.

Moreover, the paper argues that simply adding uncertainty-aware tests alongside existing accuracy-based ones is insufficient. Instead, the core evaluation metrics and leaderboards must be updated to disincentivize guessing and reward appropriate uncertainty expression.

As the researchers conclude, “If the main scoreboards keep rewarding lucky guesses, models will keep learning to guess.” This insight highlights the critical role of evaluation design in shaping the behavior and reliability of AI systems.

OpenAI’s findings offer a strategic direction for reducing hallucinations by refining incentive structures in model assessment, a key step toward more trustworthy AI.