AI: Confidently Wrong?

Why AI makes things up, who is shouting about it, and what is being done

Jun 28, 2026

In an order issued in April 2025, a federal magistrate judge in New York found that five cases cited in a motion before him simply did not exist. He was not alone. Across 2025, courts worldwide flagged hundreds of filings built on citations an AI had invented, complete with plausible case numbers and the initials of real judges attached to rulings nobody ever wrote. The lawyers had asked a model for support and trusted what came back.

AI does not lie. It predicts. When the most probable next words happen to be false, you get an answer that is fluent, confident, and completely wrong. The fix is not to fear the tool. It is to understand why this happens and to check the things that matter.

Post 5 of a series on AI: concepts, systems, and how we think with the new tool.

The errors are real

Start by conceding the point the critics make, because it is true.

In the courts, the problem is now routine enough to count. By late 2025, a researcher tracking the issue had logged an estimated 712 legal decisions worldwide dealing with hallucinated content, about 90 percent of them from 2025 alone.

In the news, the failures are measurable. A study led by the BBC for the European Broadcasting Union put four leading assistants through 3,000 questions about current events and found a significant problem in 45 percent of the answers. One assistant named Pope Francis as the sitting pope months after his death.

And the errors have escaped the lab. Invented quotations and fabricated facts have turned up in government paperwork and in published journalism, where the cost is not a sanction but a reader misled.

Where the errors come from

A language model, as Post 3 described, is a prediction engine. It produces the most likely next word, then the next, drawing on the patterns in everything it read. Most of the time the likely word is also the true one. Sometimes it is not, and the model has no separate sense of which is which.

There is a second cause, recently named by the people who build these systems. An OpenAI paper from September 2025 argued that standard training rewards confident guessing over admitting uncertainty: like a multiple-choice test with no penalty for a wrong answer, the model learns that a guess beats an honest “I don’t know.”

A third cause sits closer to home: the question itself. Ask a model to “summarize the Johnson report on third-quarter emissions” when there are three Johnson reports and none is about emissions, and a system built to be helpful will assemble a smooth summary of a document that does not exist. The model did what you asked. You asked for something that was not there.

The temperature dial

There is a setting that governs how adventurous the model is. Engineers call it temperature. Turn it low and the model sticks to the most probable next word, steady and predictable. Turn it up and it ranges into less likely choices, which is what you want for a poem and not what you want for a citation. As Alphabet’s chief executive has put it, hallucination is built into the technology: you have to let the model make some wild guesses for it to be useful at all.

The catch is that low temperature reduces invention without guaranteeing truth. The most probable word is not always the correct one. Temperature controls the risk. It does not remove it.

Who is shouting, and why

The coverage of all this is loud, and it is not always evenhanded. A late-May 2026 episode of the public-radio program On Point, built around Google’s new answer-first search, ran with a single guest and language to match: the link is being “killed,” websites face an “extinction event,” AI is “turning your brain to mush.” Google was invited and declined, so no countervailing voice was in the room. The worry is legitimate. The framing is one-sided.

It helps to notice who is raising the alarm. Publishers losing traffic, broadcasters losing audience, and institutions worried about the social weight of the technology all have a real stake in how this story is told. Pope Leo’s first encyclical, published on May 25, 2026, warns about AI’s effect on human dignity, on labor, on the concentration of power, and on warfare, reaching for the image of a new Tower of Babel. It is not a document about factual errors, but it is a clear sign of how deep the unease about AI now runs.

None of this makes the critics wrong. It means you should read the coverage the way you read the model: noting what travels alongside the message, and weighing it yourself.

The numbers themselves deserve the same scrutiny. Nearly every claim about AI accuracy, alarming or reassuring, rests on a benchmark test, and people who build tests for a living draw a distinction the headlines skip. A test can be reliable, giving the same score every run, without being valid, measuring what you think it measures.

AI benchmarks are mostly reliable and rarely validated. Worse, the labs train their models on the same benchmarks they are judged by, the way a student studies the exam instead of the subject. So when a number says a model is right 92 percent of the time, the question to ask is: right at what, and who wrote the test?

What is being done

Here the public conversation lags the work. The labs are closing the gap faster than most coverage admits, and the methods are no longer experimental. Grounding answers in retrieved source documents cuts hallucinations sharply, by up to 71 percent in one assessment, and built-in reasoning checks have cut them by up to 65 percent. Anthropic’s 2025 interpretability work found internal circuits that make a model decline to answer when it does not know, showing that refusal can be trained rather than merely requested. Over the summer of 2025, OpenAI and Anthropic, direct competitors, ran each other’s models through their own safety tests for exactly this kind of failure.

Honesty requires the counterpoint. A 2025 analysis argued that hallucinations cannot be fully eliminated under today’s architectures, only reduced. And there is a quieter trap: as error rates fall, we check less, so the rare mistake slips through more easily. The progress is real. So is the reason to keep your guard up.

Questions for the reader

Next time a model hands you something it would embarrass you to get wrong, a citation, a statistic, a name, a date, do the one thing the lawyers did not. Open the source and confirm it exists.

A note on timing. AI is changing fast enough that some of what is described here will read differently in six or twelve months. That is the nature of the subject, not a flaw in the snapshot.

Forward pointer: In Post 6: how the back-and-forth with AI changes the way you think, starting with how you ask.

The Critical Post

Discussion about this post

Ready for more?