Deep Diving into LLMs, Part 5: Hallucinations, Tool Use, and Why Models Don't Know Themselves
Part 4 covered how a base model gets turned into an assistant through post-training on conversation data. This part gets into what I'd call the psychology of these models: why they confidently make things up, how that's actually being fixed, how giving a model tools changes what it's capable of, and the genuinely strange answer to "what does a model know about itself."
Why Hallucinations Happen in the First Place
Hallucination is what it's called when a model confidently states something completely false. Once you understand post-training, the reason becomes obvious. Imagine the training set has conversations like "who is Tom Cruise" answered correctly and confidently, "who is [some senator]" answered correctly and confidently, and so on. The human labelers who wrote those answers either already knew the person or looked them up, and either way, they wrote a confident, complete answer.
Now at test time, someone asks about a person who doesn't exist, made up entirely. The model has never been shown an example in training where the correct move is "I don't know." So it does the only thing it's ever been trained to do, it imitates the confident, complete-answer style from training and just makes something up that's statistically plausible. This isn't the model being deceptive, it genuinely has no concept that it's supposed to express uncertainty here, because uncertainty was never modeled for it.
A demo made this really concrete: asking an older model (Falcon 7B) "who is Orson Kovats" (a name that doesn't correspond to a real person) repeatedly. Every single time, it confidently invented a different answer, an author, a fictional TV character, a former baseball player. Different answer each time because it's just sampling from probabilities, not retrieving a fact, because there is no fact to retrieve.
Teaching a Model to Say "I Don't Know"
The fix sounds almost too simple: you need training examples where the correct answer actually is "I don't know," but only for cases where the model genuinely doesn't know. The tricky part is figuring out where that boundary actually is.
Here's roughly the approach Meta describes using for Llama 3. Take a random paragraph from the training data, use an LLM to generate specific factual questions from it, then interrogate the model being trained by asking it those questions multiple times. If the model consistently gets the answer right across several attempts, it knows this one, leave it alone. If it consistently gets it wrong or contradicts itself, that's a question it doesn't actually know the answer to. For those cases, you construct a new training example where the "correct" response is a clear, honest "I'm not sure" or "I don't remember," and add that into the training set.
Do this across enough documents and questions, and the model starts to associate its own internal sense of uncertainty (there's likely something like an "uncertain" signal buried somewhere in its parameters) with actually saying so out loud, rather than just powering through with a confident guess. It's a genuinely elegant fix, you're not teaching new knowledge, you're teaching the model to expose uncertainty it already has internally but was never trained to voice.
Giving the Model Tools: Turning Guesses Into Lookups
There's a second, more powerful mitigation beyond just saying "I don't know": let the model actually go check. This works through special tokens again, similar to the conversation formatting tokens, except these ones trigger an action instead of just marking structure.
The model can emit something like a search-start token followed by a query, then a search-end token. When the system sees that pattern, it pauses generating from the model entirely, actually performs that search against a real search engine, takes the retrieved text, and pastes it directly into the context window. Now that information isn't a vague recollection anymore, it's sitting right there in the model's working memory, directly accessible for constructing the next tokens of its response, citations and all.
This is genuinely the same thing you and I do when we don't know something offhand, we look it up, and once it's in front of us we can reason about it accurately. The distinction worth remembering: knowledge baked into the model's parameters is like something you read a while ago, a vague, sometimes unreliable recollection. Knowledge sitting in the context window is like something you're reading right now, direct and immediately usable.
This has a genuinely practical implication for how you should use these tools day to day. If you ask a model to summarize a well known book from memory, it can probably do a reasonable job because it's absorbed a lot about it during training. But if you actually paste the text you want summarized directly into the prompt, you'll consistently get a better result, because now the model isn't recalling, it's reading. Same idea as you producing a much better summary of a chapter you just reread versus one you're recalling from months ago.
The Model Doesn't Actually Know What It Is
The last piece, and maybe the strangest one, is what happens when you ask a model "what are you" or "who built you." It feels like a fair question, but it kind of isn't, because the model has no persistent identity at all. It doesn't exist between conversations. Every single chat starts completely fresh, gets processed, and then the whole thing is discarded. There's no continuous "self" living somewhere that remembers being asked this before.
So when you ask an older, unmodified model who built it, and it confidently answers "OpenAI," that's not necessarily evidence it was trained on OpenAI's data. It's more likely that the model absorbed an enormous amount of internet text where "ChatGPT" and "OpenAI" show up constantly in exactly this kind of question and answer format, and it's simply pattern matching onto the most statistically common answer to that shape of question, regardless of who actually built it.
If a company wants a model to answer this consistently and correctly, they have to explicitly program it in, either by hardcoding a batch of conversations into the training set (literally things like "tell me about yourself" paired with the correct company/product description, repeated across a couple hundred variations), or by inserting an invisible system message at the start of every conversation that states the model's name, who built it, its knowledge cutoff, and so on. Both approaches are really just bolting an identity on from the outside. There's nothing internally that "knows" this the way a person knows their own name, it's stitched together through data and prompting, not something that emerged on its own.
That wraps the post-training side of things for now. Between this and the pretraining posts, that's the full pipeline: raw internet text, tokenized and filtered, trained into a base model that's just a very good autocomplete engine, then reshaped through conversation data into something that behaves like an assistant, complete with its own quirks around confidence, memory, and identity.