Deep Diving into LLMs, Part 6: Why Models Are Bad at Counting, Spelling, and Sometimes 9.11 vs 9.9
Part 5 wrapped up the post-training side of things, hallucinations, tool use, and the strange non-identity of these models. This part covers something a little different and honestly one of the most fascinating parts of the whole video for me: the actual computational limits of how these models think, and why that produces some genuinely bizarre failures right alongside genuinely impressive capabilities.
Models Need Tokens to Think
Here's the core idea, and once it clicks, a lot of weird model behavior suddenly makes sense.
Every time a model generates a single token, it runs through a fixed number of layers of computation, maybe a hundred or so in a large modern model. That's it. That's the entire compute budget for producing that one token. It doesn't matter how hard the problem is, the model gets the same small, roughly fixed amount of computation to produce each token in the sequence.
This means a model literally cannot cram a large amount of reasoning into a single token. If you force it to, it will fail, not because it doesn't "know" how to solve the problem, but because there simply isn't enough computation available in that one forward pass to get there.
Here's the example that made this concrete for me. Take a basic word problem: Emily buys 3 apples and 2 oranges, oranges cost $2 each, total cost is $13, what does one apple cost. There are two ways a model could be trained to answer this:
Bad training example: "The answer is $3" (and then filler justifying it afterward)
Good training example: "The oranges cost 2 times 2, which is 4. So the apples cost 13 minus 4, which is 9. Divided across 3 apples, that's 3 dollars each."
The bad version forces the model to compute the entire answer inside the single token where it says "3", then everything after that is just decoration, the actual answer was already committed to before any of the reasoning "happened" in the token sequence. The good version spreads the computation out, each intermediate step (the oranges cost 4, the apples together cost 9) is cheap enough to fit inside the compute budget of a single token, and by the time the model reaches the final answer, all the intermediate results are sitting right there in the context window for it to reference. This is exactly why ChatGPT visibly "thinks out loud" through a problem rather than just blurting out a number, it's not performing for you, it genuinely needs those intermediate tokens to arrive at a correct answer.
You can actually watch this fail on purpose. Asking a model to answer a math problem in a single token works fine when the numbers are small and easy. Make the numbers bigger (bigger multiplications, more digits) and force a single-token answer, and it starts getting it wrong, because it's trying to do too much arithmetic in one compressed step. Let it "show its work" instead, and it gets the same problem right almost every time.
Why Models Struggle to Count
This same limitation explains why models are famously bad at counting things. Take a prompt like "how many dots are in this string" followed by a big block of dots. The model has to count all of them and output the answer as basically one token, again asking for way too much computation to happen in one shot. It'll often just get the number wrong.
The fix leans on the same tool-use concept from part 5 (where we covered web search), except here the tool is code execution instead. Ask the model to use code, and it works, but not for the reason you'd expect. It's not that the model suddenly becomes better at counting, it's that copying the string of dots into a Python string is easy for the model (that's basically just a copy paste operation over tokens it can already handle), and then the actual counting gets handed off to Python's built in counting function, which is guaranteed to be correct. The model isn't doing the counting anymore, the interpreter is.
Why Spelling Tasks Often Fail Too
This connects back to tokenization from way back in part 1. Models don't see individual letters the way we do, they see tokens, and a single token can represent a chunk of multiple characters glued together. Ask a model to print every third letter of "ubiquitous," and it often gets it wrong, not because it can't count to three, but because it doesn't actually have clean access to the individual letters inside a token. The word "ubiquitous" might just be three or four token chunks to the model, and pulling individual characters back out of those chunks is a genuinely hard, unnatural operation for it.
This is exactly the root of the famous "how many Rs are in strawberry" problem that went viral a while back, older models would confidently say two when the answer is three. It's the character-level blindness from tokenization combined with the earlier counting weakness, stacked on top of each other. Interestingly, this specific query works correctly in newer models now, though it's fair to wonder whether that's genuine improvement or just this exact question being patched after going viral.
The fix is identical to the counting fix: ask the model to use code. Copying a word into a Python string is easy, and then Python can reliably index into every third character, no mental gymnastics required.
The Genuinely Baffling One: 9.11 vs 9.9
This is the example that stuck with me the most because it doesn't have a fully satisfying explanation. Ask a model which is bigger, 9.11 or 9.9, and it will sometimes confidently (and incorrectly) say 9.11 is bigger, even while showing reasoning steps that should lead it to the correct answer. It's not consistent either, sometimes it gets it right, sometimes wrong, sometimes it even flip-flops mid-answer.
What makes this stand out is the contrast: these are models that can solve PhD level physics and math olympiad problems, yet stumble on a basic decimal comparison a middle schooler would get right every time. Researchers who dug into the model's internal activations for this specific failure found something genuinely strange, neurons associated with Bible verse numbering (like chapter 9, verse 11) light up during this comparison. The theory is that the model's internal pattern matching gets pulled toward a completely unrelated context, versification, where "9:11" coming after "9:9" makes total sense, and that association bleeds into a math context where it absolutely shouldn't apply.
It's a good reminder that even with a solid mental model of how these systems work, some failures are still just going to be genuinely strange and not fully explainable. It's a useful humility check: treat these models as extremely capable but occasionally jagged tools, not oracles you copy-paste answers from without a second look.
The throughline across all of this: whenever a task requires precise, reliable computation, whether that's arithmetic, counting, or character level manipulation, lean on tools like code execution rather than trusting the model's "mental math." The model is astonishingly good at language and reasoning patterns, but it is not a calculator, and it works best when you let it use one.