Deep Diving into LLMs, Part 4: How a Base Model Becomes an Assistant
Parts 1 through 3 covered pretraining end to end: collecting and filtering internet text, tokenizing it, training a network to predict the next token, and what a raw base model actually is (a very expensive autocomplete engine with no idea it's supposed to answer questions). This part covers the next stage, post-training, which is where a base model actually turns into something like ChatGPT.
Same Training, Different Data
The thing that surprised me most here is that post-training isn't some fundamentally different technique. It's the exact same algorithm as pretraining, predict the next token, compare to the right answer, nudge the parameters. The only thing that changes is the dataset. Instead of training on raw internet documents, you throw that dataset away and swap in a new one made entirely of conversations, structured examples of a human asking something and an assistant giving the ideal response.
Because you're not training on the entire internet anymore, just a curated set of conversations, this stage is dramatically faster. Pretraining can take something like three months across thousands of computers. Post-training on the conversation dataset can take a matter of hours. Same machinery, much smaller and much more deliberately curated data.
Turning Conversations Into Tokens
Everything a model deals with still has to end up as a one dimensional sequence of tokens, including conversations. So there has to be some agreed upon format for encoding "who said what, and when did their turn end" into that sequence. It's a bit like a network protocol, a strict structure everyone agrees to follow so the data can be reliably packed and unpacked.
Here's roughly what that looks like once encoded, using special tokens that mark where a turn starts, whose turn it is, and where it ends:
<|im_start|>system<|im_sep|>You are a helpful assistant<|im_end|><|im_start|>user<|im_sep|>who are you?<|im_end|><|im_start|>assistant<|im_sep|><|im_end|><|im_start|>assistant<|im_sep|>
And underneath, that turns into a sequence of token IDs like this:
200264, 17360, 200266, 3575, 553, 261, 10297, 29186, 200265, 200264, 1428, 200266, 29997, 553, 481, 30, 200265, 200264, 173781, 200266, 200265, 200264, 173781, 200266
Tokens like <|im_start|> and <|im_end|> (short for "imaginary monologue start/end") aren't actual words, they're brand new special tokens introduced specifically for this stage. The model has never seen them before post-training. Their whole job is to teach the model the structure of a turn: here's where the user's turn begins, here's what they said, here's where it ends, here's where the assistant's turn begins, and so on. Once a conversation is encoded this way, it's just another one dimensional token sequence, so all the same training machinery from pretraining applies directly.
Where the "Ideal Answers" Actually Come From
So who decides what the assistant should say in response to any given prompt? In the earliest version of this approach (OpenAI's InstructGPT work), the answer was human labelers, contracted workers given detailed instructions on how to write the ideal response to a prompt. The instructions typically boiled down to be helpful, be truthful, and don't answer things you shouldn't, though in practice these guidelines run for hundreds of pages that labelers have to study closely.
A labeler would be given a prompt like "list five ideas for how to regain enthusiasm for my career" and would write out, by hand, what a good assistant response should look like. Do this for hundreds of thousands of conversations covering a huge diversity of topics, train the model on all of them, and the model starts to imitate the statistical pattern of that helpful, truthful, careful persona. It's not learning to reason its way into being a good assistant, it's learning to imitate what these labelers wrote.
This has shifted a lot in the last couple of years though. It's now far more common for the data itself to be generated with the help of other language models, with humans editing or verifying rather than writing everything from scratch. Datasets like UltraChat are a good example, largely synthetic, spanning millions of conversations, with humans involved mostly for quality control rather than authorship.
You're Talking to a Simulated Labeler, Not Magic
This reframes what's actually happening when you use something like ChatGPT in a useful way. You're not talking to some emergent intelligence that went off and reasoned through your question. You're talking to a system that is statistically imitating what a human labeler, following a company's internal instructions, would have written in response to a similar prompt. And often these aren't random people either, for technical questions the labelers are frequently domain experts. So it's closer to asking "what would a skilled, instructed professional write here" than asking an oracle.
A nice way to see this: if you ask for the top five landmarks to see in Paris, and that exact kind of question is well represented in the post-training data, you'll get something very close to what a real labeler produced after doing their own quick research. If the exact question isn't in the training data, the answer becomes more emergent, blending the model's pretraining knowledge of the world with the general style and structure it picked up from post-training. Either way, the "personality" you're talking to is a statistical blend of instructed human behavior, not something the model invented on its own.
Next up in part 5: why models hallucinate, how that's actually being fixed, how tool use lets a model "look things up" instead of guessing, and the strange truth about what a model actually knows about itself.