Chat Models Are Born in the Loss Mask — Reading nanochat's SFT

Say “hello” to a freshly pretrained GPT and nothing guarantees a greeting back. What you have is a text generator that predicts the continuation of web text. It does not yet have any concept of a conversation.

So where does the chat model come from? The answer is a mechanism called the loss mask. The name sounds grand; the implementation is an array of zeros and ones, one per token. That row of zeros and ones is what turns a text generator into a chat partner.

This post reads that mechanism in the code of Andrej Karpathy’s nanochat, a repository that fits the entire LLM production pipeline — tokenizer training through the chat CLI — into roughly 8,159 lines. I covered the economics before (how a $43,000 GPT-2 run in 2019 became $48 in 2026); this time it’s the single most interesting step inside the pipeline: the moment chat is born. The code read here is pinned to commit 92d63d4.

Three terms, quickly

Pretraining: the stage that drills the model on “guess the next token” over a huge pile of web text. Afterward the model is good at continuing text — and that’s all
SFT (Supervised Fine-Tuning): the stage that shows the pretrained model worked examples of conversations. In nanochat this is scripts/chat_sft.py
Loss: a score measuring the gap between prediction and answer. “Computing the loss” is essentially “grading the answer”

The loss mask, then, is simply a specification of which tokens get graded.

Pretraining never uses a single conversation token

Start with the special-token table in nanochat/tokenizer.py.

SPECIAL_TOKENS = [
    # every document begins with the Beginning of Sequence (BOS) token that delimits documents
    "<|bos|>",
    # tokens below are only used during finetuning to render Conversations into token ids
    "<|user_start|>", # user messages
    "<|user_end|>",
    "<|assistant_start|>", # assistant messages
    "<|assistant_end|>",
    "<|python_start|>", # assistant invokes python REPL tool
    "<|python_end|>",
    "<|output_start|>", # python REPL outputs back to assistant
    "<|output_end|>",
]

As the comment says — “only used during finetuning” — all eight tokens besides <|bos|> (the document separator) never appear in pretraining text. They are reserved seats in the vocabulary with nobody sitting in them.

To a model fresh out of pretraining, <|user_start|> is a nearly meaningless symbol. Turn-taking, conversation boundaries: all blank. SFT is the stage that writes meaning into those reserved seats, inheriting the base model’s knowledge and layering conversational behavior on top.

A conversation becomes a token list plus a grading sheet

render_conversation in nanochat/tokenizer.py converts conversation data into tokens — and returns a mask of the same length alongside. 1 means “grade this token”, 0 means “don’t”. Here is the tool-call section, verbatim:

elif part["type"] == "python":
    # python tool call => add the tokens inside <|python_start|> and <|python_end|>
    add_tokens(python_start, 1)
    add_tokens(value_ids, 1)
    add_tokens(python_end, 1)
elif part["type"] == "python_output":
    # python output => add the tokens inside <|output_start|> and <|output_end|>
    # none of these tokens are supervised because the tokens come from Python at test time
    add_tokens(output_start, 0)
    add_tokens(value_ids, 0)
    add_tokens(output_end, 0)

The second argument of add_tokens is the mask. Tracing every branch of render_conversation, the grading policy comes out as:

Part of the conversation	mask	Meaning
`<	bos	>`
User message (incl. its bracket tokens)	0	Not graded
Assistant message body	1	Graded
`<	assistant_end	>`
Tool-call expression	1	Graded
Tool output	0	Not graded

The model reads the whole textbook, but only the assistant’s lines are on the exam. That is the grading policy.

Tokens with mask=0 vanish from the grade book

chat_sft.py wires this grading sheet into training. In the next-token quiz, the correct answer is always “the token one position to the right”, so the mask is shifted right by one and overlaid onto the target labels:

# Apply the loss mask from render_conversation (mask=1 for assistant completions,
# mask=0 for user prompts, BOS, special tokens, tool outputs). mask[1:] aligns
# with targets (shifted by 1). Unmasked positions get -1 (ignore_index).
mask_tensor = torch.tensor(mask_rows, dtype=torch.int8)
mask_targets = mask_tensor[:, 1:].to(device=device)
targets[mask_targets == 0] = -1

The last line is the whole trick. Wherever mask=0, the target label is rewritten to -1, and nanochat’s loss (F.cross_entropy(..., ignore_index=-1) in nanochat/gpt.py) excludes those positions from grading. Think of a “not graded” stamp on parts of the answer sheet.

What actually gets graded during SFT: the assistant’s message body, <|assistant_end|>, and tool-call expressions. Nothing else. The model still reads user messages and tool outputs as context — they just never count toward the score.

Pretraining grades every token; SFT chooses which tokens to grade. What creates a chat model is not a new architecture or extra magic. It is the scope line on the exam sheet. About as glamorous as a term-exam syllabus, and just as decisive.

The calculator’s answer is deliberately not memorized

The best showcase of this design is GSM8K, the grade-school math dataset.

tasks/gsm8k.py extracts the calculator annotations <<expression=result>> embedded in the answers, and injects the expression as a python part and the result as a python_output part. Expression: mask=1. Result: mask=0.

So “when you need 17×3, call the calculator” is a graded behavior. The answer “51” itself is not. At inference time Python returns the result, so the model has no reason to memorize it. Grade the result too, and the model drifts toward memorizing answers instead of computing them — the study strategy of memorizing the multiplication table’s answers without understanding it. Humans have already proven that one fails; no need to implement it in a model.

SFT teaches behavior, not knowledge

The comment in runs/speedrun.sh says it plainly:

# SFT (teach the model conversation special tokens, tool use, multiple choice)

The training data has three pillars: SmolTalk (460K rows) for turn-taking, MMLU (100K rows × 3 epochs) for answering multiple choice, GSM8K (8K rows × 4 epochs) for math and tool use. MMLU’s model answer is a single letter like “A” — effectively teaching how to fill in a bubble sheet.

All of it is imitation of worked examples. The knowledge itself comes entirely from pretraining; SFT only teaches the form for retrieving that knowledge as conversation. The README’s own chat transcript has the speedrun model explaining plausibly why the sky is blue — and then the README urges you to ask why the sky is green. Perfect conversational form, uncertain contents. Everyone knows somebody like that.

Recap

Pretraining never uses the conversation tokens; conversation is written in later by SFT
render_conversation turns a conversation into a token list plus a grading sheet (mask)
Only the assistant’s message body, <|assistant_end|>, and tool-call expressions are graded
The mask shifts right by one onto targets; mask=0 positions are excluded via ignore_index=-1
Calculator expressions are graded, calculator answers are not — memorizing answers is a failed strategy, in humans and models alike

A loss mask is just zeros and ones in an array. And yet the choice of which tokens to grade is what shapes a chat model. That, for me, was the best part of reading nanochat’s SFT code.

Further material

The 12-slide overview deck of nanochat’s full pipeline is on Docswell (Japanese)
This post is adapted from chapter 4 of my book “nanochatで理解するLLM製造工程” (Japanese), which walks the entire pipeline from tokenizer training to running everything on a MacBook — available on Kindle (¥250)