nanochat GRPO Is Just REINFORCE + a 1-Line Regex
The RL entrypoint in nanochat is a single file, scripts/chat_rl.py. At the top of that file, Karpathy has written this sentence:
“I put GRPO in quotes because we actually end up with something a lot simpler and more similar to just REINFORCE.”
He then lists four things his implementation removes from the GRPO recipe as it appears in the DeepSeek-Math paper (arXiv:2402.03300) and in Hugging Face TRL. The reward is a one-line regex on GSM8K’s #### 10 answer format. The advantage is r - mu. That is the whole objective.
I read chat_rl.py end to end this week and I want to walk through what actually runs, what got cut, and why it still learns.
The reward is a regex. That is not an abbreviation.
Every RL result you have seen from a lab in the last two years (process reward models, verifier LMs, code-execution graders, RLHF preference heads) has a training pipeline whose first challenge is “define the reward function.” nanochat sidesteps that entirely by picking a dataset whose ground truth is already parseable text.
GSM8K’s canonical answer format is the literal string #### <number> at the end of the model’s chain of thought. Every training example has it. So tasks/gsm8k.py extracts the answer with a single regex, applied to both the reference and the generated completion:
GSM_RE = re.compile(r"#### (\-?[0-9\.\,]+)")
def extract_answer(completion):
match = GSM_RE.search(completion)
if match:
match_str = match.group(1).strip()
match_str = match_str.replace(",", "")
return match_str
return Nonereward() is 1.0 if the extracted strings are equal after normalization, 0.0 otherwise. There is no reward model, no LM-as-judge, no partial credit for showing work. The comment in the file says later versions could get more complex (“e.g. format matching etc.”), but the current implementation just re-uses evaluate() above.
This is a design constraint, not a limitation someone forgot to fix. The whole rest of the file gets to be simple because this line is simple. If reward were a model, chat_rl.py would need its own training loop for that model, its own evaluation, its own drift monitoring. Instead it needs re.compile.
The loop: 1 problem, 16 samples, one gradient step
Once reward is a function you can call, the RL loop collapses to three moves per problem.
- Pull a GSM8K problem and clip the assistant’s answer down to the
<|assistant_start|>marker. - Ask the model to complete it 16 times (
--num-samples 16) withEngine.generate_batch. - Score all 16 completions, subtract the batch mean, backprop.
The advantage calculation is 3 lines:
rewards = torch.tensor(rewards, dtype=torch.float, device=device)
mu = rewards.mean()
advantages = rewards - muIf all 16 completions were right, advantages is a zero vector and the gradient is zero, so no learning happens that step. Same if all 16 were wrong. Only when the group disagrees with itself does anything move. That “same-problem group comparison” is where the G in GRPO comes from, and it is the one part of the GRPO name that chat_rl.py keeps.
The policy-gradient objective is 4 lines:
logp = -model(inputs, targets, loss_reduction='none').view_as(inputs)
pg_obj = (logp * advantages.unsqueeze(-1)).sum()
num_valid = (targets >= 0).sum().clamp(min=1)
pg_obj = pg_obj / (num_valid * num_passes * examples_per_rank)
loss = -pg_obj
loss.backward()Log-prob times advantage, summed, normalized by valid token count, negated. That is the entire objective. There is no ratio, no clip, no reference-model KL term, no separate value head, no GAE. If you have seen the PPO paper’s boxed objective and thought “there is a lot going on there,” chat_rl.py is what happens when someone deletes everything that is not carrying its weight for this specific task.
The 4 things that got removed
Karpathy’s scare-quote paragraph lists four differences from GRPO-as-published. I found each one in the code (or, more precisely, found each one absent from the code, which is the point).
No trust region. DeepSeek-Math’s GRPO (2402.03300) sets a KL coefficient of 0.04 against a reference model. That coefficient is there to keep the policy from drifting too far from the SFT checkpoint per step. chat_rl.py does not load a reference model at all. There is no ref_model variable, no per-token KL penalty in the objective. If the policy wants to drift, it drifts.
No importance ratio, no clip. PPO and GRPO both compute pi_new(a|s) / pi_old(a|s) and clip it to some [1-eps, 1+eps] window so a single gradient step cannot change the action distribution too aggressively. chat_rl.py is on-policy: the completions in the current batch were generated by the exact weights being updated, so the ratio is definitionally 1 for the samples it just took. Karpathy skips the ratio and the clip and treats each batch as a one-shot REINFORCE update.
No z-score. GRPO’s original advantage is (r - mu) / sigma. Dividing by the standard deviation is what makes it “relative” in the paper’s technical sense. chat_rl.py divides by nothing. It uses r - mu and lets the raw variance flow into the gradient magnitude. The comment in the code says this explicitly: “Calculate the advantages by simply subtracting the mean (instead of z-score (x-mu)/sigma).”
Token-level normalization (DAPO-style). The final divisor is num_valid * num_passes * examples_per_rank, an effective-token count across the batch, rather than a per-sequence average. This is closer to DAPO’s normalization than to the sequence-averaged GRPO objective. It matters when completions vary in length, which they do in GSM8K.
None of these removals are principled proofs that the removed pieces are useless. They are engineering choices for a repo whose goal is that the whole RL implementation stay small enough for one person to read in an afternoon. The KL term needs a reference model; carrying a reference model around means twice the parameters in memory. The importance ratio needs a second forward pass on old logprobs; skipping it saves a forward. Each removal saves an amount of code and complexity larger than it looks.
Why the honesty is worth reading
There is a version of this repo that keeps GRPO in the module docstring, unquoted, and lets the reader assume the acronym means what it means everywhere else. That version would be indistinguishable from chat_rl.py in file size, in behavior, and in the resulting GSM8K score. The only thing it would lack is the scare quotes.
The scare quotes matter because RL for LLMs is at the stage of its lifecycle where the vocabulary is running ahead of the implementations. “GRPO” now covers a family of algorithms that share a group-relative advantage but disagree on almost everything else: trust region or not, z-score or raw mean, sequence- or token-normalized. A recent arXiv survey (2509.24203) argues that “group-relative REINFORCE is secretly an off-policy algorithm” and demystifies exactly the myths that a plain reading of the name creates.
chat_rl.py picks one point in that family (the simplest one, the one that is basically REINFORCE) and labels it accurately. If you are building an RL fine-tune for your own model this month, you will read a lot of papers with GRPO in the title. Some of them will be Karpathy’s variant. Some of them will have four extra knobs. You will save yourself a debugging session by knowing which one you have.
What this does not tell you
chat_rl.py is not in the speedrun. runs/speedrun.sh, the 79-line script that produces the $48 nanochat run, walks tokenizer → pretrain → SFT → eval and stops. RL is opt-in, invoked by hand with torchrun --standalone --nproc_per_node=8 -m scripts.chat_rl. The README notes that GradScaler support for float16 exists for SFT but not yet for RL, which is a fair signal that this part of the codebase is more experimental than the rest.
The GSM8K lift from this RL step is modest. Speedrun-tier report cards show GSM8K moving from around 4.5% after SFT to around 7.5% after RL, and it is measured with run_gsm8k_eval, which uses pass@k: generate device-batch-size samples per question, check whether any of them contain the right #### <number>. The training loop and the evaluation loop use the same primitive: “sample many, judge by regex.” Learning and evaluation are the same shape.
If you want to see what a maximalist GRPO stack looks like, DeepSeek-Math is one direction and Hugging Face TRL’s GRPO is the other. If you want to see how much of that stack a competent implementer will delete when the task is narrow and the reward is a regex, chat_rl.py is 326 lines and the answer is “almost all of it.”
Related reading on this blog
- nanochat: I Retrained GPT-2 for $48 in 2026 — the same repo, the pretrain side, and where the $43K → $48 number comes from
- Natural-Language Agent Harnesses (arXiv 2603.25723) — a different arXiv paper, same instinct: name the primitive you are actually running
If you liked reading a file end to end, the Japanese-language book I wrote walks through nanochat this way (8 chapters, one per major script), and this GRPO chapter is chapter 5.
Was this article helpful?