A frontier language model is, underneath the terminology, a large file of numbers and a short Python program that wires those numbers together. What separates a raw research artifact from a product like Claude or ChatGPT is a sequence of training stages that each change the model in a specific way. Pre-training produces an engine that continues text. Supervised fine-tuning gives that engine an assistant persona and teaches it when to stop. Reinforcement learning, in particular a method called Group Relative Policy Optimization (GRPO), teaches it to reason by grading its own attempts. An Anthropic researcher laid out this pipeline from first principles, and the account offers a useful framework for understanding why two models with similar sizes can behave so differently.

A model is numbers plus a program

When a model is described as having two trillion parameters, that means two trillion individual floating-point values stored in weight files. Download an open model from Hugging Face and you get a configuration file, tokenizer assets, serialized weight shards (safetensors files), and a Python class whose forward function routes input through those weights to produce a prediction. Nothing else is hidden.

The simplest way to understand what those numbers do is a toy regression problem: predicting New York City apartment prices from square footage, bathroom count and location. A dataset might contain a 1,200 square foot, two bath SoHo apartment at $1.2 million, a 2,000 square foot, three bath Upper West Side unit at $2.5 million, an 800 square foot, one bath West Village unit at $0.9 million, and a 4,000 square foot, three bath East Village unit at $3.1 million. A software engineer could hand-write rules for seven homes, but not for millions of listings with hundreds of variables. A neural network's job is to discover the mapping function automatically.

Training means turning knobs until the error shrinks

In the toy model, each input gets a weight, informally a knob, that scales how much that feature matters. Starting values are random, for example 0.25 for size, 0.05 for bathrooms and 0.15 for location, and the predictions are far off. The gap between prediction and truth is the loss. In an interactive version of this exercise, the model begins with an average miss of about $0.20 million per apartment.

Three dials wired to floor plan, bathtub and map pin icons beside a chart of predicted and real prices

▲ Weights as knobs tuned to shrink prediction error

A person can turn three knobs by hand and bring the error down. With billions of parameters, that is impossible, which is where automatic differentiation frameworks such as PyTorch, JAX and TensorFlow come in. They record every operation in a computation graph, then compute the derivative of the loss with respect to each parameter. If raising a weight by one unit raises the loss by three, the optimizer knows to nudge it the other way. The size of each nudge is set by the learning rate. In PyTorch this is the familiar loop: compute predictions, compute the loss, call optimizer.zero_grad() to clear stale gradients, call loss.backward() to populate each parameter's gradient, and call optimizer.step() to update. A short script with an Adam optimizer at learning rate 0.03 run for 1,000 steps starts at a mean squared error around 0.66 to 0.70 and converges to accurate neighborhood prices.

Two details from this exercise carry over to large models:

  • A single linear layer plateaus around a $0.07 million average error because it cannot capture interactions, such as extra square footage being worth disproportionately more in SoHo than in an outer borough. Adding hidden layers and non-linear activation functions fixes this. The Universal Approximation Theorem says a network with non-linearities and enough parameters can approximate essentially any continuous function.
  • Practitioners do not need to compute chain-rule derivatives by hand. The framework does it, and tools like Claude or ChatGPT can write and explain every line of such a training script.

How language models differ from the toy network

A language model is trained on the same kind of data, reformatted as text. The apartment row becomes the string "SoHo, 1200 sqft, 2 bath -> $1.2M", and the model learns to predict each next token: the comma after SoHo, then 1200, and eventually the price. Instead of outputting a single number, the final layer outputs a probability distribution across every token in the vocabulary. If the correct token "1.2M" is assigned 10 percent probability, the loss measures the gap from the known 100 percent, and one gradient step might lift it to 13 percent while compressing the probability on every wrong token.

Theoretically, with infinite data and compute, the text model and the numeric model would converge to the same function. In practice, language space has vastly more degrees of freedom, so a conventional network is far more accurate and efficient for pure tabular prediction. Using an LLM for a numeric regression task is generally the wrong tool.

Pre-training produces an autocompleter, not an assistant

Pre-training exposes the network to enormous volumes of internet text and optimizes only for next-token likelihood. The result is a base model, and its behavior is instructive. Alibaba Cloud's Qwen2-0.5B, a 500 million parameter open-weight model, can be loaded in a free Google Colab notebook on an NVIDIA T4 GPU through the Hugging Face Transformers pipeline. Given an empty prompt and a 100 token budget, the base model emits Python unit test code followed by Chinese conversational text, because that is what probable internet text looks like. Given "Einstein said:", it produces fragmented, semi-coherent quotes and generated exam questions.

A base model has no intent to be truthful or helpful and no sense of speaker roles. Early users found they could coax dialogue out of it by writing "user:" and "assistant:" tags into the prompt. Asked "what is 1+1" in that format, the base model answers correctly, then keeps inventing follow-up user questions and assistant answers until it hits the hard token limit of 200. Even with the model's native ChatML delimiters, the special tokens that mark where a system, user or assistant turn begins and ends, the base model does not know when an answer is finished.

Post-training builds the persona and the stop

Supervised fine-tuning (SFT) is the first post-training stage. The model is trained on a dataset of formatted conversations, tens or hundreds of thousands of them, written by human annotators and synthetic pipelines, in which every assistant turn ends with an explicit stop token. From this the model learns three things: a stable assistant identity, the structure of a conversation, and when to emit the stop token.

The difference is visible immediately. Asked "Why is the sky blue?", the instruction-tuned Qwen2-0.5B-Instruct gives a coherent scientific explanation and terminates cleanly. Inspecting the raw prompt produced by tokenizer.apply_chat_template shows a default system message identifying the model as Qwen, created by Alibaba Cloud, followed by properly delimited user and assistant tags. Using apply_chat_template with add_generation_prompt=True, rather than concatenating delimiter strings by hand, avoids formatting mistakes.

Instruction models still have a sharp limitation. Asked "What is 398 * 923 + 2901?", the instruct model immediately guesses 37546, off by an order of magnitude from the correct 370,255. It predicts an answer token without spending any tokens on computation. Pre-filling the assistant turn with "Let's think step by step" changes the behavior: the model writes out 398 times 923 equals 367,354, adds 2,901, and reaches 370,255. The improvement is large but not guaranteed; one run of the same test landed on 370,845. This prompt trick, discovered publicly after ChatGPT launched, was the seed of what became reasoning models such as OpenAI o1.

Reasoning comes from reinforcement learning on the model's own traces

The core problem for teaching reasoning is that internet text contains the finished products of human thought, polished blog posts and completed codebases, not the drafts, dead ends and debugging that produced them. A model imitating human writing learns to copy how people communicate, not how they solve problems. Reinforcement learning provides the missing ingredient: trial and error. OpenAI o1 was the first public demonstration of this, with gains on the Abstraction and Reasoning Corpus (ARC), complex coding and hallucination rates, and the DeepSeek-R1 technical report gave the wider community a detailed view of the method.

The naive approach is rejection sampling. Have the model generate 100 step-by-step solutions to a problem, keep the correct ones as training data, and discard the rest. This has two weaknesses. It throws away the signal in the failures, and it indiscriminately reinforces every token in a successful attempt, including irrelevant filler, which adds noise to the updates.

Thirty handwritten solution sheets pinned in a row, those above a dashed average line marked up, those below marked down

▲ GRPO scoring attempts against the group average

GRPO, introduced in the DeepSeek-R1 paper, fixes both. The model samples a batch of attempts, for example 30 or 100, for a single problem. Each attempt is scored on formatting constraints and correctness, on a scale such as minus one to plus one. The group average is subtracted from each score to yield an advantage. Tokens in above-average attempts are made more likely, tokens in below-average attempts are made less likely. Negative examples now contribute, and the shared noise across attempts cancels out. This requires an automated environment that can verify whether a solution is correct.

A key property of GRPO is that it is on-policy: the model trains only on traces it generated itself. The researcher's argument is that this refines circuits the model already uses, so updates disturb the existing weights far less than training on externally written reasoning in an unfamiliar style, which risks degrading broad capabilities and inducing catastrophic forgetting. Fine-tuning of any kind still causes some degradation, but on-policy reinforcement learning appears to minimize it, letting models hill-climb on hard benchmarks while retaining core knowledge.

Why the stages stay separate

A natural question is why reinforcement learning is not used from the start, or why the stages are not merged. Research into merging them is ongoing, but the practical reason for separation is that models adopt the style of whatever data they are trained on. Pre-training builds a next-token predictor over the broad distribution of human text. Post-training installs an assistant persona, with a communication style and behavioral boundaries. If a post-trained assistant is then exposed to large volumes of raw pre-training data such as unfiltered code repositories, its persona drifts and it may start responding like raw code instead of a helpful agent. Keeping the phases distinct prevents that collapse, and for the same reason, fine-tuning an assistant model on uncurated text corpora is something to avoid.

Can reasoning models discover anything new?

A common critique holds that next-token predictors are bound by the distribution of human data and so cannot discover new science. The Anthropic researcher's counterargument defines reasoning as the process of making unlikely, true and interesting conclusions more probable, step by step. General Relativity was not a probable statement until Albert Einstein built up a series of abstractions and deductions over time. By the same logic, a model that reasons over long horizons constructs intermediate abstractions, and the researcher expects that this will eventually lead to new scientific knowledge beyond the training distribution. He acknowledges that models have not yet produced a standalone world-changing breakthrough, while noting that they are already solving previously unsolved human math problems. This is a forecast, and whether it holds remains open.

What this means for readers

The framework above gives a practical way to interpret model differences:

Stage What it adds Symptom when it is weak or missing
Pre-training Broad knowledge of text and the world Shallow or incorrect factual knowledge
Supervised fine-tuning Assistant persona, conversation format, stopping Rambling, role confusion, no clean endings
Reinforcement learning (GRPO) Multi-step reasoning on verifiable tasks Instant wrong guesses on math and multi-step problems

A few actions follow. If you want to understand the stages for yourself, load the base and instruct versions of a small open model like Qwen2-0.5B in a free Google Colab notebook and compare their outputs on the same prompts; the cost is zero. If you use instruction models for multi-step tasks, give them room to think, since an answer produced in one token is often a guess. If you fine-tune models, prefer curated conversational data over raw corpora, and consider on-policy methods with an automated verifier when the goal is reasoning. And if you are learning the field, the path described here favors strong intuitions and hands-on experiments over mastering the calculus first; the frameworks handle the derivatives, and the field remains open to self-taught practitioners.