
Tech • AI • Robotics • Game
⏱ ≈ 4 min read · 47-min video — 43 min saved
Pre-training teaches large language models to predict the next token from vast unlabeled text, using cross-entropy loss, backpropagation, and massive distributed compute to turn raw internet-scale data into a base model.
Pre-training is the most expensive stage in building modern systems such as ChatGPT, Claude, and Gemini. A model starts with random parameters and learns general patterns in language, code, facts, formatting, and style by repeating one task trillions of times: predict the next token. The result is a base model, closer to a powerful autocomplete than a finished assistant.
Unlike classic supervised learning, pre-training does not require humans to annotate examples. In a sentence such as “the capital of France is Paris,” the target token is already present in the text, so the data effectively labels itself. This self-supervised setup is what lets training scale to internet-sized corpora.
After tokens are embedded and passed through transformer layers, the model outputs logits for every token in its vocabulary, often 50,000 to 128,000 options. A softmax converts those logits into probabilities. If the model assigns 64% probability to Paris as the next token, that may be decent, but training still needs a precise measure of error.
The standard pre-training objective is cross-entropy loss, defined as the negative log of the probability assigned to the correct token. A correct token at 90% probability gives a loss near 0.11; at 1%, loss rises to about 4.6. Engineers also track perplexity, which is simply the exponential of the loss and can be read as the model’s effective number of choices. At random initialization over a 50,000-token vocabulary, loss should start near 10.8 and perplexity near 50,000, a basic sanity check that the run is configured correctly.
The gradient tells each parameter which direction reduces loss. For language models, combining softmax with cross-entropy yields a simple signal: predicted probability minus correct probability. That pushes the correct answer up and incorrect competitors down, with larger corrections for tokens that stole more probability mass.
Real models have billions of weights, so updating outputs directly is impossible. Backpropagation applies the chain rule through every operation in reverse, tracing how each weight affected the final loss. Numerical gradient checking by nudging each parameter separately would require billions of extra forward passes and is infeasible at scale; one forward pass plus one backward pass computes all gradients efficiently.
Frameworks such as PyTorch and JAX automate differentiation by building a computational graph during the forward pass and replaying it backward. This convenience comes with a large memory bill because intermediate activations must be stored until the backward pass. In large transformers, activations can consume more memory than the model weights.
Modern large models are usually trained with AdamW, which combines momentum, per-parameter adaptive step sizes, and weight decay. Learning rates are not fixed: they typically use a short warm-up from near zero, then a long cosine decay. Llama 2 used about 2,000 warm-up steps, with peak learning rates around 3×10⁻⁴ for smaller models and 1.5×10⁻⁴ for larger ones. Batch sizes are enormous, often millions of tokens, because larger batches reduce gradient noise and better utilize GPUs.
Research showed that final loss follows smooth scaling laws as model size, data, and compute grow. A common training compute estimate is roughly 6 × parameters × training tokens. For GPT-3, with 175 billion parameters and roughly 300 billion tokens, that implied about 3×10²³ FLOPs. Later DeepMind work on Chinchilla found many earlier models were undertrained: 70 billion parameters on 1.4 trillion tokens outperformed Gopher at 280 billion parameters trained on 300 billion tokens, popularizing a rule of about 20 tokens per parameter for compute-optimal training.
Training memory far exceeds inference memory. A 7 billion parameter model can require roughly 112 GB just for weights, gradients, and Adam optimizer state, before counting activations, while an Nvidia H100 has 80 GB of memory. That forces training across many GPUs using data parallelism, sharding such as FSDP or ZeRO, tensor parallelism, and pipeline parallelism. Meta trained the 405 billion parameter Llama 3 model on up to 16,000 H100 GPUs, with fast interconnects essential because communication can become as important as computation.
Pre-training is a mathematically simple but industrially immense process: predict the next token, measure error, compute gradients, and update billions of weights across vast clusters. The main constraints are no longer just algorithms, but data quality, memory, networking, and the economics of compute at scale.
Ask a question