Pre-training

Directly training a model for a specific task looks like the shortest path on the surface, but in reality it incurs two high costs. The first is data: task-specific training requires manually labeled data, which is expensive to annotate and limited in quantity. The second is capability narrowing: parameters trained this way learn only the patterns of that small task, and switching to another task requires starting nearly from scratch. Pre-training is a two-stage approach designed to bypass these two costs: first train a 'general foundation' on massive general-purpose data—letting the model first learn how language is used and roughly what the world is like; then make small task-specific adjustments. That is exactly the meaning of the word 'pre': before formally facing a specific task, first build a solid general foundation.

To state the whole task completely: Pre-training takes two inputs—large-scale general-purpose data and a model that has not yet learned the patterns of any task; the output is a parameter checkpoint, which can be used directly for subsequent fine-tuning or directly for representation extraction. The mechanism loop during training is straightforward: the model repeatedly constructs a prediction target from the data itself, computes the loss, and then updates parameters based on the loss. What deserves attention here is the step of 'constructing the target from the data itself'—general-purpose data itself carries no human-labeled task answers; the target is automatically generated from the raw data. This is also the key that allows pre-training to use massive amounts of unlabeled data.

At the other end of the causal chain is the evaluation criterion. A decrease in training loss only shows that the model has memorized the training data; it cannot prove that the 'general foundation' has formed. What really matters is the loss on held-out data that the model never saw during training, as well as performance on multiple capability evaluations. Only when the model can also make stable predictions on unseen data and shows improvement across multiple capabilities can we say that this general foundation has been established.

This route has clear applicability boundaries. When the target task heavily overlaps with the patterns contained in the pre-training data, the broad patterns accumulated during pre-training can be directly reused, yielding the greatest benefit. But if the target domain differs greatly from the pre-training data, the general foundation does not apply directly. In that case, domain adaptation is needed, and one should even consider switching to a domain-specific model for that field rather than forcing general parameters onto it.

To summarize this causal chain in one sentence: first become a generalist, then become a specialist. Just as a person first receives many years of general education and then undergoes job-specific pre-employment training—the general education step is the most expensive and time-consuming, but as long as it is done well once, all kinds of specialization can be picked up quickly afterwards. Pre-training's role in the entire process is exactly the same: it is the most costly and longest-running stage, and it is also the prerequisite that allows all subsequent specialized capabilities to be acquired cheaply.

This “general foundation” is not instilled through human explanations; rather, supervisory signals are constructed from the data itself, allowing the model to learn on its own. This is the most essential difference between pre-training and task-specific training: no per-sample human labels are needed; the raw data itself is the problem, and part of the content carried by the data itself is the answer.

There are three mainstream ways to construct signals, each corresponding to different input construction methods, model output forms, and applicable directions. Autoregressive models perform next-token prediction: given the already-appeared left-side text as a prefix, the model outputs a probability distribution over the entire vocabulary for the next token; this type of objective is suitable for training autoregressive generative models. BERT-type encoders perform masked prediction: some tokens in the sequence are masked, and the model outputs vocabulary probabilities corresponding to these masked positions; this type of objective is suitable for training bidirectional encoding representations. Multimodal systems commonly use contrastive or reconstruction objectives: the input is paired views (such as image-text pairs) or corrupted samples, and the model outputs matching scores or reconstructed samples; this type of objective is suitable for training visual and multimodal representations. All three objectives answer the same question—how to let raw data provide the learning target on its own—but they are different algorithms and cannot be interchanged arbitrarily: the objective determines what the model can see and output at each step, and also determines whether the final model is better at generation, encoding, or cross-modal matching.

Take autoregressive training as an example to see one complete parameter update loop: the model performs computation on the prefix and obtains a predictive distribution for the next token; cross-entropy measures the gap between this predictive distribution and the actual next token—the larger the gap, the higher the loss; then gradient descent is performed on this loss to update the parameters in the direction that makes the prediction closer to the actual token. Repeating this loop over massive amounts of text, the model gradually learns the statistical regularities of text sequences.

The freedom brought by “reducing human labels” is not unlimited; its ceiling is constrained by data quality. High-quality data itself is a scarce resource: acquisition licenses, cleaning, deduplication, and the balance of languages and domains—each link directly affects the final model's capability distribution, bias tendencies, and memorization risk. If the data is poorly selected, no matter how many self-supervised signals there are, a good general foundation cannot be learned.

The validation stage must likewise respect consistency between objectives and data. You must use held-out data that matches the training objective to evaluate loss and performance, not use training loss as a substitute; at the same time, you should additionally check for data leakage, duplicate samples, and whether there is a mismatch between training objectives and downstream tasks—if the objective trains for generation but the downstream task requires encoding, this mismatch will directly weaken the validity of the evaluation conclusions.

Compress the whole matter into one sentence: pre-training means repeatedly performing the single self-supervised task of “predicting the next token” on as much text as possible. Simple to the point of being counterintuitive, yet it is the foundation of the entire edifice.

The loop from the previous section can be worked through by hand on a single sentence. Take the text “猫 喜欢 鱼 <EOS>” (<EOS> is the end-of-sequence token). A training sample is not this sentence as a whole; instead, the sequence is shifted one token to the right, offset position by position: each prefix becomes an independent sample, and the sample’s correct answer is the token immediately after it. Thus this sentence produces three training positions: seeing the prefix “猫” requires predicting “喜欢”, seeing the prefix “猫 喜欢” requires predicting “鱼”, and seeing the prefix “猫 喜欢 鱼” requires predicting <EOS>.

Suppose the model assigns the correct token probabilities 0.50, 0.25, and 0.80 at the three positions, respectively. The loss at each position is expressed as the negative log probability: −ln p. The first position is −ln 0.50 ≈ 0.693, the second is −ln 0.25 ≈ 1.386, and the third is −ln 0.80 ≈ 0.223. The intuition here is that the lower the probability, the larger the negative log and the heavier the loss; the position where the model is least certain contributes the largest loss.

Averaging the losses at the three positions gives the average cross-entropy of the whole sentence: L̄ = (0.693 + 1.386 + 0.223) ÷ 3 ≈ 0.767. In notation, L̄ represents the average cross-entropy over the three prediction positions, t is the position index, pₜ is the probability the model assigns to the correct token at position t, and ln is the natural logarithm. Then applying the exponential transform to the average cross-entropy: PPL = exp(L̄) = e^0.767 ≈ 2.15. Here exp is the exponential function with the natural constant e as its base; it converts the average negative log loss back to a probability scale—2.15 can be understood as follows: on the evaluation data, when the model predicts the next token at each step on average, it feels as if it is choosing among about 2.15 equally likely candidates. On the same tokenizer and the same data, a smaller PPL usually means the model is better at predicting subsequent tokens.

After the loss is computed, how do the parameters change? Backpropagation simultaneously adjusts all shared parameters in the model so that, under similar contexts, the logit of the correct token rises relative to the others. Returning to the second position: if after one update the model’s probability for “鱼” rises from 0.25 to 0.50, the loss at that position drops directly from −ln 0.25 = 1.386 to −ln 0.50 = 0.693. An improvement at one position immediately shows up as a decrease in the whole-sentence average loss; this is the smallest unit at which gradient descent operates on text sequences.

The real value of this example is that it reveals the density of the training signal. Training does not provide just one label per passage of text; rather, a sequence of length L usually contributes about L next-token supervision positions. A ten-token sentence can be cut into about ten training samples. Massive raw text multiplied by this density automatically produces a training signal so dense that it needs no manual annotation—this is the arithmetic foundation that makes pre-training viable at the data level.

Finally, the boundaries of this metric must be made clear. Low perplexity does not mean the facts are more true, nor does it mean the assistant is more useful. It only means the model is better at predicting tokens on the evaluation-text distribution. Data contamination, rote memorization, learning only stylistic shortcuts, or a mismatch between the training objective and the real task can all cause PPL to diverge from real capability—the metric declines, but capability does not necessarily improve in step.

The most critical question is: by merely predicting the next token, how does it learn grammar, common sense, and even reasoning? The answer lies in the strength of the constraints that the prediction task imposes on representations. The model faces a massive number of samples across topics, with infinitely many prefixes and infinitely many combinations. To reduce prediction error simultaneously across large amounts of different text, the model cannot rely only on rote memorization—memorizing just one sentence cannot explain why infinitely many new, never-seen prefixes can also be predicted accurately. The only way to keep the loss decreasing is to form reusable internal structures: grammatical regularities, semantic associations, code structure, and co-occurrence patterns among facts. Thus the direct output of training is lower loss and an updated set of internal representations; translation, code, or reasoning performance are external manifestations that emerge when these representations are invoked by prompts and evaluation tasks.

Here it is necessary to separate 'being able to continue reasoning text' from 'reliable reasoning'. The model may actually reuse abstract regularities, or it may only memorize templates and exploit surface cues. The criterion depends on the strength of generalization evidence: if it still answers correctly and stably on newly composed problems, counterfactual problems, and out-of-distribution samples, then it has mastered transferable regularities; if it succeeds only on familiar formats, one cannot claim it has learned a general algorithm.

The position of scale in this causal chain is equally worth clarifying. Increasing the number of parameters, the amount of data, and compute usually smoothly reduces prediction loss, and may allow certain capabilities to cross the threshold of 'usable'—such empirical relationships are called scaling laws. But scaling laws describe trends, not guarantees. Scale is not a sufficient condition: data quality, the design of the training objective, model architecture, and evaluation criteria—any change in any link will alter the final result. 'Bigger' does not automatically mean facts are more reliable, nor does it automatically mean fairer for everyone. Scale provides an upper bound on potential; turning potential into capabilities still depends on every earlier design choice.

Once pre-training is finished, do you have an assistant you can use directly? Not yet. What you get at the end of this stage is a base model checkpoint: knowledgeable, but not obedient. Ask it a question, and it may follow the inertia of “how text usually continues” and continue generating more questions instead of giving an answer. The reason lies in the training objective itself—it has only learned the probabilistic patterns of text sequences, and has never learned the behavioral norm of “when asked a question, you should answer.”

To turn the base model into an assistant that can converse, two more training steps are needed: instruction fine-tuning teaches it to “answer when asked,” and preference alignment teaches it to answer both usefully and safely (see the deep-dive pages on “Fine-tuning” and “Alignment”). So the boundary of this stage becomes clear: the input is the pre-training data stream, and the output is the base model checkpoint and its training records; “being able to continue text and having low held-out loss” shows that the language modeling objective has made progress, but it does not show that the model already follows instructions.

From massive amounts of text to a base model and then to an assistant, the division of labor along this chain can be summarized in two sentences. Pre-training is responsible for pouring in “knowledge and abilities,” producing a knowledgeable but not obedient base model; fine-tuning and alignment are responsible for calibrating “behavior,” turning the base model into an everyday assistant. All three steps are indispensable. To compress it one step further: pre-training determines “what it can do,” while fine-tuning and alignment determine “how it behaves”—most of the model’s knowledge and abilities are poured in during the pre-training step.

One more boundary must be maintained: subsequent training can change the model’s answering habits and preferences, but it cannot guarantee factual correctness. No matter how much knowledge the base model has, it is still probabilistic statistical knowledge, not a verified fact base. High-risk uses must rely on independent evaluation, retrieval, or tool verification, and cannot skip checking just because “this model was pre-trained on a lot of data.”

Pre-training is critical because it is the primary source of capability; it is expensive because instilling that capability requires simultaneously burning through four types of resources. Pre-training frontier general-purpose models from scratch consumes large-scale data, accelerator clusters, and long-term engineering effort, and the cost is usually affordable only by a few organizations—but note that 'small-model pre-training' is not absolutely infeasible; what is expensive is the frontier-scale tier, not pre-training itself.

The four cost sources have different growth logics. Compute and memory: the more parameters and tokens there are, the more forward and backward computation is required, which directly determines training duration and accelerator cluster size. Communication: multiple devices must continuously synchronize gradients, parameters, or activations; once the network becomes a bottleneck, cluster utilization is dragged down, and machines are waiting for data instead of computing. Data engineering: collection, licensing, filtering, deduplication, and mixing ratios require repeated iteration; it does not directly show up on the compute bill but determines the model's contamination level, bias tendencies, and long-tail coverage quality. Failures and validation: hardware faults, numerical anomalies, and bad batches can interrupt training runs that often last months, so checkpoints, monitoring, and recovery systems must be in place; otherwise a single interruption can lose a large amount of completed training progress.

When doing budget planning, the inputs are target model size, number of training tokens, hardware efficiency, and data plan, and the outputs are estimated compute, time, cost, and a recoverable checkpoint plan. One budget line is most easily overlooked: the total training budget must be allocated rationally between parameter count and token count. Work such as Chinchilla shows that increasing parameters alone while training data is insufficient is not compute-optimal—a model that is large enough but has not seen enough data amounts to spending compute on capacity that is not fed enough.

Reading the state of the training process also requires distinguishing cause and effect. Actual throughput below theoretical peak does not necessarily indicate a model design error; it may be a bottleneck in communication or the data pipeline. Conversely, training running very fast does not mean data and capability quality meet the bar. Speed only reflects engineering efficiency, not that the model is learning well.

Since pre-training from scratch is this expensive, most people will not redo it themselves; instead they fine-tune on top of someone else's pre-trained model—this is transfer learning: at very small cost, you borrow that extremely expensive general-purpose foundation (see the 'Fine-tuning' deep-dive page). Pre-training and fine-tuning therefore form a division of labor: one expensive generalist development, paired with countless cheap specialist adaptations. In a specific scenario, which path to choose must be weighed by data volume, task difference, and budget: with little data and the task close to the pre-training domain, direct fine-tuning is most cost-effective; with a lot of data and a specialized domain, a small-scale domain model or continued pre-training is worth considering.

String together the previous links, and the entire causal chain of pre-training is coherent. The starting point is an economic judgment: training directly for a task is both expensive and narrow, so instead learn a general foundation first and then adapt to the task—this is how the two-stage "pre"-training arrangement comes into being.

How is a general foundation learned? By having the data itself construct supervision signals. Autoregressive models predict the next token, and other architectures can use masked, contrastive, or reconstruction objectives; in any case, learning signals are obtained from large-scale unlabeled data without relying on manual annotation. The signal falls on every position: each position contributes loss as −ln p, and shifting a text one position to the right automatically produces dense supervision positions; gradient descent then makes the relative probability of the correct token rise step by step. The micro-level arithmetic accumulates at the macro level into a constraint: the prediction objective forces the model to compress out a large amount of reusable structure—syntax, semantics, code structure, and factual co-occurrence. But this step does not promise that results automatically improve; final capability is jointly determined by data, scale, and task distribution.

What the end of the chain produces is a base model: knowledgeable but not obedient, because the language modeling objective never taught it to "answer when asked." To become an assistant, two further steps are needed: fine-tuning and alignment. The cost structure of the whole chain—the multiple drains of data, compute, communication, and engineering time—explains why most teams do not train from scratch but instead reuse an existing base model and then adapt it.

To grasp the core, you can self-check with two questions: Can you explain clearly why pre-training can learn so much without annotations? Can you say which two steps separate a base model from a conversational assistant? Only when you can answer both questions have you truly got this chain in your hands.

Before entering pre-training, there are four prerequisite foundations that need to be established first: supervised learning—it provides the basic loop of 'constructing a loss from data and updating parameters with gradients'; self-supervised learning—it explains how to create supervision signals from the data itself, which is the core mechanism of pre-training; large language models and Transformer—they determine the model's structure, capacity, and sequence modeling approach. With these, the four core concepts on the pre-training page can take root: the overall two-stage (pre-training → fine-tuning) route, the specific objective of self-supervised next-word prediction, the intermediate product known as the base model, and the division of capabilities between generalists and specialists.

The immediately adjacent extension concepts unfold along both sides of the chain. Going toward 'after training' leads to fine-tuning, instruction fine-tuning, and alignment: it is here that the base model is shaped into an obedient, useful assistant. Going toward 'training scale' leads to scaling laws: they describe the empirical relationship between parameters, data, compute, and loss, and are the key to understanding why pre-training requires such large-scale investment. Further extensions touch on large-scale application and engineering realities: synthetic data supplements the scarcity of high-quality data, distillation compresses the capabilities of large models into smaller models, and quantization and deployment determine at what cost the trained model ultimately reaches users.