Scaling Laws: Using Power Laws to Plan Parameters, Data, and Compute

If you increase model scale and feed more training data, validation loss will keep decreasing; yet no matter how far this curve is extended, it stays above some positive number and does not automatically go to zero. Scaling laws (scaling law) characterize the quantitative relationship behind this phenomenon: near a fixed model family, data distribution, and training recipe, by what rule roughly does the loss shrink each time resources are scaled up a notch? It is not a constant derived from physical laws, but a statistical fit to a set of training experiments—placing measurement results at different scales on an empirical curve and summarizing its trend with a few parameters.

The form used for the fit is

L(x) = L∞ + A·x⁻ᵅ

Here, x represents scale, which can be parameter count N, training token count D, or compute C; L(x) is the target loss measured at that scale. L∞ is the lowest limit of this curve: under the current setting of task, data, and model family, the part of the loss that is hard to eliminate just by continuing to increase x; it marks the boundary between “can still be reduced” and “temporarily cannot be reduced further”. A determines the size of the reducible part at the reference scale, and α is a positive scaling exponent. Since x is in the negative exponent, A·x⁻ᵅ decreases monotonically as x increases; but note the rhythm of the decrease: each time you multiply the scale by the same factor, what is removed is only a fixed proportion of the remaining reducible loss, not a fixed amount. The loss therefore always approaches L∞ but never reaches it in one step.

For this empirical curve, the inputs are a series of scale points and their validation results under the same conditions, and the outputs are the fitted parameters, as well as predictions and corresponding uncertainties near the measured scales. Before using it, you must confirm what the vertical axis actually is: training loss, validation cross-entropy, or downstream capability score. Different metrics cannot be mixed on the same curve—training loss can be lowered to much lower than validation loss, and capability scores need not even be monotonic, so plotting them in the same fit will only lead to misleading conclusions.

The same set of measurement points forms a slowly downward-curving decreasing curve on ordinary graph paper, but once both the horizontal and vertical axes are changed to logarithmic scales, they approximate a straight line. This change comes from two properties of logarithms: multiplication becomes addition under the logarithm, and exponentiation “drags” the exponent to the front as a coefficient. The power-law form of scaling laws is exactly straightened out under this set of rules.

The specific procedure has two steps. First subtract the lower bound L∞ from the total loss, keeping only the reducible part; then take the logarithm of both sides of the equation with the same base. From L(x) = L∞ + A·x⁻ᵅ, we obtain L(x) − L∞ = A·x⁻ᵅ, and after taking logarithms:

log(L − L∞) = log A − α·log x

Treat log x as the horizontal-axis coordinate u, and log(L − L∞) as the vertical-axis coordinate v; the above equation is then v = log A − α·u: a straight line with slope −α and intercept log A. As long as both axes use the same logarithmic base, the conclusion about the slope remains unchanged—changing the base is equivalent to multiplying both coordinates by a constant, so the line is merely scaled overall, and its steepness remains −α. Figure 1 compares the same power law in two coordinate systems: in ordinary coordinates it is a decreasing curve, in double-logarithmic coordinates it is a straight line, and only after subtracting L∞ does the linear relationship on log-log strictly correspond to the formula. Therefore the slope of the straight line on a log-log plot directly gives the scaling exponent, and the intercept gives the logarithm of the constant A.

The value of L∞ is critical to this way of reading the plot. If the lower bound is estimated inaccurately, what the vertical axis measures is no longer the true reducible loss, and the points in the plot will deviate from a straight line and show curvature, so the exponent estimated from them will also be distorted. Conversely, in practice the lower bound is often estimated by adjusting L∞ until the points line up as straight as possible.

The inputs to this plot-reading procedure are multiple different scales x and their corresponding loss measurements; the outputs include the slope of the straight line, the intercept, the estimate of the lower bound, and the confidence interval of the fit. It is important to be cautious that the point sequence is only approximately straight within the limited interval that was measured; outside this interval, linear extrapolation is no longer guaranteed.

First plug in a concrete number to get a feel for the size of this exponent: take α = 0.10, and increase compute tenfold; how much reducible loss remains? The power law gives the answer:

R₂/R₁ = (10C/C)⁻⁰·¹⁰ = 10⁻⁰·¹⁰ ≈ 0.794

Here R = L − L∞ denotes reducible loss, R₁ and R₂ are the reducible parts remaining before and after increasing compute, C is the original compute amount, and the exponent −0.10 comes from the assumed scaling exponent α = 0.10. The ratio 0.794 means: after the increase, the remaining reducible loss is 79.4% of the original, so this round cuts only 20.6%. A tenfold increase in resources does not make the loss one-tenth—the power law's "discount" is far stingier than intuition would expect.

Now consider an example that writes out the lower bound explicitly. Let L∞ = 1.50 and the current loss L = 2.00; then the reducible part is R = 2.00 − 1.50 = 0.50. The predicted loss after tenfold compute is

L = 1.50 + 0.50 × 0.794 ≈ 1.897

rather than the 1.59 obtained by multiplying 0.794 directly by the total loss—that would assume that even L∞ shrinks according to the power law, while L∞ is exactly the part that the power law cannot reach. If you scale up another tenfold, it only multiplies the reducible part remaining at that time by 0.794:

The ratio corresponding to 100 times the compute is 0.794² ≈ 0.631, and the predicted loss is 1.50 + 0.50 × 0.631 ≈ 1.816. Diminishing returns are very clear here: the first tenfold step reduces loss by about 0.103, while the second tenfold step reduces it by only about 0.081, but each step nibbles away at the remaining reducible loss by the same relative proportion.

These numbers hold only under two conditions: the estimates of α and L∞ are reliable, and the scaled-up size still falls within the range used for fitting. They are extrapolations based on measured curves, not guarantees for any particular actual training result.

When faced with a fixed training compute budget, parameter count N and training token count D become a pair of mutually squeezing variables: the budget is only enough to perform that many multiplication operations; giving more to parameters means less is left for data. For dense Transformers, the forward and backward computation for processing a single token is roughly proportional to the parameter count, so the total floating-point operations for the entire pre-training are often roughly written as C ≈ 6 × N × D. Here C represents the number of floating-point operations executed during the entire training process, commonly known as FLOPs; N is the number of model parameters; D is the total number of tokens processed during training. The coefficient 6 is a rough approximation of the forward plus backward cost for common dense Transformers, not a fixed constant that holds for all architectures.

This approximation directly gives an intuitive conclusion: when the budget C is fixed, doubling N roughly halves D; increasing N by a factor of four leaves D at only one-fourth of its original value. The benefit of making the model larger is not free—it simultaneously takes away tokens that the data side could otherwise have seen more of. Degradation in the two directions has its own manifestations. When there are too few parameters, the model struggles to express the patterns in the data no matter how much data it sees; this is a capacity bottleneck. When there are too many parameters but only a small number of tokens have been seen, the parameters themselves have not been sufficiently trained, and convergence quality actually decreases; this is under-training. The optimal point under the same budget lies on the empirical balance between the two types of bottlenecks, rather than simply maximizing parameters.

A comparison table at the same budget makes this clearer. Option A chooses a smaller model, with parameters counted as 1× and tokens as 8×; the typical risk is a capacity bottleneck. Option B takes the balance point, with parameters 2× and tokens 4×; whether it is truly optimal requires experimental confirmation. Option C chooses a larger model, with parameters 4× and tokens 2×; the typical risks are insufficient data and expensive inference. The three options consume the same total amount of floating-point operations; the differences in loss come entirely from different allocations of N and D.

Therefore, the input to this type of experiment is a compute budget and several candidate combinations of N and D, and the output is the experimental frontier under equal-compute conditions—all points on the frontier correspond to the same C, and the level of loss reveals which allocation is more cost-effective under a fixed budget. It should be noted that this estimate only characterizes the amount of floating-point computation. Attention sequence length, sparse structure, gradient recomputation, and hardware utilization will all change the real wall-clock cost; identical FLOPs guarantee neither identical training time nor identical final quality.

Why did early conclusions lean toward making models larger, while later research in turn emphasized feeding more data? The two sets of conclusions are not as diametrically opposed as slogans suggest; the experimental settings behind them were different from the start: the range of model scales, the coverage of training points, the fitting methods, and their respective constraints all differed. Kaplan et al.'s experiments supported smooth power laws and accordingly gave a budget allocation that leaned toward increasing parameters; Chinchilla, by contrast, collected more scale points and performed joint fitting, pointing out that many large models were actually in a state of insufficient training tokens, and that under a fixed computational budget, parameters and data should be increased together. The core of the disagreement is not which side was wrong, but under what conditions each frontier was measured.

Behind this lies a more practical lesson: there is no universal constant for how many tokens per parameter. The truly transferable approach is to use the current architecture, data, and optimizer to first run a set of small-scale equal-compute experiments, fit your own family's loss frontier, and then allocate budget according to that frontier. Taking the allocation ratio fitted in one paper and applying it unchanged to another model family often yields not a shortcut but an illusion.

Also beware of drift in empirical exponents. If the model family changes, the tokenizer changes, the data filtering strategy changes, or the context length changes, the exponents in the power laws may shift accordingly, and you must re-measure them. Directly copying exponents across papers may be more dangerous than not fitting at all.

One trillion duplicated, contaminated, or low-quality tokens are by no means equal to one trillion tokens of new information. D in scaling laws is only a nominal count; what actually drives loss reduction is the amount of new information carried in the data. Every time repeated content appears once more, the marginal information decays by one unit; erroneous content may teach the model incorrect patterns; benchmark contamination will fabricate false capability scores. All three cases remind us: between token quantity and effective data volume lie data quality and composition.

This is where the value of deduplication and quality filtering lies. Deduplication can reduce memorization burden and compute waste, but one must watch whether rare samples are mistakenly deleted; quality filtering improves the average signal, yet may shift domain or language coverage. Filtering is not a pure benefit; narrowing coverage may also amplify bias for certain languages or groups. Once the data mixture ratio changes, the effective data volume corresponding to the same D is no longer the same, and this is a hidden variable easily overlooked when comparing losses across experiments.

The same limited data can also be squeezed by repeating multiple epochs, but diminishing returns are obvious, and after multiple epochs overfitting and rote memorization become the main risks. Synthetic data attempts to supplement the target skills, provided the generation quality is high enough; otherwise errors will flow back across generations, even inducing model collapse. Every step of data engineering brings both possible benefits and side effects that must be monitored; the two always come packaged together.

Loss curves can decline more and more smoothly day by day, while a capability score on a leaderboard may suddenly jump up a step at some point. Seeing this kind of jump, it is easy to conclude that “a brand-new mechanism suddenly emerged inside the model,” but the measurement method itself may create steps. Cross-entropy is a continuous average quantity; scores such as accuracy, pass@1, or “all steps correct” have all-or-nothing thresholds. Suppose the model’s probability for the correct answer rises from 0.45 to 0.55: the loss improves continuously over this interval, while argmax accuracy flips from wrong to right at the point where the probability crosses 0.5. Sample mixing also brings a similar step effect.

Conversely, sustained improvement in the average loss does not guarantee that factuality, safety, or rare capabilities improve in tandem; changes in these dimensions may not be guaranteed by the decrease in cross-entropy alone. Therefore, evaluation should plot continuous metrics, discrete metrics, and grouped results together, and increase the density of scale points near the threshold to observe whether the jump comes from the measurement method or a real mechanism. So-called emergence may be a real mechanism change inside the model, or it may merely be a product of discrete measurement; a discrete curve alone cannot distinguish the two.

Budget allocation that focuses only on training loss often ends up paying back double during deployment. Training a model with fewer parameters may make the pre-training bill higher, but it can still lower total cost, because inference cost is recurring: when serving massive numbers of requests, the memory, latency, and energy consumption of each inference recur; the more requests, the more this cost is magnified. Conversely, for one-off research models with very low inference volume, the frontier training loss level is often more worth optimizing than inference cost.

Express this as an equation: K_total = K_train + N_requests × K_infer + K_ops. K_total is the total cost over the planning period; K_train is the one-time training and experimentation cost; N_requests is the expected number of requests; K_infer includes the hardware, energy, and latency cost of each served request; K_ops covers deployment, monitoring, rollback, and error handling. This equation is meaningful only when all terms use the same time horizon and quality threshold; otherwise different candidates cannot be compared fairly.

Thus the inputs to this kind of decision are not just training candidates, but also request scale, context length, concurrency, quantization scheme, hardware conditions, and quality constraints; the output is a lifecycle cost frontier that satisfies the quality condition. Only by comparing along the frontier will one discover that a strategy that only optimizes training FLOPs essentially shifts cost to the deployment stage.

A straight line fitted from small models, once extended to unfamiliar scales, may quietly break at some point at any time. Breakpoints appear in different ways, and identifying them requires checking symptoms one by one. When the architecture or training recipe changes, newly measured points will depart from the old line; at this point you should fit piecewise and redo ablation experiments to confirm which change caused the shift. When data is exhausted or the distribution changes, gains will saturate earlier than expected; the response is to truthfully report the data mixture proportions and effective token count. Optimization instability can make large-scale points perform abnormally worse; in this case prioritize investigating learning rate, parallelism scheme, and numerical precision issues. When metrics hit ceiling or threshold effects occur, loss and task score become decoupled; you should switch to continuous metrics and do per-slice analysis on samples. If the prediction points are far beyond the fitting interval, the confidence interval will expand rapidly; adding intermediate-scale anchor points can constrain uncertainty.

Putting together the sources of breakpoints, their manifestations, and responses reveals that they all point to the same discipline: extrapolation is reliable only within the range of 'similar systems, nearby scales.' Scaling laws are a map, not a guarantee. They can predict approximate trends for similar systems at adjacent scales, but cannot prove safety, reliability, or controllability, and can never replace real evaluation on the target task.

From why loss can be reduced to how a budget should actually be spent, there lies a causal chain that can be verified link by link. The first step is to fix the experimental conditions: choose the same model family and a stable training recipe, excluding the interference introduced by the architecture and the optimizer. The second step is to actually train at multiple combinations of N, D, and C, giving the power-law fit enough observational support. Only the third step is fitting: estimate the three parameters L∞, A, and α—the irreducible loss lower bound, the power-law coefficient, and the scaling exponent—which respectively determine the curve’s lowest point, overall position, and rate of decline.

Once fitting is complete, under a fixed budget, find the N–D allocation frontier along the curve, answering “How should this compute be allocated to minimize loss?” But the frontier is only an interpolation result; you must use intermediate-scale points to verify whether the extrapolation still holds, preventing the straight line from being extended into regions where it no longer holds. The final acceptance does not look at a single number; instead, it uses task slices and full-lifecycle cost as criteria, confirming that the loss reduction actually translates into an improvement in the target capability and that costs are not quietly shifted to the deployment stage. If any link in the chain breaks, the earlier fit loses its meaning; this is also why the entire process emphasizes that experiment, fitting, and validation must use the same set of conditions.