Loss Function: Turning ‘Where It Went Wrong’ into a Learnable Direction
From business cost to a differentiable proxy, from single-sample error to empirical risk; using the same numerical example, see how the loss, gradient, and parameter update connect into a closed loop.
- Why does training need a scalar loss, rather than just saying “the answer is bad”?
- What error assumptions do MSE, MAE, Huber, and cross-entropy respectively imply?
- Why can't true objectives such as accuracy and user satisfaction usually be directly backpropagated?
- How do you manually compute the loss, gradient, and one parameter update from a single sample?
- Why can decreasing training loss still make the product worse, and how should you detect this?
ŷ=wx to learn the relationship y=3x. Here w is the “multiplier” parameter that the model needs to learn: the correct relationship requires w=3, but training starts from a deliberately wrong initial value w=2 starts. Take the sample x=2,y=6 , the model only predicts ŷ=4; this “difference of 2” will run through the three pages on loss, Gradient Descent, and Backpropagation.1Why “being wrong” must first become a numberIntuition
A model has thousands of parameters; how does a phrase like “this answer wasn’t good” determine which way each parameter should move?
Ordinary gradient training ultimately needs a total objective that can be compared and differentiated. Loss function L(ŷ,y) receives the prediction and the target and outputs a number: the smaller it is, the better it usually is according to the rules we have written. The slope of this number connects “how good the result is” to parameter changes, solving the problem that natural-language evaluations cannot directly guide updates to many parameters; without it, the optimizer neither knows which of two versions is better nor knows the impact of a small modification.
During computation, details need not be discarded too early: the system can first keep the loss for each sample, group, or task separately, and finally form a scalar for updating through averaging, weighting, or multi-objective rules. The optimizer adjusts parameters based on the local changes of this aggregated scalar, and how these per-item losses are combined is itself part of the training design.
But compressing into a single number also loses information. Counting a missed diagnosis and a false alarm both as “one mistake” is equivalent to declaring that they have the same cost; averaging a long answer over tokens is equivalent to choosing a length weight. Loss is not a neutral thermometer; it is a value trade-off written into the training loop, so we still need to return to per-sample, per-group, and real task metrics to interpret the results.
2How a Single Error Becomes a Training ObjectiveMath
A single sample has a loss; what should the whole training set optimize?
First look at the i-th sample:xᵢ is the input,yᵢ is the target; model parameters are collectively called θ, so the model prediction is written as fθ(xᵢ). Passing the prediction and target to the loss function gives this sample's loss.
The training set has n samples, the most direct approach is to sum all per-sample losses and divide by n. This step combines the cost of many predictions into a single optimizable objective:
Jdata is the average loss on the training data, also calledempirical risk. The letter J is used to remind us: it is no longer the loss of a single sample, but the objective that the training procedure needs to lower overall.Jdata decreasing only means that the average cost under the current training set and current loss criterion is reduced.
Sometimes we also want the parameters not to be too large or the model not to be too complex, so we add a regularization term:
R(θ) measures parameter characteristics that are undesirable,λ determines how much weight this constraint has:λ=0 means no such constraint is added; the larger λ is, the more importance it is given. The procedure first averages per-sample loss over a batch or the training set, and only then adds the regularization term. Regularization will be formally introduced in a later node; on this page you only need to know that the training objective may include more than prediction error.
The training procedure tries to reduce J; what we really care about is the average loss when the model encounters future samples. There can be a gap between the two; sample bias, incorrect labels, and improper regularization weights can all distort empirical risk; this is the problem to be studied later under "overfitting and generalization".
| Level | What it is | Common misreading |
|---|---|---|
| Per-sample loss | How much one prediction is wrong under the current rule | Treating an outlier sample as overall performance |
| Batch loss | The average objective over the current mini-batch of samples; only after taking its gradient do you get an estimate of the overall gradient. | Mistaking batch fluctuations for training collapse |
| Training empirical risk | Average loss over the entire training set | Equating with future performance |
| Validation/production metric | Checking whether unseen data and real-world scenarios improve | Using it directly to compute gradients without processing |
3Why Regression Losses Have Different ShapesMath
They are all “prediction minus true value”; why do squared, absolute, and Huber learn different models?
Scroll horizontally to view the full diagram on small screens.
Let error e=ŷ−y. MSE, MAE, and Huber convert the difference between the prediction and a continuous target into a non-negative per-sample cost; the difference among the three losses is not their names, but “how quickly the cost grows as the error increases,” that is, how strong a training pressure different-sized errors receive.
| Loss | single-sample form | where it pushes the model | fitting intuition |
|---|---|---|---|
| MSE / squared loss | e² | when expected squared loss is minimized, it predicts the conditional mean | large errors should be corrected more strongly; can also be derived from the negative log-likelihood of fixed-variance Gaussian noise |
| MAE / absolute loss | |e| | when expected absolute loss is minimized, it predicts the conditional median | error is priced linearly by size, more robust to outliers; also corresponds to a Laplace noise model |
| Huber | quadratic for small errors, linear for large errors | a compromise between mean sensitivity and median robustness | hopes that ordinary errors are optimized smoothly, while not allowing extreme errors to be infinitely amplified |
Huber requires a transition threshold δ:
For example, take δ=1: error e=0.5 uses the quadratic part, and the loss is 0.125; error e=4 uses the linear part, and the loss is 3.5, while half squared loss would reach 8. Therefore Huber will correct large errors, but limit how much they dominate training. After computing per-sample costs separately by squared, absolute, or this piecewise rule, aggregate the per-sample costs into the training objective; only under the same data and the same formula does a lower value indicate a better fit.
4Why Cross-Entropy Fits Probabilistic ClassificationMath
When a classification model outputs a set of probabilities, why is it not enough to just look at whether it "guessed correctly"?
A classification model does not just report one class; instead, it first assigns a probability to each class. Denote pθ(y|x) as: the parameter is θ model, upon seeing input x , assigns to the true class y probability. Cross-entropy takes this set of class probabilities and the true class as input and outputs a nonnegative single-sample loss:
At computation time, read the probability corresponding to the true class, take the negative logarithm, and finally average across samples. Here we explicitly use the natural logarithm ln, so the unit is nat; if instead you use log₂, the numerical unit is bit. The logarithm base only uniformly scales the loss and does not change which model probability is better.
| Probability assigned by model to true class | Binary classification result | Cross-entropy −ln(p) | Description |
|---|---|---|---|
| 0.90 | Correct | about 0.105 | The model assigned a relatively high probability to the true class |
| 0.49 | Wrong | about 0.713 | just slightly short of crossing the 0.5 decision boundary |
| 0.01 | Wrong | about 4.605 | The model very confidently excluded the true class |
Accuracy only looks at the final class: the last two rows are both recorded as one error. Cross-entropy also looks at the probability, so it lightly penalizes "hesitant mistakes" and heavily penalizes "confident mistakes," solving the problem that judging only correct or incorrect cannot distinguish the two. Summing the cross-entropy over all training samples gives the negative log-likelihood; minimizing it is equivalent to increasing the joint probability the model assigns to the training data. The smaller the value, the more probability the model on average places on the true class.
Language models predict the true next token at each position and likewise use cross-entropy. The formal definition of token is in 2.8; here, temporarily understand it as the text unit that the model predicts step by step. The exponential of the average cross-entropy is perplexity; it describes how well the predicted distribution fits the text, and does not directly guarantee calibrated probabilities, minority-class recall, factual correctness, instruction following, or better actual decision cost. Label noise, class weights, and distribution shift can also change the meaning of the reading.
5Why Training Loss and Real Objectives Need a Division of LaborDisambiguation
Ultimately, we care about accuracy, conversion rate, or satisfaction; why not directly let the optimizer maximize it?
First, distinguish the two roles.Real objectiveis the result the product ultimately cares about, for example, a diagnostic system reducing missed diagnoses, shopping recommendations helping users complete an appropriate purchase, or an assistant receiving high satisfaction ratings after a task. “Conversion rate” is the proportion of users who complete a purchase after seeing a recommendation, and “satisfaction” may come from ratings, complaints, or follow-ups; they are only examples of product objectives, and not all models need to pursue these metrics. During validation, independent samples, user feedback, and safety records serve as inputs, and the resulting real metrics such as missed diagnosis rate, conversion rate, satisfaction, or accident rate are outputs.
Proxyis a quantity that temporarily replaces the real objective and is directly optimized by the training procedure. The so-calleddifferentiable, means that when the parameters change by a very small amount, this quantity usually changes smoothly as well, allowing a useful local slope to be computed. Therefore, a “differentiable proxy” is a training signal that is not necessarily equal to the final objective, but can be computed frequently, differentiated smoothly, and is expected to remain consistent with the final objective, such as cross-entropy. During training, samples and supervised targets serve as inputs, and proxy losses such as cross-entropy serve as outputs; the training procedure first updates parameters along the proxy's local slope, then uses validation and launch metrics to check whether this update serves the real objective.
Why do we need proxies? Accuracy often does not move at all when parameters change slightly; it jumps only when predictions cross a classification boundary, so there is almost nowhere a useful gradient. Conversion rate and satisfaction may also appear only hours or days later, and a single result is affected by factors outside the model such as price, interface, and inventory. A differentiable proxy provides a timely, continuous training signal; if cross-entropy decreases while missed diagnosis rate, conversion rate, or satisfaction does not improve, it indicates that the proxy has become misaligned with the real objective. Delayed feedback, these confounding factors, and changes in user behavior can all invalidate an originally relevant proxy, so a decrease in the proxy cannot be directly interpreted as the product improving.
| Role | Required properties | Examples |
|---|---|---|
| Training loss | Differentiable, frequent, numerically stable | Cross-entropy, MSE, ranking proxy |
| Offline metric | Interpretable, close to the task | F1, recall, calibration error |
| Product objective | Reflects real benefits and harms | Success rate, human takeover, accident rate |
| Release threshold | Must not be masked by averages | Minimum recall for critical slices, safety red lines |
6Weights, masks, and regularization terms: how they rewrite “what the model cares about”Engineering
How do class imbalance, different tokens, or high-cost errors enter the same scalar?
Weights, masks, and regularization terms are control quantities that reshape the relative importance of each training signal. They take the per-sample loss and corresponding control values, and incorporate class imbalance, invalid positions, and complexity constraints into the same training objective: first remove positions marked invalid by the mask, then weight the retained items and normalize them according to the agreed convention, and finally add the regularization term. Sample weights amplify gradients for certain cases; class weights keep minority classes from being drowned out by majority classes; regularization terms add “parameters should not be too complex” to the objective. Focal loss further reduces the contribution of easy samples, making training focus on hard examples.
These choices also change probability calibration and the effective training distribution. A decrease in the recombined loss only means that the cost computed under this set of importance definitions has become smaller; it does not mean that the risk under natural frequencies has decreased simultaneously. Increasing minority-class recall may increase false positives; weighted probabilities may no longer equal the incidence rate under the natural distribution. Excessive weights, changes in the normalization convention, or mistaken mask deletions can also let a small number of samples dominate training, so you must inspect each component at the same time and re-measure on the unweighted validation distribution and key business slices.
7Complete Worked Calculation: How One Error Drives ParametersNumerical Example
Now turn “the loss provides the training signal” into a sequence of numbers that can be checked item by item.
We use the simplest straight-line model ŷ=wx.x is the input,ŷ is the model prediction,w is the parameter to adjust during training, which can be understood as the slope of the line. The data pattern is y=3x, so ideally we should learn w=3.
At the beginning of training the model does not know the answer yet. We deliberately set the initial parameter to w=2, meaning it currently only predicts “the output is about 2 times the input.” Now give it a sample x=2,y=6, it will predict ŷ=2×2=4. To keep the derivative clean, take the half-squared loss L=½(ŷ−y)², and use a learning rate of 0.1 to make one update step; these inputs will eventually produce a new parameter and a new loss.
| Step | Calculation | Result |
|---|---|---|
| Forward Prediction | ŷ=2×2 | 4 |
| Error | e=4−6 | −2 |
| Loss | L=½×(−2)² | 2 |
| Slope with respect to prediction | ∂L/∂ŷ=e | −2 |
| Gradient with respect to parameter | ∂L/∂w=e·x | −4 |
| Update with learning rate 0.1 | w←2−0.1×(−4) | 2.4 |
| Loss after update | ½(2.4×2−6)² | 0.72 |
The gradients in the table serve first as a preview of the next page: a single-step parameter update converts the local slope of the loss with respect to the parameter into a single parameter move. The slope of the half-squared loss with respect to the prediction is exactly the error e; and w increases a little, the prediction changes by the input x times, so the total slope with respect to w is e×x. The calculation goes through prediction, error, loss, and gradient in order, and then uses the learning rate to obtain w=2.4. The formal derivative, step size, and update direction will be expanded in 1.3 “Gradient Descent”.
w from 2 to 2.4, closer to 3; a single update with one sample and one parameter cannot prove that multi-step training is stable, nor can it prove that the model has learned the complete pattern or can generalize.8How Loss Curves Can Deceive YouFailure Modes
If the training curve keeps falling, what serious problems are completely invisible?
A loss curve plots the loss at each step or epoch during training as a diagnostic view over time; placing validation metrics, group slices, and sub-objective statistics alongside it gives you a better chance of detecting overfitting, long-tail degradation, objective conflict, or proxy misalignment. It takes in a set of process records with clear definitions, and gives only troubleshooting clues, not an automatically issued quality certificate.
| Phenomenon | Hidden Problem | Additional Checks |
|---|---|---|
| Average loss decreases | Minority groups or long-tail tasks worsen | Slice by group, difficulty, and scenario |
| Training loss is very low | Memorizes the training set, validation set worsens | Independent validation and out-of-time testing |
| Proxy metric improves | True objective misaligned or gamed | Human review and online guardrail metrics |
| Different experiments show smaller numbers | Normalization or weight definitions differ | Lock down definitions and report breakdowns |
| Total loss is normal | Some sub-loss has collapsed | Record each objective and gradient contribution |
When reading curves, the first step is not to look for the minimum point, but to confirm that data splits, normalization, weights, and the x-axis definitions are consistent; then compare training and validation trends, and then drill down to groups and sub-objectives. If the average decreases while a key slice worsens, the correct interpretation is that the overall result is still not acceptable; you need to locate the failure hidden by the average, rather than letting training continue to chase the same total.
9How to choose and validate a loss for a taskEngineering
When facing a new task, should selecting a loss start from a list of formulas, or from the cost of errors?
What we need to produce here is not a “formula that is always correct”, but a training and validation plan that can be reviewed from real decision-making all the way to online monitoring. It uses decision cost, output semantics, data distribution, and deployment constraints to answer two connected questions: what differentiable proxy to choose, and why we should believe this proxy still serves the true objective; finally, it leaves behind a clear loss definition, weight specification, and acceptance metrics.
- First, write the real decision:What action the model output will trigger, and which type of error does the most harm.
- Define the output semantics:Is it a numeric value, a probability, a ranking, a sequence, or a multi-objective combination?
- Choose a differentiable proxy:Specify the noise assumptions, weights, mask, and normalization.
- Do a mini hand calculation:Check the correct direction, extreme inputs, and zero-gradient regions.
- Run a baseline experiment:Confirm that it can overfit on small data, and that the training signal is actually connected.
- Independent validation:Report proxy loss, task metrics, calibration, and key slices at the same time.
- Monitor misalignment:After deployment, observe real benefits, harm, and distribution drift, and do not let the training objective replace the product objective.
The execution sequence starts with clarifying the cost and output semantics, goes through proxy selection, hand calculation, and mini overfitting, and finally enters independent validation and online monitoring. Only when the proxy and the true metric improve together on key slices is the plan considered temporarily effective; once the task, users, or distribution changes, the original loss and acceptance relationship cannot be applied directly and must be revalidated.
10Connecting the entire causal chainSynthesis
From a real problem to a single parameter update, why is every step in between indispensable?
- Real tasks contain errors of different types and costs.
- The loss function encodes the selected error costs into comparable scalars.
- Averaging over finite data yields a computable empirical risk.
- A differentiable proxy provides local slopes for predictions and parameters.
- Backpropagation efficiently distributes the slopes to all parameters.
- The optimizer uses gradients to update parameters, decreasing the training proxy.
- Validation sets, slices, and product metrics check whether the proxy still serves the real objective.
- When misalignment is found, change the data, loss, weights, or decision process instead of continuing to blindly push down the same number.
11Common MisconceptionsDisambiguation
| Misconception | More accurate statement |
|---|---|
| Loss is the evaluation metric | The loss serves differentiation; the metric serves judgment, and the two should be related but have different responsibilities. |
| Zero loss means the model is perfect | It only means the proxy is zero under the current data and definition; there may still be leakage, overfitting, or omitted objectives. |
| MSE is always smoother than MAE, so it is better | MSE's strong gradient can also let outliers dominate training; the choice depends on noise and cost. |
| Class weighting only affects training speed | It changes the optimal solution and probabilistic semantics; recalibration and evaluation are necessary. |
| Multi-objective losses can simply be added directly | Different scales can let one term dominate the gradient; normalization, weighting, and conflict diagnosis are needed. |
12Check whether you really understandSelf-test
Check step by step from symbols and values to goal misalignment; answers should explain the reasoning, not just give terminology.
- Why does ordinary gradient training ultimately need a scalar objective, yet we should still retain per-sample, per-group, or per-task losses?
- In
Jtotal=Jdata+λR, λ=0 and a very large λ respectively mean what? - A regression dataset has mostly small errors but contains a few extreme outliers. How will MSE, MAE, and Huber handle them differently?
- In binary classification, two samples are both predicted incorrectly, and their true-class probabilities are 0.49 and 0.01. Why is accuracy the same but cross-entropy different?
- What is a differentiable proxy? Why can't product satisfaction be used directly as the per-step training loss without processing?
- In the running example, if you change the learning rate from 0.1 to 1, the updated
wand loss are what? What does this show? - After increasing class weights, why do we still need to check calibration on the natural distribution?
- If both training loss and validation loss decrease, does that already prove the product objective has improved? What evidence is still missing?
Reference answers
- The update rule ultimately needs a clear direction for comparison and differentiation; but if you keep only the average too early, you will miss minority groups, difficult samples, and task conflicts. You should first keep the components, then specify how to aggregate them.
- λ=0 means the training objective does not include that regularization penalty; a large λ means the model attaches more importance to satisfying the constraint expressed by R, possibly even sacrificing data fitting. The appropriate scale of λ depends on the magnitudes of the loss and R.
- MSE squares and amplifies extreme errors, so outliers can dominate training; MAE charges linearly, is more robust but not smooth at zero; Huber uses squared within the threshold and switches to linear beyond it, striking a balance between smoothness and robustness.
- If the threshold is 0.5, both are counted as errors by accuracy; the natural-log cross-entropies are approximately
−ln0.49=0.713and−ln0.01=4.605, so it will more heavily penalize predictions that confidently exclude the true class. - A differentiable proxy is a training quantity that can be computed frequently and provides smooth local slopes, used to temporarily stand in for the true objective. Satisfaction is usually delayed, sparse, and affected by external factors such as interface, price, or task environment, so it cannot stably provide gradients with clear attribution for each small parameter update.
- The current gradient is −4, so
w=2−1×(−4)=6; the new prediction is 12, the half-squared loss is 18, which is actually worse. This shows that a correct local direction does not mean any step size is safe. - Reweighting changes the effective class frequency and optimal probability during training, so the output may no longer represent the true occurrence rate under the natural distribution.
- No. You also need key slices, calibration, human or online task results, and safety and harm metrics; proxy loss and limited validation data may both be misaligned with the true objective.
13Concept Dependencies and Further LearningPath
| Direction | Next Read | Questions to Take Away |
|---|---|---|
| How gradients become a one-step update | Gradient Descent | When the direction is correct, why can the step size still fail? |
| How to compute gradients for all parameters | Backpropagation | How does the chain rule reuse intermediate results? |
| How to limit memorization of the training set | Regularization, Overfitting | Why do the training proxy and unknown risk separate? |
| Proxy objective being gamed | Reward Hacking | How do metrics become distorted when they become targets? |
| How to ultimately evaluate a model | Model Evaluation | Which slices and thresholds cannot be replaced by averages? |
- Deep Learning — Machine Learning Basics: maximum likelihood, empirical risk, and generalization.
- Deep Learning — Optimization for Training Deep Models: surrogate loss and optimization.
- Google ML Crash Course — Loss: a teaching reference on MSE, MAE, and outlier sensitivity.
- Robust Estimation of a Location Parameter: Huber piecewise loss and robust estimation.
- Focal Loss for Dense Object Detection: class imbalance and loss reweighting.
The main text, figures, and numerical calculations are original organization by this project; sources are used to verify definitions, assumptions, and boundaries.