Gradient Descent: How Direction, Step Size, and Noise Together Shape Learning

1Why can you still descend while looking only at the slope under your feet?Intuition

If you cannot see the entire loss landscape and only know how much the current position tilts in each direction, how should you choose the next small step?

When there is only one parameter,derivativeis the slope at the current position: if the derivative is positive, a small step to the right will increase the loss, so you should try moving left; if the derivative is negative, the opposite is true.

Models usually have many parameters. The training program reads the parameters at the current position and the loss function, and when computing, first fixes the other parameters to find the slope in each dimension: we temporarily change only one of them and keep the others fixed, and the slope obtained is called this parameter'spartial derivative. Then arrange these slopes in order into a vector, and the output is thegradient. For example, the gradient of two parameters (3,−1) indicates: the loss rises quickly when the first parameter increases, and the loss decreases instead when the second parameter increases.

A gradient component with a larger absolute value indicates that the loss is more sensitive near that coordinate; but the gradient is not the entire map, so you cannot conclude from this that this parameter is more important, nor do you know where the distant minimum is.

2Why the Negative Gradient Descends Fastest LocallyMath

Is “the gradient points in the direction of fastest ascent” a metaphor, or a conclusion that can be derived from formulas?

First understand in words: the gradient records, for each parameter, “how moving a little in the positive direction changes the loss.” Given the gradient at the current position and the allowed step length, we want to choose a unit descent direction; if we want each component's change to be as opposite as possible, we should move in the direction in which every component is negated, that is, the negative gradient.

The formula states this more precisely.θ represents all current parameters,Δθ represents the small step to be taken,J is the training objective. When looking only at a very small local region:

Dot productcan be understood as “how much this step projects onto the gradient direction”: when in the same direction, it is positive and the loss tends to increase; when in the opposite direction, it is negative and the loss tends to decrease. If we fix the Euclidean length of one step to ε, first use the dot product to estimate the loss change caused by each candidate displacement; walking completely opposite to the gradient makes the dot product most negative, so within this first-order approximation it descends the most:

||∇J|| is the length of the gradient vector; dividing by it keeps only the direction, and multiplying by ε determines how long this step is. This conclusion only guarantees that “a step near the current position, with the same short length” is the steepest; it does not guarantee reaching the global minimum.

3How does the update rule turn a local approximation into actual parameter updates?Math

Now that we have a direction, how do we write it as a step actually executed in the training loop?

The update rule turns the abstract direction into an action the training program actually executes: it takes the current parameters, the current gradient or the gradient estimated from a mini-batch, and the learning rate, and outputs the next step's parameters. Let's translate this formula term by term:

• t is the current update count;
• θt is all parameters before the update,θt+1 is the parameters after the update;
• gt is the gradient at the current position, or the gradient estimated from a mini-batch of samples;
• ηt is read as eta, and islearning rate, controls how far to go along the gradient this time;
• The preceding minus sign means moving in the direction opposite the gradient.

The program first computes the gradient, then scales it by the learning rate, and finally subtracts this change from the old parameters. Now let's go back to the example defined at the top of the page:w is the multiplier parameter, with ideal value 3 and current incorrect initial value 2. The current gradient is −4, and taking learning rate η=0.1:

The loss dropping from 2 to 0.72 shows that this local move was effective; but a single-step decrease does not prove that subsequent steps will be stable, nor that the model has learned the true pattern or can generalize to new samples. If we take η=1, we get w=6, prediction 12, and loss 18: the same correct gradient was ruined by an overly large step size.

4Why does the learning rate have a stable upper limit?curvature

What exactly controls “small enough,” and why does a steeper bowl wall make it easier to cross the valley bottom in one step?

curvatureDescribes how quickly the slope changes. On flat ground, moving a little farther does not change the slope much; on a steep, curved bowl wall, moving a little farther can cause the slope to reverse sign quickly, so one step crosses from the left side of the valley bottom to the right. The learning rate must be small enough that the local slope remains informative.

A running example can give a fully concrete answer. Here w is still the multiplicative parameter the model needs to learn,J(w) is the loss corresponding to the parameter value,g(w) is the gradient of the loss with respect to w,η is the learning rate. Substitute ŷ=2w and y=6 into the half-squared loss:

The optimal parameter is w*=3. Define the current parameter's distance from the optimum as et=wt−3, after one update step:

Here, taking the local curvature and learning rate as inputs, we first obtain the error multiplier 1−4η, then check whether its absolute value is less than 1: only when the absolute value is less than 1 will it shrink gradually; a negative multiplier means each step crosses the optimum and oscillates back and forth on both sides.

A general quadratic function is written as J(w)=½a(w−w*)², where a controls the curvature, and the stability condition is 0<η<2/a. The larger a is, the lower the safe upper limit. Real neural networks are not a perfect quadratic bowl, and curvature also changes with position, so this interval explains a local stability boundary, not a fixed license valid for all training processes.

5Why narrow valleys make ordinary gradient descent move in a zigzagGeometry

A “narrow valley” is not a new algorithm, but a loss landscape where two parameter directions differ greatly in steepness.

Imagine the loss is determined by two parameters; connecting points with equal loss gives a set ofcontour lines. If the contour lines are close to circular, the steepness is similar in all directions; if they are elongated into a long ellipse, it is like a narrow valley:

• The direction crossing the valley is steep:A small change in the parameters causes a large change in the loss;
• The direction along the valley floor is flat:You have to go a long way for the loss to change noticeably;
• The lowest point is deep in the valley:Training really wants to move forward along the valley floor, but the gradient is often dominated by the steep slopes on both sides.

Starting from the current loss surface and starting point, ordinary gradient descent outputs a parameter trajectory; if the curvature differs greatly between directions, this trajectory can explain the phenomenon of “the gradient is not zero, the loss is decreasing, but training is still slow.”

When computing updates, the gradient is first dominated by the steep direction. A single learning rate must accommodate the lateral steep slope and cannot be too large; but the same small learning rate is too small when used in the valley-floor direction, so the trajectory oscillates laterally back and forth while slowly moving toward the lowest point. The momentum in Section 7 will remember the long-term consistent valley-floor direction and partially cancel the left-right alternating swings.

The more pronounced the zigzag, the more severe the scale mismatch usually is, but the picture is affected by parameter coordinates and scaling, so a two-dimensional schematic cannot be used alone to diagnose a real high-dimensional network. High-dimensional here means that there are many parameter coordinates; mentioning it is to remind the reader that two-dimensional contour lines can only help understand the local mechanism.

6Why mini-batch deliberately uses imprecise gradientsRandomness

If full-batch data can provide more accurate gradients, why does deep learning usually estimate with a small batch of samples?

Full-batch gradientFirst compute the gradient of each sample in the training set, then average, and only then update the parameters once.mini-batch(small batch) takes as input a randomly drawn set of samples and the current parameters, computes the average gradient for only this set, then updates immediately; it repeats this process with a new batch. As long as the sampling method is reasonable, the mini-batch gradient will approximate the full-batch gradient only in the long-term average, but any single step will have sampling error, so the trajectory jitters.

For example, if the training set has 1 million samples and the batch size is 256: the full-batch method must process all 1 million samples before providing feedback once; the mini-batch method can provide feedback about 1,000,000/256≈3906 times. Both are similar in per-epoch sample computation, but the waiting time for a single update and the feedback frequency are completely different.

Here,hardware efficiencymainly refers to how many samples can be processed per unit time, whether the compute units are fully utilized, and how much GPU memory and inter-device communication are needed:

• If the batch is too small, the GPU only does a little work each time, the overhead of launching and moving data accounts for a high proportion, and many compute units sit idle;
• As the batch size increases, matrix operations are easier to parallelize, and per-unit-time throughput usually improves;
• If the batch is too large, it will occupy more GPU memory, and multiple devices will also need to synchronize many gradients; full-batch data usually cannot fit in GPU memory at all, and waiting for a single update is also very long.

Large jitter between batches indicates higher estimation variance, but does not necessarily mean training has failed; noise can sometimes help escape local flat regions, but it should not be mythologized as automatically finding better solutions. Batch size is still jointly constrained by GPU memory, throughput, device synchronization, and task generalization; after changing the batch size, the learning rate, number of updates, and validation performance should all be re-compared.

7From SGD to Momentum and AdamOptimizer

The optimizer is the rule for "how to update parameters after obtaining the gradient"; the difference among the three methods is whether they remember history.

These rules all take the current gradient and hyperparameters as input and output new parameters, to mitigate batch noise, back-and-forth oscillation, and differences in parameter scale.SGD is stochastic gradient descent, i.e.,stochastic gradient descent. Historically, "random" could refer to using only one random sample at a time; in modern training, updates that use a random mini-batch are also commonly called SGD. At each step it takes the current mini-batch gradient gt directly updates:

SGD is simple and uses little extra memory, but if the current batch is noisy, the direction will jitter along with it; in narrow valleys it also tends to oscillate left and right.

Momentum adds a "velocity" v, preserving a decaying average of past gradients:

β controls how long the memory is. Components that point in the same direction over consecutive steps accumulate; components that alternate left and right cancel each other out. So it can accelerate along the valley floor and reduce lateral oscillation.

Adam also keeps two kinds of history:m is the moving average of the gradient, which can be understood as the recent direction;v is the moving average of the squared gradient, which can be understood as the recent scale. During the update, use m/v to adjust the effective step size of each parameter: coordinates with larger long-term gradients are appropriately scaled down, and those with smaller gradients are relatively scaled up. The formal implementation also includes ε to prevent division by zero and bias correction in the early stages of training.

A smoother or faster training curve only shows that optimization is smoother under the current recipe; it does not mean Adam necessarily generalizes best.

8What warmup, decay, and gradient clipping each solveEngineering

They respectively control the early stage of training, the later stage of training, and occasional anomalous steps; they are not three interchangeable names.

In actual execution, warmup and decay produce the actual learning rate for the current step based on the current training step and the learning rate schedule; clipping reads the gradient norm and then decides whether to shrink it. The three do not modify the same quantity: the first two modify the learning rate, and clipping modifies the gradient for that step.

When observing curves, the early training no longer surges, late-stage oscillation weakens, and occasional spikes are clipped—these respectively indicate that the corresponding mechanisms are taking effect. Below, they are broken down in execution order.

9How to Diagnose a Training Stall Layer by LayerWorked Example

When loss fails to decrease, first identify where in the chain the signal is broken; don't immediately switch to Adam and hope for luck.

This diagnostic process breaks “loss not decreasing” into verifiable failure hypotheses. It takes small-sample experiments, predictions and labels, gradient norms, update ratios, and validation curves, progressively narrowing the search across data, computation chain, step size, optimizer, or target misalignment:

1. First, run the “small-sample memorization test”:Give the model only a few dozen samples and train repeatedly until it almost memorizes them. If it cannot even do that, the problem is usually not generalization but a disconnect in the data, loss, or update chain.
2. Check predictions and loss inputs:Directly print the inputs, targets, predictions, and per-sample loss for a few samples to confirm that labels are not misaligned and that positions requiring loss computation have not been entirely masked out.
3. Check the gradient norm:The gradient norm is the length of the vector composed of all parameter gradients. When it stays at 0, check whether parameters are 'frozen' (excluded from updates) or whether the computation chain has been accidentally disconnected; when a huge spike suddenly appears, check for abnormal batches and numerical overflow.
4. Check how large the actual update is:Compute ||Δθ||/||θ||, that is, “length of the parameter change in this step ÷ length of the parameters themselves”. If the gradient is non-zero but this ratio is close to 0, it indicates the learning rate is too small, numerical precision has swallowed the change, or the optimizer is not updating the intended parameters.
5. Run a learning rate sprint:Train for only a few steps with multiple learning rates from small to large to find the range between 'starts to decrease clearly' and 'starts to oscillate or diverge', rather than running full experiments directly.
6. Then compare optimizers and schedules:Fix the model, data order, batch size, and number of training steps, and change only one factor at a time among SGD, momentum, Adam, or the learning rate schedule.
7. Finally, look at independent validation:A faster decrease in training loss only means the surrogate objective is being optimized faster; the stopping point should still be determined by data not involved in updates and by real task metrics.

This sequence first proves the model can memorize small samples, then verifies loss inputs and gradients, then checks actual updates, and only finally compares optimizers and independent validation. An anomaly in one item only narrows the search and does not automatically prove the sole root cause; when log metrics are defined incorrectly, random fluctuations, or multiple faults coexist, you still need controlled comparison experiments.

10Connecting the entire causal chainSynthesis

From local slope to reliable training, how does each link constrain the next?

1. The loss turns current prediction quality into a scalar.
2. The gradient gives the local sensitivity of that scalar to each parameter.
3. The negative gradient makes the first-order approximation decrease fastest under the given geometry.
4. The learning rate turns the direction into an actual distance and must respect local curvature.
5. Mini-batch trades noisy estimates for more frequent, parallelizable updates.
6. Momentum, preconditioning, and scheduling improve the trajectory and step sizes at different stages.
7. Diagnostic metrics confirm that gradients, updates, and numerical states are genuinely valid.
8. Independent validation determines whether this optimization trajectory also improves unseen data and real tasks.

11Common MisconceptionsDisambiguation

12Check if you really understandSelf-test

From local slopes and concrete hand calculations to batch and optimizer choices; answers must explain the symbols or mechanisms.

1. What are partial derivatives and gradients, respectively? Why is the gradient not a map to the global minimum?
2. When looking only at the local first-order approximation, why does a fixed-length step along the negative gradient decrease the most?
3. In the running example, when the learning rate is 0.25, after one step w, what are the prediction and loss?
4. In J(w)=2(w−3)² in, why does η=0.4 cross over the optimum yet still converge, while η=1 diverges?
5. What specific curvature relationship between the two directions does the “narrow valley” represent? Why does the trajectory proceed in a zigzag?
6. What specifically does the “better hardware efficiency” of mini-batch refer to? Why is a larger batch not always better?
7. What historical information do SGD, Momentum, and Adam remember, respectively? How do they change the trajectory in a narrow valley?
8. Which phase of training or which type of anomaly do warm-up, learning rate decay, and gradient clipping address, respectively?

13Concept Dependencies and Further LearningPath