Neural Network

1Why We Need Neural NetworksIntuition

The core understanding this section really aims to establish is:Neural networks exist not because linear models are useless, but because many important relationships in the real world cannot be expressed with a “straight boundary”; neural networks enable models to describe these complex relationships by learning nonlinear intermediate features.

First, look at what a linear model does. The formula

It means the model multiplies all input features by their respective weights, adds them together, and finally adds a bias to obtain a prediction score. For example, to predict house prices based on area, age, and distance from city center, you can multiply the three features by their corresponding weights and add them. Weights represent the direction and strength of a feature’s effect on the outcome: positive weights usually mean that larger feature values lead to higher prediction scores; negative weights mean larger feature values lead to lower prediction scores. Bias b does not depend on any input; it adjusts the baseline position of the overall prediction result.

In linear regression, this score can be used directly as the predicted value; in binary classification, the class can be determined by whether the score exceeds a certain threshold. Assuming zero is used as the threshold, the classification boundary is:

When the input has only two features, this boundary is a straight line; with three features, it is a plane; in higher dimensions, it is called a hyperplane. Although the names differ, the essence is the same: it can only perform a single “flat cut”.

This reveals the fundamental limitation of linear models. Suppose there are two classes of points on a plane: one class is concentrated at the center, and the other class forms a ring around the center. No matter how you move or rotate a straight line, it cannot completely separate the inner and outer circles, because the correct boundary needs to bend around the center and close on itself. The difficulty here is not that the model has not yet found the “correct straight line”; rather,the correct answer does not belong to the class of straight-line boundaries at all. Continuing to train a linear model can only search for a relatively better one among many imperfect straight lines; it cannot fundamentally solve the problem.

However, “the original data is not linearly separable” does not mean that linear models can never handle it. You can first manually construct new features. For example, if the difference between the inner and outer circles depends on the distance from a point to the center, then you can compute

Then let the linear model classify based on r^2 whether it exceeds a certain threshold. In the original x_1,x_2 coordinates, the boundary is a circular curve; but on the new feature r^2 it is just a simple threshold. This reveals a very important fact:Many so-called nonlinear problems can be transformed into easier-to-handle problems through appropriate feature transformations.

Traditional feature engineering is exactly the process of humans looking for such transformations. For problems with clear structure and a small number of features, this approach can be very effective. For example, when predicting an object’s motion, you can construct velocity and acceleration; when analyzing financial data, you can construct returns, moving averages, and volatility. The problem is that when the input is an image, a sound, or an article, the original dimensionality can reach thousands or even millions, and the truly useful combinations often cannot be enumerated in advance.

Taking image recognition as an example, a single pixel usually does not carry much meaning by itself. To determine whether there is a cat in an image, you may first need to combine neighboring pixels to form edges, then combine edges to form textures and local shapes, then combine local shapes to form ears, eyes, and outlines, and finally integrate these structures to make a judgment. If you rely entirely on humans manually specifying which pixels should be combined and how, not only is the number of combinations enormous, but it is also very difficult to cover variations in lighting, angle, pose, and background.

Therefore, the most important value of neural networks is not the superficial “having many layers,” but rather that during training they canautomatically learn a series of useful feature transformations. After passing through the hidden layers, the original input is gradually represented as new intermediate features. Taking images as an example, the shallow layers may be sensitive to edges and color changes, the middle layers may combine textures or local structures, and deeper layers may form representations related to complete objects. Specific networks do not necessarily work strictly according to such human-nameable levels, but the core idea is: the model no longer requires people to design all key features in advance; instead, based on the training objective, it finds internal representations from the data that help reduce error.

Next, we need to understand a point that is easy to get wrong: since a single linear layer has limited capacity, can we solve the problem simply by stacking many linear layers? The answer is no. Suppose the first layer is

The second layer is

where x is the original input,h is the intermediate result produced by the first layer,y is the final output. Substituting the first layer into the second layer, we get:

If we let

then the entire two-layer network becomes:

It is still just a single affine transformation. More strictly speaking, the expression with bias Wx+b belongs to affine transformations; everyday explanations often broadly call it a linear layer. Regardless of the name, the key conclusion remains:Multiple linear or affine layers without nonlinear activations can be combined into an equivalent layer.

This means that even if such a network has one hundred layers, the types of functions it can express do not fundamentally change. Although many variables are produced in between, in the end it can still only form flat classification boundaries. Depth only adds parameter representation and computation; it does not increase the expressive capacity needed to solve complex nonlinear problems.

What truly changes the situation is adding nonlinear functions between the linear transformations. The structure goes from

to something like

where σ is a nonlinear activation function. Since σ cannot be simply eliminated by merging like matrix multiplication and addition, the two layers are no longer equivalent to a single linear layer. The network can repeatedly perform “linear combination—nonlinear transformation—recombination,” gradually constructing curved and complex functions and decision boundaries. From this we can see that the power of neural networks comes from the combination of two factors: linear layers are responsible for combining information, and nonlinear activations are responsible for breaking the limitation of only being able to perform flat transformations.

We also need to avoid misinterpreting “neural networks can express complex relationships” as “any problem should use a neural network.” Whether a model is appropriate depends on the data and the task. Neural networks are usually suitable for high-dimensional data such as images, text, and speech, and for problems with complex nonlinear patterns and sufficient training data. But if the data size is very small, the data is mainly structured tables, the business rules can be expressed with a few clear rules, or the task requires strict interpretability at every step, then linear models, tree models, or rule systems may be less costly, faster to train, more stable, and easier to inspect.

Therefore, this section can be condensed into a complete logical chain:

2Network structure: what it consists ofIntuitionMath

As explained in the previous section, a neural network cannot gain new expressive power by just stacking linear transformations; it must add nonlinearity between linear transformations. This section further answers: how these transformations are organized in the network, and what roles the input layer, nodes, hidden layers, and output layer play.

You can initially view a minimal neural network as an information-processing pipeline:

For example, a network has two inputs x_1,x_2, containing two nodes h_1,h_2 in its hidden layer, and an output node y-hat. The inputs are first sent simultaneously to the two nodes in the hidden layer; each hidden node produces a response; these two responses are then passed to the output node, ultimately producing the prediction. Every connection in the figure carries a weight that can be adjusted through training, and each non-input node usually also has its own bias.

The input layer is where the network receives data. It usually does not perform actual learning computations, but carries the feature vector that has been converted into numbers x. In tabular tasks, inputs may be age, income, and number of purchases; in image tasks, inputs may be pixel values; in text models, raw words are first converted into numeric vectors before entering subsequent networks. The role of the input layer is not to understand these numbers, but to make clear to the network "which values make up a sample".

It is important to note that the meaning and organization of input numbers matter a great deal. A neural network cannot directly access real-world objects themselves; it always sees numeric representations. If the input representation loses critical information, no matter how complex the subsequent network is, it cannot recover it out of thin air. For example, if you give the model only the average brightness of an image but ask it to recognize the objects in the image, the input itself does not retain enough shape information.

After the input enters an ordinary node, it first computes:

Then it computes:

These two steps correspond to the two stages inside a node.

The first step is a weighted sum. Each input x_i is multiplied by a weight w_i, all the results are added together and then the bias is added b, giving z. This z is called the pre-activation value, because it is the result before entering the activation function.

The second step is activation. The activation function φ to z transforms it, producing the output that the node actually passes to the next layer a. If the activation function is ReLU, then

Negative pre-activation values become zero, while positive values are retained. If the output layer uses Sigmoid, then

Any value of z will be compressed to between 0 and 1, so it is often used to represent a probabilistic output in binary classification.

A node can be understood as a "learnable pattern detector," but this statement needs to be unpacked. The weight vector w describes the direction of the pattern the node cares about, and the dot product w· x measures how well the current input matches this direction, and the bias b adjusts the firing threshold, and the activation function decides how the matching result is converted into a response.

Suppose the input has two features and a node's weight is

and its bias is -0.5, then it computes:

This means the node tends to produce a higher response when x_1 is large and x_2 is relatively small. If ReLU is used, only when

does it output a positive number. The bias -0.5 changes the triggering condition here: without a bias, the boundary must pass through the origin; after adding a bias, the boundary can be shifted as a whole. Thus, weights determine the detection direction, and bias determines the detection threshold.

The statement "one node detects one pattern" is an approximation to aid understanding; it does not mean that each node must correspond to a concept that can be clearly named by humans. Some nodes may respond clearly to edges, colors, or word relationships, but many nodes learn dispersed and abstract mathematical features that are hard to name directly. What really matters is that different nodes can learn different weights and biases, thereby responding differently to different directions or combinations of the input.

Real tasks usually cannot be accomplished by relying on a single pattern. Therefore, a hidden layer will contain multiple nodes that process the same input in parallel. Suppose this layer has m nodes, and each node computes its own pre-activation value and activation result:

If you arrange their outputs, you get a new vector:

Therefore, the output of a layer is usually not a final answer, but a new representation made up of many node responses. Each output component summarizes some aspect of the input, and the next layer will treat this new vector as its input.

This is exactly the core meaning of a hidden layer:A hidden layer does not directly give the task answer; instead, it transforms the raw input into a representation that is more suitable for subsequent judgment.For example, in an idealized interpretation of image recognition, the first layer may respond to horizontal edges, vertical edges, or color changes; the next layer combines these responses to form corners or textures; deeper layers continue to combine them into local shapes. The representations learned by real networks are usually more complex than this example, but the logic of transforming representations layer by layer is consistent.

The word "hidden" does not mean these layers cannot be inspected; it means that the training data usually only directly specifies the input and target output, without telling the model one by one what each intermediate node should represent. Hidden layer representations form on their own during the process of optimizing the final task. For example, the training data only labels images as cat or dog, but does not additionally require that a particular node must detect ears; if ear-related features help reduce classification error, the network may form corresponding responses during training.

When a layer contains many nodes, writing out formulas with the same structure one by one becomes tedious. Matrix notation simply combines these computations into a single expression. By placing each node's weight vector as a row of the matrix, we obtain the weight matrix W; arranging each node's bias into a vector b, the entire layer computation can be written as:

Suppose the input x has n_(in) components, and this layer has n_(out) nodes, then the shapes of the objects are:

Matrix W 's j row is the j node's weight. Matrix multiplication Wx completes the dot products of all nodes at once. Therefore, matrices do not introduce new neural network principles; they are just an efficient way to organize parallel computation.

The output layer is responsible for converting the representation formed by the last hidden layer into the result required by the task. Its specific form depends on the task:

• Regression tasks may output just one unrestricted numerical value, such as predicting temperature or price;
• Binary classification tasks usually output one numerical value, then convert it to a result between 0 and 1 through Sigmoid;
• Multi-class classification tasks usually output a score for each class, then convert them into a class distribution through Softmax;
• More complex tasks may output a sequence of words, multiple bounding boxes, or pixel results of the same size as the input.

Therefore, the output layer is not a fixed structure; it must match the meaning of the target. Even if the preceding hidden layers are exactly the same, changing the output layer allows the network to serve different types of tasks.

Next, consider the number of parameters. For a fully connected layer, every input is connected to every output node. If there are n_(in) inputs and n_(out) nodes, the number of weights is:

Each output node also has a bias, so the number of biases is:

The total number of parameters is:

For a network of "2 inputs → 2 hidden nodes → 1 output node", the number of parameters in the first layer is:

There are 4 weights and 2 biases. The number of parameters in the second layer is:

There are 2 weights and 1 bias. The entire network has:

learnable parameters.

Here, "learnable" means these parameters are not hard-coded one by one by the designer, but are continuously adjusted according to the error during training. The network structure determines how many layers there are, how many nodes are in each layer, and how the nodes are connected; training then finds suitable weights and biases within this pre-specified structure. The structure is the container, and the parameters are the specific values in the container that need to be learned.

More parameters usually mean the network has greater representational capacity, because it has more degrees of freedom to adjust. But greater capacity does not necessarily mean better performance. Increasing parameters brings at least three kinds of costs: it requires more computation, so training and inference will be slower; it requires more storage space and video memory; and it is more likely to memorize accidental details in the training samples, causing overfitting. On the other hand, a network that is too small may not have enough capacity to express the patterns in the task, leading to underfitting.

Therefore, what really needs to be balanced when designing a network is: make the model capacity sufficient to represent the patterns in the task, while also matching the data scale, computational resources, and generalization requirements. A network is not necessarily better just because it has more nodes or deeper layers.

The structural relationships of this section can be summarized as:

3Forward Propagation: How the Network Gives AnswersMath

The previous two sections have established two things: a neural network is composed of layers of nodes, and each node performs "weighted sum, then passes through an activation function"; nonlinear activation makes a multi-layer network no longer collapse into a linear model. Forward propagation aims to answer:When the inputs and parameters are given, how does data pass through these layers and ultimately become a prediction result?

"Forward" describes the direction of computation. Starting from the input layer, information flows through each hidden layer in turn and finally reaches the output layer, without going backward in between. Suppose the network has two learnable transformations; its complete computation can be written as:

Here, the superscript (1) and (2) indicate different layers, not squares. x is the input fed into the network, W^((1)),b^((1)),W^((2)),b^((2)) are the current parameters, φ is the hidden layer activation function, g is the transformation chosen by the output layer according to the task, y-hat is the prediction given by the network. The hat symbol is used to distinguish "the model's predicted result" from the true target y.

The first step of forward propagation actually occurs before the network computation: converting real-world data into numbers suitable for the network to process. Different types of data require different preprocessing. For example, continuous numerical values may need standardization, categories may need encoding, image pixels may need scaling, and text needs to be converted into tokens and vectors first. Preprocessing is not trivial packaging, because the scale and representation of input values directly affect the information the network subsequently sees.

Take numerical standardization as an example. If one feature ranges from 0 to 1 and another feature ranges from 0 to 1,000,000, the latter may naturally produce larger values in the weighted sum. Although the network can in theory compensate by adjusting its weights, training often becomes more difficult. Adjusting features to a more reasonable scale usually makes computation and learning more stable. Note that the same preprocessing rules must be used during training and actual use; otherwise, even if the network parameters do not change, the meaning of the inputs it receives will change.

After obtaining the input vector x, the first hidden layer computes:

This is the weighted sum of all nodes in the layer. The matrix W^((1)) has each row corresponding to the weights of one node, and the bias vector b^((1)) provides an independent bias for each node. The computed z^((1)) is the pre-activation value of all nodes in this layer.

Then, the activation function is applied element-wise to these pre-activation values:

If the hidden layer uses ReLU, then each component is processed according to max(0,z). Positive values are retained, and negative values become zero. The resulting a^((1)) is no longer just a linear combination of the original input, but a new representation extracted by the first hidden layer based on the current parameters.

This new representation becomes the input to the next layer. In other words, the second layer no longer directly processes the original feature x, but instead processes the a^((1)) produced by the first layer. If there are more hidden layers, continue repeating:

where l represents the current layer,a^((l-1)) is the output of the previous layer. The first layer usually treats a^((0)) as the original input x. Therefore, regardless of how many layers the network has, forward propagation can be understood as repeatedly performing "a linear transformation to produce pre-activation values, then a nonlinear function to produce a new representation".

We can use the "2 inputs → 2 hidden nodes → 1 output node" network from the previous section to do a concrete calculation. Suppose the input is:

The hidden layer parameters are:

First compute the pre-activation values:

If the hidden layer uses ReLU, then:

Next, suppose the output layer weights and bias are:

The output layer's pre-activation value is:

If this is a binary classification task and the output layer uses Sigmoid, the prediction is:

If we interpret y-hat as the predicted probability of belonging to the positive class, then the network currently thinks the likelihood of the positive class is about 26.9%. If the classification threshold is set to 0.5, it will predict the negative class. This example shows the entire process of forward propagation: each step is determined by the current input, weights, biases, and activation function, with no random guessing and no parameter modification during the computation.

What transformation the output layer should adopt depends on the task requirements. A regression task may directly output z, because the target may be any real number; binary classification tasks often use Sigmoid to compress a score to between 0 and 1; mutually exclusive multi-class tasks often use Softmax to convert multiple class scores into a distribution that sums to 1. The output transformation must work together with the loss function and the meaning of the target; you cannot use a certain activation function just because it is common, regardless of the task.

Forward propagation itself is just a computation process and does not mean the network is learning. During actual inference, the network usually does the following: receives a new input, uses the same preprocessing as during training, completes forward propagation with the determined parameters, and then outputs the prediction. Throughout the process, the weights and biases remain unchanged. The same parameters, given the same input, will produce the same result when there is no additional random mechanism.

Training includes forward propagation but is not limited to it. During training, first forward propagation is used to obtain y-hat, then the prediction is compared with the true target y to calculate the loss. Next, determine how responsible each parameter is for the loss, and update the parameters in the direction that can reduce the loss. Therefore, the logical chain of training is:

Pure inference usually only includes the first half:

It is important to distinguish the two. Forward propagation answers "what will this network output when the parameters remain at their current values"; the loss function answers "how wrong is this output"; backpropagation answers "which parameters caused these errors and in what direction"; the optimizer answers "exactly how much to change the parameters next". Forward propagation provides the predictions and intermediate computation results needed for training, but it does not itself correct errors.

This section can be summed up in one sentence:Forward propagation is to let the input, according to the network's established structure and current parameters, go through "weighted sum and nonlinear transformation" layer by layer, and ultimately form a prediction that matches the task format; inference ends here, while training still needs to adjust the parameters in reverse based on the prediction error.

4Activation Functions: Where Nonlinearity Comes FromIntuitionMath

An earlier key question remains: even if multiple linear or affine layers are stacked deeply, the whole can still be combined into a single affine transformation, so why can neural networks represent curved and complex relationships? The answer is to add activation functions between the linear layers.Linear layers are responsible for recombining information, and activation functions are responsible for breaking linearity so that these combinations can produce turns, gating, and complex boundaries.

A node usually first computes the pre-activation value:

Then the output is obtained through the activation function:

If φ(z)=z, the activation function just lets it pass unchanged, and the network is still linear. Only when φ is a nonlinear function can the two adjacent layers not be simply combined. For example:

Since φ is located between two linear transformations, the whole expression usually cannot be rearranged into a single Wx+b. This seemingly small change is exactly the key to deep networks gaining extra expressive power.

One of the most common hidden-layer activations is ReLU:

When z<0, it outputs 0; when z>0, it outputs z. ReLU's graph consists of two straight line segments, but overall it is not a straight line, because at z=0 it changes slope. This kink is the source of the nonlinearity.

A single ReLU node, depending on whether w· x+b is positive or negative, divides the input space into two regions. In one region the node is off and outputs zero; in the other region the node is active and the output varies linearly with the input. Multiple ReLU nodes with different weights and biases create many boundaries at different positions. Their outputs, when combined by later layers, divide the input space into more regions.

Therefore, the complex function formed by a ReLU network can be understood as "piecewise linear": inside each small region there is still a linear relationship, but after crossing the activation boundaries of different nodes, the function's slope changes. Just as many short straight line segments can approximate a circular arc, a large number of nodes and multiple layers can also use many locally flat pieces to piece together an overall curved decision boundary. This is not a case of one node drawing the complete curve by itself; rather, many nodes jointly provide turns, and later layers combine those turns.

ReLU is simple in form and computationally cheap. On the positive half-axis, its derivative is 1, and it does not enter a saturation region where the gradient is close to zero as the positive input increases, unlike Sigmoid; but on the negative half-axis, both its output and derivative are 0. However, ReLU also has a clear risk: when z<0, its output is exactly zero, and the gradient with respect to z is also zero. If a node lands on the negative half-axis for a long time on the training data, it may no longer receive effective updates and will continue to output zero; this is called the "dead ReLU".

Leaky ReLU is a direct mitigation method:

where alpha is a small positive number. In this way, the negative half-axis is no longer completely flat but keeps a very gentle slope; nodes in the negative region can still receive gradients. However, alpha needs to be set manually or determined by some method, and it only mitigates some problems and does not guarantee that all training difficulties will disappear.

The formula for Sigmoid is:

It compresses any real number into the range between 0 and 1. When z is large, the output is close to 1; when z is small, the output is close to 0; when z=0, the output is 0.5. This range makes it especially suitable for binary classification output, or for gating structures inside a network that need "how much on, how much off".

The problem with Sigmoid is saturation. When the input lies in a region of large positive or negative values, the curve becomes almost flat and the derivative is close to zero. When backpropagation passes through such a region, the gradient may be significantly reduced; if the network has many layers, this reduction accumulates layer by layer and easily leads to vanishing gradients. Therefore, although Sigmoid is common in binary classification output layers, it is usually not the default choice for hidden layers in modern deep networks.

tanh can be written as:

It compresses the input to -1 and 1. Compared with Sigmoid, tanh's output is zero-centered: negative inputs produce negative responses and positive inputs produce positive responses. This is more convenient in some models. However, both ends of tanh also flatten, so the problems of saturation and vanishing gradients still exist.

GELU is a common choice in modern Transformer hidden layers. Unlike ReLU, it does not make a hard cutoff at zero; instead, it performs smooth gating based on the input magnitude: large positive values are mostly kept, large negative values are mostly suppressed, and the transition near zero is smooth. It preserves nonlinearity and is smoother than ReLU, but its computational expression is also more complex. Choosing GELU does not mean it is necessarily better than ReLU on all tasks; it should be understood as a common design validated in practice in certain modern architectures.

Softmax does not process a single number but a set of class scores. For the i-th class, it computes:

Each output is between 0 and 1, and the sum of all outputs is 1. The larger a class score is relative to the other classes, the higher the output it receives. Therefore, Softmax is often used in multi-class classification tasks where the classes are mutually exclusive, such as an image that can only be classified as one of "cat, dog, bird".

Softmax outputs are often called class probabilities, but this needs to be understood carefully. It first normalizes the relative scores into a distribution; whether these values actually match the true probabilities in reality also depends on the training data, the loss function, and whether the model is well calibrated. A model that outputs 0.99 does not automatically guarantee that it has 99% accuracy on all samples where it outputs 0.99.

A point of easy confusion is that ReLU, Sigmoid, tanh, GELU, and Softmax may all be collectively called activation functions, but their roles in the network are not exactly the same.

The question facing hidden layers is: "How do we turn the current representation into a new representation that the next layer can continue to combine?" Here we use ReLU, GELU, or tanh mainly to introduce nonlinearity. If none of the hidden layers has nonlinearity, the multiple affine computations from the hidden layers to the output transformation can be combined into a single affine transformation. If the output layer also has no nonlinearity, the whole network is still an affine function; if the output uses Sigmoid or Softmax, the final probability mapping is not an affine function, but the hidden layers still have not gained new nonlinear representational capacity.

The question facing the output layer is: "What range and meaning should the final answer have?" Regression tasks may use a linear output, allowing the result to be any real number; binary classification may use Sigmoid to produce a value between 0 and 1; mutually exclusive multi-class classification may use Softmax to produce a class distribution. The output layer can use a linear mapping, or nonlinear transformations such as Sigmoid or Softmax. Which one to choose first depends on the output range and meaning required by the task.

This distinction also explains why the output function cannot be chosen in isolation. The output layer, target encoding, and loss function must be compatible with one another. For example, binary classification often pairs a single output score, Sigmoid, and binary cross-entropy; mutually exclusive multi-class classification often pairs multiple class scores, Softmax, and multi-class cross-entropy; real-valued regression may use a linear output and mean squared error. If the combination does not match, the meaning of the model output, the way the loss is computed, and the direction of training may conflict with one another.

Activation functions are not necessarily better just because they are more complex. The choice affects representational capacity, gradient propagation, computational cost, and training stability. For ordinary fully connected networks or convolutional networks, ReLU is often a robust starting point for hidden layers; when worried about completely losing gradients in the negative region, consider Leaky ReLU; GELU is common in Transformers; Sigmoid and Softmax are used more according to the output task or gating needs. The final choice should still be based on architecture, data, and actual validation.

The logic of this section can be summarized as:Linear layers can only linearly recombine information; activation functions create turns between these recombinations; the turns of many nodes, combined across layers, can piece together complex nonlinear functions. The main task of hidden-layer activations is to create composable internal representations, and the main task of the output-layer transformation is to convert the final numerical values into answers that match the task's meaning.

5Fundamental Principle: Differentiable Function CompositionIntuition

The first four sections respectively introduced why neural networks need nonlinearity, what components networks are made of, how predictions are computed forward, and how activation functions produce nonlinearity. Now we can unify these ideas into one core model:A neural network decomposes a complex mapping into the composition of many simple functions and, as far as possible, makes these functions differentiable with respect to the parameters, so that it can both express complex relationships and be trained through gradients.

Suppose a network has three layers, and each layer can be seen as a function:

The whole network is:

This relationship of “handing the output of one function to the next function” is called function composition. Input x is first f_1 into h_1,h_1 then by f_2 into h_2, and finally by f_3 becomes the prediction. The depth of the network, in essence, is how many levels this transformation has occurred consecutively.

Why is function composition valuable? Because complex regularities can often be decomposed into several levels of relatively clear simple processing. When recognizing an image, jumping directly from millions of pixels to “this is a cat” is very difficult; if you first form representations of local changes, then combine them into texture and shape evidence, and finally aggregate them into a category judgment, the problem is divided into multiple levels of transformation. Neural networks do not necessarily literally follow the strict “edge–part–object” division of labor from human language, but function composition allows them to build this type of hierarchical computation.

Every layer here is re-encoding information. Re-encoding is not simply copying, nor does it mean that the information necessarily becomes richer; rather, it transforms the previous layer's values into a coordinate system that is more convenient for subsequent tasks. For example, the same image is initially represented in pixel coordinates; after several layers, the network may internally form directions that are more sensitive to shape, texture, or category judgment. Samples that are originally difficult to separate with simple boundaries in pixel space may be easier to separate in the new representation space.

We can analogize this change to reselecting the language used to describe the problem. Suppose the task is to distinguish points inside and outside a ring, in the original coordinates (x_1,x_2) you need a curved boundary; if you re-represent the input as “distance to the center of the circle,” classification may only need to compare a threshold. What the hidden layers of a neural network do is similar to automatically finding such useful new representations, but it is usually not specified in advance by humans as a concrete formula; instead, it searches through training for transformations that can reduce the error of the final task.

This is representation learning. Traditional feature engineering has humans decide which features should be extracted; representation learning lets the model automatically adjust intermediate-layer parameters based on data and objectives, forming internal features that are helpful for the task. Here we must emphasize “helpful for the task”: the representation learned by the network is not necessarily good for all uses, nor does it necessarily correspond to concepts that are easy for humans to understand. A network trained for cat and dog classification may retain the information needed to distinguish the categories while ignoring background details irrelevant to the classification.

Function composition provides expressive power, but the network must also be trainable. Training needs to answer: if the final prediction is wrong, in which direction should each layer and each parameter be adjusted? This is where “differentiability” comes into play.

If the loss is denoted as L, network parameters are denoted as θ, training wants to know:

This derivative describes roughly how the loss changes when the parameter changes by a small amount. If the derivative is positive, increasing the parameter in the positive direction will cause the loss to rise locally; if the derivative is negative, increasing the parameter will cause the loss to fall locally. Optimization algorithms use these gradients to find update directions that can reduce the loss for a large number of parameters.

Function composition is especially suitable for this kind of training because the chain rule can pass the final error back layer by layer along the composite relationship. For:

The influence of a certain early parameter on the final loss can be decomposed into its influence on the output of its own layer, the influence of that layer’s output on the next layer, and so on until the influence of the final output on the loss. This is exactly the mathematical basis of backpropagation. Forward propagation computes from input to output along the composite function; backpropagation uses the chain rule to compute the gradients of all parameters in reverse, starting from the loss.

Strictly speaking, networks in practice do not require every point to have a derivative that is smooth and continuous in the ordinary sense. ReLU at z=0 is not differentiable, but it is differentiable at other positions, and a conventional subgradient can be chosen at the kink, so gradient methods can still work. Therefore, “differentiable” here should be understood as: most of the network’s computation can provide local change information suitable for backpropagation, rather than requiring every function to be mathematically smooth everywhere.

The width and depth of a network affect function composition in different ways. Width is the number of nodes per layer. A wider layer can form more responses in parallel, equivalent to providing more adjustable feature directions at the same stage. However, increasing nodes usually means more parameters, higher computation and memory requirements, and may also increase the risk of overfitting.

Depth is the number of hidden layers. A deeper network can perform more levels of transformation, combining simple patterns layer by layer into complex patterns. For problems that themselves have hierarchical structure, this composition may be more effective than a single layer directly representing them. But depth also makes optimization difficult: gradients must propagate through more layers and may gradually become smaller, larger, or be disrupted by different layer scales. Techniques such as residual connections, normalization, and appropriate initialization largely serve to make deep function composition easier to train.

Both width and depth can increase model capacity, but they cannot simply replace each other, and model quality cannot be judged solely by the number of parameters. Greater width may allow the same-level representation to contain more patterns, while greater depth allows more levels of sequential composition. What ratio is suitable for a specific task needs to be judged in combination with data structure, computational budget, and training stability.

“Universal approximation” is one of the most easily exaggerated concepts when understanding neural networks. The general idea of the relevant theory is that, under certain conditions, a network with sufficient capacity can approximate many continuous functions to arbitrary precision. It shows that the family of functions represented by neural networks has strong expressive power, but it does not guarantee that the training algorithm will necessarily find the required parameters, nor does it specify how much data and computation are needed, nor does it guarantee that the model will still be correct when faced with new samples.

Therefore, at least four questions must be separated:

First, expressive power: does there exist a set of parameters for the network structure that can represent the target relationship?

Second, optimizability: can existing training methods find sufficiently good parameters within a reasonable time?

Third, sample efficiency: how much representative data is needed to learn this relationship?

Fourth, generalization ability: is the model still effective on new samples outside the training data?

A network may be theoretically capable of representing the answer, yet fail to train because of gradient problems; it may also fit the training set very well but merely memorize the samples and fail to generalize; it may also fail to learn a reliable relationship no matter how large the model is, because the key regularities were never contained in the data. The model can only utilize information present in the training signal; strong expressive power cannot create facts out of thin air.

The final thing to remember from this section is:The “neural” appearance of neural networks is not their most fundamental mathematical feature; a more essential description is a differentiable computational system composed of many parameterized small functions. Function composition allows it to learn representations layer by layer and express complex mappings, and differentiability allows the final error to be passed back to the parameters through the chain rule. Being able to represent, being able to train, learning efficiently, and being able to generalize are four interrelated but not conflatable issues.

6From Internal Values to Answers: Output Layer and LossMath

After the preceding network passes through multiple layers of transformations, the last hidden layer produces a set of internal values. These values are merely a representation formed by the model; they are not yet answers to the real-world task. For the network to be trainable, two consecutive translations must be performed:The output layer first translates the internal representation into a prediction with task meaning, and the loss function then translates the difference between the prediction and the true answer into a scalar that can be optimized.

Assume the last hidden layer output is h, and the output layer usually first computes a set of raw scores:

These raw scores are often called logits. Next, whether to apply z a transformation, and what transformation to use, depends on what the task wants the output to express. The output layer is not a fixed component detached from the task; it is the interface between the model's internal values and the real-world answer.

For regression tasks, the target is usually an arbitrary real number, such as house prices, temperature, or sales volume. The output layer often uses a linear output, that is, directly setting:

The "linear" or "no activation" here does not mean the entire network becomes a linear model, because the preceding hidden layers already contain nonlinearity; it only means the final step no longer restricts the output range. If Sigmoid is mistakenly used, predictions will be compressed between 0 and 1, clearly unable to directly represent typical house prices. Some regression targets do have range constraints, in which case a corresponding transformation can be chosen, but this choice must come from the target's meaning, not from mechanical application.

For binary classification tasks, usually only one output score z, and then pass it through Sigmoid:

This way y-hat is between 0 and 1 and can be interpreted as a probabilistic score assigned by the model to the positive class. For example, y-hat=0.8 indicates the model strongly favors the positive class, and the corresponding score for the negative class is 1-0.8=0.2. If a final class is needed, a threshold must be chosen; 0.5 is a common default, but in tasks where the cost of missed diagnoses differs from false positives, the most suitable threshold may not be 0.5.

For mutually exclusive multiclass tasks, the network produces a logit for each class, denoted z_1,…,z_K, and then through Softmax:

All p_k are between 0 and 1, and their sum is 1. Increasing the relative score of one class affects the share occupied by other classes, reflecting competition among classes. This is suitable for tasks where "each sample can belong to only one class," such as choosing one among cat, dog, and bird in an image.

Multilabel classification differs from mutually exclusive multiclass. An image can simultaneously contain "person," "car," and "night scene"; multiple labels can hold at the same time, so they cannot be made to compete for a share that sums to 1 via Softmax. The common practice is to set an independent logit for each label and use Sigmoid separately:

This way each label has its own 0-to-1 output, and increasing the probability of one label does not require lowering other labels. This shows that the difference between Softmax and multiple Sigmoids is not just a different formula; they express two different real-world assumptions: classes are mutually exclusive, or labels can coexist.

After determining the meaning of the output, it is also necessary to define "how wrong." The model cannot directly optimize vague evaluations like "looks good" or "basically correct"; it needs a numerical objective. The loss function maps the prediction y-hat and the true answer y into a scalar L. In general, a smaller loss means the prediction better matches the training objective.

Let all weights and biases of the network be θ, and the network prediction is written as f(x;θ). For a training set containing N samples, the basic objective can be written as:

where x_i is the i input, y_i is its true answer, L is the loss for a single sample; after summing and dividing by N we get the average training loss.argmin asks not "what is the minimum loss" but "which set of parameters can make this objective as small as possible." The training process is to continuously adjust θ to try to approach such parameters.

For regression, the common mean squared error, for a single scalar prediction, can be written as:

If the true value is 10 and the prediction is 12, the error is 2 and the squared loss is 4; if the prediction is 20, the error is 10 and the squared loss is 100. Squaring imposes a stronger penalty on large errors and avoids positive and negative errors canceling each other. It also has convenient differentiation properties. However, MSE is sensitive to unusually large errors; if the data contains severe outliers, other regression losses may need to be considered.

For binary classification, binary cross-entropy is common:

When the true label y=1, the second term disappears and the loss becomes:

The closer the probability the model assigns to the true positive class is to 1, the smaller the loss; if it very confidently gives a probability close to 0, the loss will be very large. When y=0, the loss becomes -ln(1-y-hat), and the model should give a low probability to the positive class. Cross-entropy cares not only whether the final classification is correct, but also how much probability the model assigned to the correct class.

For example, two models both classify samples above the threshold as positive: Model A outputs 0.51, Model B outputs 0.99. From an accuracy standpoint, both answer correctly once; but if the true label is positive, cross-entropy will consider B's prediction more consistent with the objective. Conversely, if the true label is negative, Model B's high-confidence error will receive a much larger penalty than A. This continuously varying penalty provides a detailed signal for gradient training.

Mutually exclusive multiclass often uses multiclass cross-entropy:

If the true label uses one-hot encoding, only the y_k corresponding to the true class is 1 and the rest are 0, so the formula actually simplifies to:

That is to say, it mainly checks how much probability the model assigned to the true class. The higher the probability of the true class, the smaller the loss; if the model gave almost all probability to the wrong class, the loss is very large.

In practice, implementations often directly combine logits and cross-entropy into one calculation, rather than first explicitly computing Sigmoid or Softmax and then taking the log. This usually has better numerical stability. For example, very large or very small logits may cause exponentiation to overflow, or cause probabilities to be rounded to exactly 0, and then computing ln 0 will cause problems. The framework-provided "cross-entropy with logits" usually computes with an equivalent but more stable formula. This does not change the output meaning; it only improves the numerical implementation.

The loss function and evaluation metrics must not be conflated. The loss is the signal minimized by the optimizer during training, and usually needs a usable gradient with respect to the model output. Accuracy, F1, and AUC are evaluation metrics mainly used by people to judge how well the model performs under a certain business criterion; they are not necessarily suitable for direct differentiation.

Taking accuracy as an example, when a binary classification output changes from 0.49 to 0.48, if the threshold is 0.5, the classification result does not change and accuracy has no change; when it changes from 0.49 to 0.51, accuracy suddenly jumps. Such plateaus and jumps cannot provide a smooth direction for every tiny parameter update. Cross-entropy, by contrast, changes continuously with the predicted probability, so it is more suitable as a training signal.

But a lower optimizable loss does not automatically mean actual evaluation is better. The average training loss may decrease while validation performance worsens, which is a sign of overfitting; the model may also have low cross-entropy but perform poorly on F1 due to an inappropriate threshold. Therefore, use the loss to guide parameter updates during training, and use metrics that match the actual objective during validation and testing; each has its own role.

The basic objective formula also only states the average loss on the training set. Actual training often adds regularization, class weights, or other constraints, and what is really desired is a model that is also effective on unseen data, not simply pushing the training loss to the lowest possible value.θ^* In a complex neural network, this is usually not the globally optimal solution that can be solved for exactly, but a set of good enough parameters found by the optimization algorithm within a finite time.

The complete logic of this section is:The last hidden layer only produces an internal representation; the output layer converts it into a real number, a binary classification score, a mutually exclusive class distribution, or multiple independent labels according to task rules; the loss function then compresses the difference between the prediction and the true answer into a differentiable scalar; training updates parameters by minimizing this scalar, while metrics such as accuracy, F1, and AUC evaluate the model from the perspective of human task objectives.

7Gradient Descent: Why Move in the Opposite Direction of the Gradient?Math

In the previous section, the loss function compressed prediction error into a scalar, but knowing “how wrong we currently are” is not enough. A neural network may have tens of thousands or even billions of parameters, so training must further answer:For each parameter, should it increase or decrease, and by how much, to make the loss go down?The gradient provides the local direction, and the learning rate controls the step size; together they form the basic update of gradient descent.

Let’s start with the case of a single parameter θ to understand. The loss is written as L(θ), and its derivative with respect to the parameter is:

This derivative describes how the loss changes near the current position when the parameter increases slightly. If the derivative is positive, increasing θ will cause the loss to rise, so the parameter should be decreased appropriately θ; if the derivative is negative, increasing θ will instead make the loss decrease, so the parameter should be increased appropriately θ. The two cases can be written in a unified way as:

where η>0 is the learning rate. Subtracting the derivative automatically chooses the opposite direction: when the derivative is positive, the parameter decreases; when the derivative is negative, the parameter increases.

When there are many parameters, write all parameters as the vector θ=[θ_1,θ_2,…,θ_m]^T. The partial derivatives of the loss with respect to each parameter form the gradient:

The gradient has the same shape as the parameter vector. Each component describes how the loss locally responds when the corresponding parameter changes slightly. The gradient is not a vague “error magnitude”; it is a list of local sensitivities for all parameters.

Why does the gradient indicate the direction of fastest increase? Suppose from the current position we move a small step u along the unit direction ε, the loss can be approximated to first order as:

With the same step size, the change in loss is mainly determined by the dot product ∇ L· u. According to the geometric property of the dot product, when u is in the same direction as the gradient, this value is largest, meaning the loss increases fastest; when u is opposite to the gradient direction, this value is smallest, meaning the loss decreases fastest. Therefore, the local steepest descent direction is:

This is the direct reason behind the name “gradient descent”: the gradient itself points in the direction of local fastest increase, and taking the negative gives the direction of local fastest decrease.

The “fastest” here has two limitations. First, it compares directions when taking small steps of the same length; if different directions were allowed arbitrary step sizes, they could not be compared directly. Second, it relies only on a first-order approximation near the current position, so it is a local conclusion and does not mean the gradient points directly to the global minimum. The loss surface may be curved, narrow, contain saddle points, or have multiple valley floors; the gradient only tells us which side is steepest under our feet, not an entire topographic map.

The original text mentions that there is no unique “slowest descent direction,” which can also be seen from the dot product. If the direction u is perpendicular to the gradient, then:

Under the first-order approximation, the loss is almost unchanged. Choosing a direction nearly perpendicular to the gradient but slightly toward the descent side can make the decrease arbitrarily close to zero. Therefore, unlike the clear direction of “steepest descent,” there is no direction of “slowest descent” with the same unique meaning.

The basic gradient descent update is written as:

The negative gradient determines the direction, and the learning rate η determines the step size. Even if the direction is correct, the step size determines whether training succeeds, because the gradient is reliable only in a sufficiently small neighborhood; if a step is too large, the actual loss surface may already curve elsewhere.

When the learning rate is too small, each step moves only a little, and the loss may decrease steadily, but training requires many steps with high time and computational cost. In flat regions, updates may even be so small as to be almost invisible. When the learning rate is too large, the parameters may overshoot the valley floor and oscillate between the two sides; in more severe cases, they may jump each time to a region of higher loss and larger gradient, eventually causing numerical divergence. An appropriate learning rate must balance speed and stability, and in actual training it is often adjusted as training progresses.

Gradient descent does not guarantee finding the global optimum at every step. For deep networks, the loss surface is usually high-dimensional and non-convex, and may contain saddle points, flat regions, and many different low-loss regions. The realistic goal of training is usually not to prove that a unique global minimum has been found, but to find a set of parameters with low training loss and good validation performance within an acceptable computational budget.

Full training begins with parameter initialization. If the incoming weights and biases of hidden nodes in the same layer are identical, they will produce the same response to the same input; if the corresponding downstream connections, optimizer states, and update rules also maintain node permutation symmetry, they will continue to receive the same gradients and remain duplicates. Different downstream weights may break this symmetry. Random initialization of weights is a common way to give different nodes the opportunity to learn different patterns; biases can often be initialized to zero, and not every parameter needs to be randomized. Setting all weights to zero risks symmetry or hindered gradients, but the specific behavior depends on the activation function, connections, and update rules. The scale of random numbers must still be reasonable; too large or too small may affect signal and gradient propagation.

Next, a batch of data is taken for forward propagation. Why is a small batch commonly used instead of looking at only one sample at a time, or the entire training set each time? Gradients computed over the full training set are closer to the overall average direction, but each update is costly; single-sample updates are cheap but have large random fluctuations. Mini-batch training provides a compromise among computational efficiency, hardware parallelism, and gradient stability, and is a common practice in modern neural network training.

For the current batch, the network completes forward propagation and obtains predictions, then computes the average loss over the samples in the batch. Backpropagation then uses the chain rule to compute the gradient of the loss with respect to each weight and bias. It is important to distinguish: Backpropagation is responsible for efficiently computing the gradient, while gradient descent or another optimizer uses the gradient to update parameters. They are closely connected but not the same step.

The most basic stochastic mini-batch gradient descent updates parameters according to the current gradient; more common optimizers further process the gradient. For example, momentum methods combine past update directions to reduce oscillations, and Adam adjusts the effective step size for different parameters based on first- and second-order statistics of the gradient. No matter how complex the rule, the common input remains the gradient obtained by Backpropagation, and the purpose remains to find parameters with lower loss.

A complete training step can be written as:

This process is repeated many times. A batch is a small batch of samples used for one parameter update; an iteration or step usually represents completing one “forward, backward, update”; an epoch means that all samples in the training set have been roughly used once. If the training set has 10,000 samples and the batch size is 100, then one epoch usually contains about 100 steps.

Parameters and hyperparameters must also be distinguished. Parameters are the internal values that the model learns during training, such as weights and biases; they are updated by the optimizer based on the gradient. Hyperparameters are set by the trainer or an external search process, such as learning rate, batch size, number of hidden layers, width of each layer, and regularization strength. Parameters are the answer the training process seeks; hyperparameters define the model structure and the way the answer is sought.

During training, you must not focus only on training loss. If training loss continues to decrease while validation loss starts to increase, the model may be overfitting to the training data. Training curves can help detect oscillations caused by a learning rate that is too large, stagnation caused by a learning rate that is too small, numerical divergence, and overfitting. Gradient descent is responsible for reducing the current optimization objective; whether it generalizes to new data also depends on validation, regularization, data quality, and a reasonable stopping strategy.

The final understanding to establish from this section is:The gradient is not a map to the lowest point, but slope information at the current position; for small steps of the same length, the gradient points to the fastest increase, and the negative gradient points to the fastest decrease. The learning rate determines how far to go along the negative gradient, Backpropagation is responsible for computing the gradient, the optimizer uses the gradient, and many batches and epochs of repeated updates together constitute training.

8Backpropagation: How the Chain Rule Assigns ResponsibilityMath

The previous section explained that the optimizer needs the gradient of the loss with respect to each parameter before it can decide the parameter update direction. However, a neural network's prediction is produced by many parameters together, and the loss function gives only a final value; it does not directly indicate how much each weight and bias contributed. Backpropagation solves exactly this computational problem:It uses the chain rule to pass sensitivity layer by layer from the loss backward to earlier nodes, reusing already-computed downstream influences, and thereby obtains the gradients of all parameters in a single backward pass.

First, we must distinguish between backpropagation and the optimizer. Backpropagation is responsible for computing:

That is, the gradient of the loss with respect to all parameters. Gradient descent, SGD with momentum, or Adam then read these gradients and update parameters according to their own rules. For example, the most basic gradient descent performs:

Therefore, backpropagation answers “what is the gradient,” and the optimizer answers “how to change the parameters after obtaining the gradient.” Backpropagation itself usually does not modify the weights; it is inaccurate to call it an optimizer.

First consider the simplest computation chain. Suppose a weight w affects the loss through the following process:

From a causal perspective,w first affects z,z then affects a,a affects the prediction y-hat, and the prediction finally affects the loss L. The chain rule writes the entire influence chain as:

Each term describes only the local influence between two adjacent steps. Multiplying these local derivatives gives the total influence of the parameter w on the final loss. This is the mathematical expression of “if the parameter changes a little, how much the node changes; how much the node's change makes the next step change; and ultimately how much the loss changes.”

We can work through a concrete scalar example by hand. Let:

The loss uses:

Take x=2,w=1,b=1,v=3,y=5. Forward propagation gives:

Next, compute backward from the loss. The derivative of the loss with respect to the prediction is:

The derivative of the prediction with respect to the activation is:

Because currently z=3>0, the derivative of ReLU here is:

The derivative of the pre-activation value with respect to the weight is:

So:

This positive gradient indicates that near the current position, increasing w will make the loss rise, so gradient descent will decrease w. Similarly, we can compute:

and:

This shows that a final error can be specifically distributed to multiple parameters through different local derivatives. The “responsibility” here is not responsibility in a moral sense, nor is it splitting the loss into several parts and requiring the sum to equal the loss; it represents the local sensitivity of the loss to each parameter.

The sign and magnitude of the gradient must be interpreted relative to the current position. A positive gradient indicates that slightly increasing the parameter will locally increase the loss, and a negative gradient indicates that slightly increasing the parameter will locally decrease the loss. A larger absolute value indicates that the loss is more locally sensitive to changes in that parameter, but it does not necessarily mean that the parameter is globally “more important,” nor does it mean that it should be modified by an arbitrarily large amount. The actual update magnitude is also affected by the learning rate, optimizer state, regularization, and other factors.

If we estimate the gradient using the most naive method, we can increase a parameter by a very small ε, redo a complete forward propagation, and then compare the change in the loss:

This is called finite difference approximation. It is useful for checking whether a small number of gradients are implemented correctly, but it is unsuitable for daily training. If a network has one million parameters, perturbing each parameter separately and recomputing the loss would require about one million extra forward calculations; the more parameters, the more serious the repeated computation.

The key to backpropagation's efficiency is not the novelty of the chain rule itself, but the reuse of common downstream computations. Suppose many early parameters first affect the same intermediate value a, and then a affects the subsequent output and loss. The naive method would repeatedly compute for each early parameter “a how it affects the loss afterwards.” Backpropagation instead first computes once:

Then take this already-obtained downstream sensitivity and multiply it by each local derivative entering a to obtain the gradients of earlier variables and parameters respectively. The common later part is computed only once, rather than walking the entire chain again for each parameter.

The network is not always a chain without branches. If a variable affects the loss through multiple paths, then its total gradient is the sum of the contributions from each path. For example,h simultaneously enters two subsequent branches u(h) and v(h), and the loss is L(u,v), then:

Influences occurring in sequence along one path are connected by multiplication, and influences from multiple different paths are aggregated by addition. These two rules are sufficient to explain how gradients flow in complex networks: multiply along a chain, add across branches.

In actual implementations, in addition to computing the prediction, forward propagation also saves the intermediate results needed for the backward computation, such as a layer's input, pre-activation values, or activation values. Backpropagation starts from the loss and processes each operation in the reverse order of the forward pass, computing the local derivative of each operation and passing the gradient coming from upstream further to earlier positions. Saving intermediate results consumes memory, which is one reason why training generally requires more GPU memory than inference alone.

The “computation graph” is a tool for describing this dependency relationship: nodes represent values or operations, and edges represent how a result is used by subsequent computations. “Dynamic programming” emphasizes reusing already-computed sub-results. Understanding these terms helps analyze complex implementations, but the core of backpropagation does not depend on mastering them first. As long as you grasp “first compute downstream sensitivity once, then distribute it along all incoming paths,” you have understood the fundamental reason for its efficiency.

Backpropagation can also explain the vanishing-gradient and exploding-gradient problems mentioned earlier. The gradient of early parameters often contains the product of local derivatives across many layers. In the simplified case of a scalar chain product, if the absolute value of each local derivative does not exceed some constant less than 1, the magnitude of the repeated multiplication will decay as the number of layers increases; sustained amplification may also make the gradient very large. Actual networks involve products of Jacobian matrices and multiple paths, so they also depend on the weights, activation states, and gradient propagation direction. This is not backpropagation making an error; rather, the chain rule faithfully reflects the sensitivity changes in deep function composition.

Automatic differentiation frameworks do a large amount of differentiation and reuse for us, but they do not change the basic logic. The model first performs forward propagation and records the computational relationships; the loss produces the initial gradient; the backward process computes the gradients of all parameters according to the chain rule; and finally the optimizer reads the gradients and updates the parameters. Usually, old gradients should also be cleared or reset before the next batch of data to avoid unintentional accumulation; some training designs deliberately accumulate gradients over multiple batches, but that is an explicit choice.

This section can be summarized as:The loss gives only the final error. Backpropagation uses the chain rule to connect “parameter change—intermediate node change—prediction change—loss change.” It reuses common downstream influences from the output side backward, multiplying derivatives along the chain and adding contributions across branches, so that one backward traversal obtains the gradients of all parameters. Only after the gradients are computed does gradient descent or Adam update the parameters.

9Complete Manual Calculation: One Forward Pass + One UpdateMath

This section ties together the previously separate topics of structure, activation functions, output layer, loss, backpropagation, and gradient descent into one complete training step. The goal is not to memorize these specific numbers, but to see clearly:how a sample produces a prediction and loss, how the loss becomes the gradient for each parameter, and why the prediction changes after the parameters are updated.

The task provides the input:

The true label is:

The network has two inputs, two hidden nodes using ReLU, and one output node using Sigmoid. The initial parameters are:

Matrix W_1 corresponds to one hidden node. Therefore, the first row [0.5,-0.25] are the weights of hidden node 1 for the two inputs, and the second row [1.0,0.5] are the weights of hidden node 2.

First, complete the forward propagation. The hidden layer pre-activation values are:

The first hidden node gives:

The second hidden node gives:

So:

The hidden layer uses ReLU:

This means that with the current input and parameters, the first hidden node does not pass a positive response forward, and the second node outputs 1. The output layer receives a_1, not the original input x. Its pre-activation value is:

After the Sigmoid:

The true label is 1, but the model only gives an output of about 0.4013 for the positive class, so the prediction is too low. Binary cross-entropy when y=1 simplifies to:

Substituting the prediction:

At this point, forward propagation has answered 'what do the current parameters predict?', and the loss has answered 'how wrong is this prediction?'. Next, backpropagation starts from the loss to find the gradient of each parameter.

When Sigmoid and binary cross-entropy are combined, the derivative of the loss with respect to the output layer logit can be simplified to:

This result can be obtained from the chain rule. Binary cross-entropy with respect to y-hat differentiated, then multiplied by the derivative of Sigmoid y-hat(1-y-hat), where some terms cancel out, ultimately leaving y-hat-y. This simple and numerically well-behaved gradient is an important reason why Sigmoid and binary cross-entropy are often used together.

Define the output layer error signal:

The negative sign means that if z_2 is increased slightly at the current position, the loss will decrease. This matches intuition: the true label is 1, and the prediction is too low, so the model needs to increase the output logit and the positive-class prediction.

Because:

So the output weight gradient is:

Substituting the values:

The gradient of the first output weight is zero because the hidden node connected to it outputs zero. The input passed through this connection for the current sample is zero, so slightly changing this weight does not change the current output. The second hidden node outputs 1, so the second output weight receives a non-zero gradient.

The output layer raw score z_2 with respect to the output bias b_2 has a local derivative of 1, so:

Next, pass the influence of the output layer back to the hidden layer. The gradient of the loss with respect to the hidden layer activations is:

This vector is still only the sensitivity of the loss to the ReLU output. To continue propagating to z_1, you also need to multiply by the local derivative of ReLU. Here we adopt the convention that ReLU at z=0 has derivative 0; at positive values the derivative is 1. Therefore:

The hidden layer error signal is the element-wise multiplication:

The downstream influence of the first hidden node was not originally zero, but it is cut off when passing through the ReLU derivative. Therefore, the first-layer parameters corresponding to this node have no gradient on this sample. The second hidden node is in the positive half-axis, so the gradient can continue to pass through.

Because:

The first-layer weight gradient is the outer product of the hidden layer error signal and the input:

The first-layer bias gradient is:

At this point, backpropagation has obtained the gradients of all parameters. Now gradient descent performs the update, with learning rate η=0.1:

The second weight of the output layer is updated to:

The output bias is updated to:

The second row weights of the first layer are updated to:

The first layer's second bias is updated to:

Parameters with zero gradient remain unchanged. Therefore, W_2's first element, W_1's first row and b_1's first element did not change in this update. This does not mean they can never learn; with a different input, the first hidden node may enter the positive half-axis and thus receive a non-zero gradient.

We can run forward propagation again with the new parameters to check whether this step actually improved the current sample. The new pre-activation value of the second hidden node is approximately:

After ReLU it is still about 0.7486. The new output logit is approximately:

So the new prediction is approximately:

The new loss is approximately:

The prediction rose from 0.4013 toward the true label 1 to about 0.4702, and the loss dropped from about 0.913 to about 0.755. This verifies that this update step is effective for this sample. However, gradient descent only provides a descent guarantee locally and with sufficiently small step sizes; in actual mini-batch training, a single step does not guarantee that the loss of every individual sample will decrease; what is optimized is the batch or overall objective.

This update also reveals the true meaning of 'learning'. The network did not store 'the input [1,2] has the answer 1' completely in some node, nor did any single connection bear the learning alone. The output weight, output bias, and the input weight and bias of the second hidden node all underwent small interrelated changes. Together they changed the hidden representation and the final output, reducing the loss for the current target.

The changes produced by a single update are usually small, and it only uses local information provided by one sample or a mini-batch. Training repeats 'forward—loss—backpropagation—update' through thousands of steps, allowing different samples to continuously influence the parameters. The resulting parameters are the product of many training signals acting together, and the network's internal representations gradually become useful for the task through these coordinated updates.

This section can be summarized as:Forward propagation uses the current parameters to compute the prediction and loss; backpropagation starts from the output error and uses the chain rule to obtain the gradients of each layer's parameters; the optimizer then updates the parameters along the negative gradient. Learning is not shoving the answer into a node, but rather causing all the parameters in the network that participate in the current computation to make coordinated fine-tuning, and through many repetitions gradually reduce the overall loss.

10Why Training FailsEngineering

Understanding forward propagation, loss, backpropagation, and parameter updates does not mean that actual training will necessarily succeed. The mathematical formulas describe how training should proceed, but engineering practice must also ensure correct data, numerical stability, gradient propagation, and that the model has not merely memorized the training samples."Training failure" is not a single problem: sometimes the parameters do not learn at all, sometimes values diverge, sometimes the program cuts off gradients, and sometimes the training set performs well but cannot generalize. The first step in diagnosis is to determine which kind of failure it is.

Vanishing gradients occur when the gradients received by earlier layers become extremely small. Backpropagation uses the chain rule, and the gradients of early parameters typically include the product of many local derivatives across layers. When backpropagation is continuously compressed along the relevant direction as it passes through layers, the gradient reaching early layers can be very small. As an intuitive illustration using scalar products, this is multiple factors with magnitude less than 1 multiplied together; real networks must also consider the combined effect of matrices and multiple paths. At this point, layers near the output can still update, while layers near the input barely move, making it difficult for the network to learn effective low-level representations.

Sigmoid and tanh have very small derivatives in saturated regions, so deep networks that use many of these activations are more prone to vanishing gradients. ReLU or GELU in appropriate operating ranges usually improve gradient propagation, but they are not a universal solution by themselves. Proper initialization, normalization, and residual connections control signal scale from different angles or shorten gradient propagation paths, making deep networks easier to train.

Exploding gradients are the opposite phenomenon: factors in the chain product or weight scales are too large, causing gradients to grow rapidly as they propagate toward earlier layers. Parameter updates can suddenly become very large, the loss oscillates violently, and even infinity or NaN appears. Reducing the learning rate can reduce the size of each update; gradient clipping can limit the magnitude when the gradient norm exceeds a threshold; proper initialization and normalization try to avoid activation and gradient scales being amplified layer by layer from the source.

Note that gradient clipping is more like a safety guardrail. It can prevent extreme gradients from directly wrecking training, but it does not necessarily eliminate the root cause of the explosion. If training repeatedly triggers clipping, you should still check the learning rate, initialization, data anomalies, loss implementation, and network structure, rather than simply repeatedly lowering the clipping threshold.

Dying ReLU refers to a ReLU node staying in the negative half-axis for a long time. Because:

And the derivative in this region is zero, so this node neither outputs a signal downstream nor can obtain gradients for its input parameters through the current sample. If an overly large update pushes many nodes into a long-term negative interval, or initialization makes them hard to activate from the start, the effective capacity of the network decreases.

Lowering the learning rate can reduce the risk of parameters being pushed too far by a single update; proper initialization can make initial pre-activation values distributed in a more usable range; Leaky ReLU preserves a small gradient on the negative half-axis, giving the node a chance to recover. But a node occasionally outputting zero is not the same as dying ReLU. ReLU is by design allowed to turn off nodes for some inputs; only when a node remains inactive for almost all relevant data for a long time does it become a problem worth addressing.

Another common phenomenon is that the training loss does not decrease from the start. The cause may be very basic: input-label misalignment, label encoding errors, inconsistent preprocessing, mismatch between loss function and output layer, parameters not passed to the optimizer, forgetting to perform updates, gradients disabled at the wrong locations, or parameters effectively frozen. It could also be that the learning rate is too small for changes to be observed, or too large, causing each step to overshoot the effective region.

When troubleshooting this kind of problem, a very effective method is to deliberately overfit the model to a tiny dataset, for example taking only a few samples and training for many steps. A network with sufficient capacity should usually be able to drive the training loss on these few samples very low. If it cannot even fit the tiny dataset, the problem is more likely in the implementation, data pipeline, loss configuration, or optimization process, rather than "insufficient generalization".

After failing the tiny dataset test, you can inspect the training chain segment by segment: whether the input values and labels are correct; whether the range and shape of forward outputs meet expectations; whether the loss is finite and changes with predictions; whether each parameter requires gradients; after backpropagation, whether gradients exist, are all zero, or are abnormally huge; and whether the parameters actually change after the optimizer update. Breaking complex training into these observable links is more effective than blindly swapping models or optimizers.

There is also a situation where the training loss decreases normally and training metrics are good, but validation performance is markedly worse. This is usually not "the network cannot learn", but rather the network has learned to fit the training set too closely, including its accidental noise and details, forming overfitting. At this point, continuing to reduce training loss may further worsen validation performance.

Mitigating overfitting can be approached from both the information side and the capacity side. More and more representative data helps the model distinguish stable patterns from accidental details; data augmentation can generate more variation under reasonable assumptions; regularization methods such as weight decay and Dropout limit excessive reliance on specific parameters or paths; early stopping stops training when validation performance no longer improves; and reducing the model directly lowers available capacity. The response depends on the task; you cannot always blame a large model for "gaps between training and validation", and you should also check whether the data distributions, preprocessing, and evaluation procedures for training and validation are consistent.

Initialization is the starting point of training stability. Hidden nodes in the same layer with identical incoming weights and biases can only guarantee that they have the same forward response to the same input. If the corresponding downstream connections, optimizer states, and update rules also preserve node permutation symmetry—that is, swapping these nodes does not change the relevant computation and updates—then they will receive the same gradients and remain duplicated in subsequent updates, making it difficult to specialize in learning different patterns. Identical incoming parameters alone are not enough: for example, if the input is 1, two ReLU nodes both have incoming weight 1 and bias 0, but their linear output connection weights are 1 and 2 respectively, the output is 3. When the target is 0 and the loss is half the square of the output, the gradients of the two nodes' incoming weights are 3 and 6 respectively, and ordinary gradient descent will separate them. Randomly initializing weights is a common way to break symmetry; when weights are already properly initialized, biases can usually be set to zero. Setting all weights to zero may preserve symmetry or block gradients, so it cannot be used as a general initialization scheme.

But random does not mean choosing an arbitrary scale. If initial weights are too small, signals and gradients may shrink layer by layer; if too large, activations may saturate or values may amplify layer by layer. Xavier or Glorot initialization controls weight variance based on the input and output widths and is often paired with activations such as tanh; He initialization adjusts scale mainly based on the input width and is often paired with ReLU-type activations. Their common goal is to prevent forward activations and backward gradients from amplifying or shrinking too quickly as they propagate across layers.

Normalization methods also control intermediate values, but in different specific ways. Batch normalization uses statistics within a batch to standardize intermediate activations, then restores an appropriate scale through learnable parameters; it is common in convolutional networks, but its behavior is affected by batch size and the switch between training and inference modes. Layer normalization normalizes across the feature dimension of a single sample and does not depend on other samples in the same batch, so it is very common in Transformers.

Normalization is not just about making numbers "look nice". It allows different layers to face more stable numerical ranges, making optimization less sensitive to parameter scale and learning rate, and helps train deep models. But the position, dimension, and training mode of normalization must be implemented correctly. For example, batch normalization uses batch statistics during training and usually uses accumulated statistics during inference; if the mode switch is wrong, validation results may be abnormal.

A residual connection writes a certain transformation as:

rather than requiring this group of layers to directly learn the complete mapping from scratch y=F(x). In this way, the transformation branch can learn "what correction needs to be added on top of the input". If the optimal operation is close to preserving the original, making the residual branch close to zero is relatively easy. More importantly, the addition provides a more direct path for signals and gradients, so the gradient does not have to rely entirely on the chain product of all local derivatives in the deep transformation.

Residual connections do not eliminate all optimization problems and cannot guarantee effectiveness at any depth, but they significantly change the trainability of deep networks. Normalization, residual connections, proper initialization, activation function choice, and gradient clipping are not defining components that "a neural network must have to be a neural network"; a simple network can compute and learn without them. They belong to engineering structures: when facing deeper, larger, and harder-to-optimize models, they help turn theoretical trainability into stable training in practice.

When diagnosing, distinguish symptoms from root causes. A NaN loss may come from exploding gradients, or from taking the logarithm of zero, dividing by zero, invalid data, or precision overflow; zero gradients may come from saturated activations, or from parameters being frozen or the computation graph being accidentally cut off; poor validation may be overfitting, or it may be inconsistent data processing between validation and training. Do not immediately apply a fixed answer based solely on a surface symptom; instead verify layer by layer through values, gradients, parameter changes, and data samples.

A practical troubleshooting order is: first manually inspect a small number of inputs and labels; then confirm a simple baseline and evaluation logic; make the model overfit a tiny dataset; observe whether activations, loss, and gradients in each layer are finite and at reasonable scales; confirm that parameters actually change before and after updates; and only then systematically adjust learning rate, initialization, normalization, regularization, and model capacity. Eliminate correctness issues first, then address stability and generalization issues, to avoid repeatedly tuning parameters on incorrect code.

The core insight of this section is:Successful training requires not only that the network has expressive capacity and computable gradients, but also that gradients remain usable in deep structures, parameters break symmetry, numerical scales are stable, data and implementation are correct, and the model learns generalizable patterns. Initialization, normalization, and residual connections are engineering tools that make deep networks easier to optimize, while the gap between training and validation sets reminds us that fitting and generalizing are two different things.

11Generalization: Good on the Training Set Is Not EnoughIntuitionEngineering

Good performance on the training set only shows that the model has successfully adapted to data it has already seen; after real deployment, it usually faces new samples unseen during training.Generalization is the ability of a model to correctly apply patterns learned from training data to new data.This is not an extra requirement after training is complete, but the real goal of supervised learning.

Neural networks usually have a large number of parameters, so they may learn stable patterns or memorize incidental details in the training set. Suppose cats in training images mostly appear on sofas, while dogs mostly appear on grass; the network may classify using background colors instead of learning the animals' own features. It can still achieve high accuracy on the training set but will easily fail on a cat on grass. This shows that low training error does not automatically prove that the model has learned the patterns we hoped for.

Overfitting means the model adapts excessively to the training data, including noise, incidental correlations, and sample-specific details, so that performance on new data worsens. Underfitting means the model has not even captured the main patterns in the training data, and both training and validation performance are poor. The remedies for the two often differ: overfitting usually requires more effective data, stronger constraints, or smaller capacity; underfitting may require more suitable features, a larger model, longer training, or more effective optimization.

To observe generalization, data is usually split into training, validation, and test sets. The training set directly participates in gradient updates, and model parameters are learned from it. The validation set is not used to update weights but to compare choices such as architecture, learning rate, regularization strength, number of training epochs, and decision thresholds. The test set should be kept until all design choices are complete and used only to estimate the final solution's performance on unseen data.

The division of labor among the three can be summarized as: the training set is used to learn parameters, the validation set is used to select solutions, and the test set is used for final evaluation. Although the validation set does not directly participate in backpropagation, repeatedly modifying the model based on validation results effectively makes the design process gradually adapt to the validation set. Therefore, you cannot continue tuning parameters after seeing test results and then claim the same test result as a completely independent estimate of generalization.

Data splitting must reflect the real deployment scenario. If multiple records from the same person are randomly scattered across training and test sets, the model may indirectly recognize test samples through individual characteristics; if a time series is randomly shuffled before splitting, training data may contain information from after the test time points; if different crops of the same original image end up in different sets, near-duplicate leakage also occurs. These situations can make test metrics look good while overestimating actual generalization ability.

During training, such curves often appear: training loss continues to decrease, while validation loss first decreases and then rises. In the early stage, the model learns patterns shared by the training and validation sets, so both improve; after continued training, the model increasingly fits details of the training set, so training loss keeps decreasing but validation loss begins to worsen. Near the lowest validation loss is usually an appropriate early stopping point.

Early stopping does not mean terminating immediately upon seeing one validation fluctuation. Validation metrics themselves can be noisy; in practice, a certain patience is often set: stop only when several consecutive evaluations show no improvement. At the same time, you should save the model checkpoint with the best validation performance rather than directly using the parameters from the last step. Otherwise, even if later overfitting is correctly detected, what remains may still be a model that has already degraded.

L2 regularization constrains the model by adding a squared weight penalty to the original training loss:

where L_(data) measures prediction error, and the second term penalizes larger weights,lambda controls the strength of the constraint. It tends to make the model use more diffuse and smaller weights rather than relying extremely heavily on a few connections. Intuitively, this often makes the function less sensitive to input changes, thereby reducing excessive fitting to training-sample details.

Weight decay is implemented in many optimizers as directly shrinking the weights and is closely related to L2 regularization. In ordinary gradient descent, the two can correspond, but in adaptive optimizers such as Adam, decoupled weight decay and adding the L2 term directly to the loss are not always completely equivalent. The most important understanding for beginners is: both try to limit unconstrained weight growth, but the specific implementation should follow the definitions of the optimizer and framework used.

Regularization strength also needs to be balanced.lambda too small may fail to suppress overfitting; too large will push useful weights too small as well, making the model unable to express even the training patterns, resulting in underfitting. Therefore, regularization strength is a hyperparameter that needs to be selected based on the validation set.

Dropout randomly sets a portion of node outputs to zero during training, so the network cannot fully rely on the same set of internal paths at every step. Some nodes may be masked in one batch and restored in the next, forcing other nodes to also carry information representation and reducing fragile fixed co-adaptation among multiple features. It can be approximately understood as training many subnetworks with shared parameters and combining them into a complete network at inference.

Dropout behaves differently during training and inference. During training it randomly masks; during inference you should turn off random masking, use all nodes, and handle scaling according to the chosen implementation. Modern frameworks typically use “inverted Dropout”: during training, retained outputs are automatically scaled up so that their expected scale matches the inference stage; at inference, you simply disable Dropout. If you forget to switch to inference mode, predictions will continue to change randomly and results are usually unreliable.

Data augmentation does not directly constrain parameters; instead, it expands the effective training distribution by generating reasonable variations that conform to task invariances. For example, in image classification, slight cropping, flipping, or color perturbations may be used so the model cannot just memorize fixed pixel positions. Speech can include noise that matches the scenario, and text tasks may use carefully designed rewriting.

Whether data augmentation is correct depends on whether the label still holds after the transformation. Horizontally flipping images of ordinary objects may be reasonable, but flipping traffic signs, text, or medical images with left-right semantics may change meaning; arbitrarily rotating the digit 6 may approach the digit 9. Incorrect augmentation provides contradictory training signals to the model. Therefore, augmentation policies must come from real task invariances, not simply applied because a method is common.

Reducing model size is the most direct form of capacity control. Reducing the number of layers or nodes usually reduces parameters and computation; sharing parameters in task-appropriate structures can also reduce the number of independent parameters and storage requirements. These adjustments change the model's expressive power but do not guarantee better generalization. However, too little capacity leads to underfitting. A larger model is not necessarily worse at generalizing; with appropriate data, optimization, and regularization, a large model may find a good solution. Model size should be determined jointly by task difficulty, data quantity, and deployment budget.

These methods address overfitting from different angles: L2 or weight decay constrains parameter scale; Dropout perturbs internal paths and reduces co-adaptation; early stopping limits the time the model has to adapt to the training set; data augmentation expands the range of reasonable variations; reducing model size directly lowers capacity. They can be combined, but more methods does not automatically mean better. Overly strong stacked constraints may prevent the model from learning sufficiently.

Generalization issues also cannot be judged only by the gap between training loss and validation loss. If the training and validation sets come from different distributions, poor validation performance may reflect distribution shift rather than ordinary overfitting; if label quality is poor, the model may be incorrectly supervised; if classes are severely imbalanced, overall accuracy alone may hide failures on minority classes. Therefore, evaluating generalization requires choosing metrics aligned with task costs and examining performance on different data slices.

“Good performance on the test set” only provides evidence of generalization under the premise that the test set can represent future data. Deployment environments change over time; input devices, user behavior, language expressions, or business rules may all change. Real systems also need to monitor data drift, performance changes, and error types, rather than treating one offline test as a permanent guarantee.

The core logic of this section is:The goal of training is not to memorize the training set to the extreme but to extract stable patterns from limited samples that can be used on new data. The training set is responsible for learning parameters, the validation set for selecting solutions, and the test set for final evaluation; early stopping, weight decay, Dropout, data augmentation, and capacity control suppress overfitting from different directions, but reliable generalization also depends on leakage-free data splitting, appropriate evaluation metrics, and the representativeness of test data for the real environment.

12Connecting the Whole Causal ChainSynthesis

The previous sections have separately explained the components and training mechanisms of neural networks, but truly understanding them requires seeing that these concepts are not independent terms. Each link solves the problem left by the previous link and provides the condition for the next.The whole chain starts from the numerical representation of real-world objects, goes through representation, prediction, error measurement, and parameter optimization, and finally must be tested on unseen data.

The first step is to represent real-world objects as numerical inputs x. Images can be represented as pixel arrays, sound can be represented as sample values or spectra, text can be converted into tokens and vectors, and tabular objects consist of multiple numerical and categorical features. The network cannot directly process abstract objects like “cat,” “a sentence,” or “credit risk”; it can only perform operations on numbers. Therefore, the input representation determines what information the model can access and also limits what it can learn.

If the input representation omits key information, the later network cannot recover it from nothing. For example, keeping only the average brightness of an image but requiring shape recognition means the information needed for the task has already been lost. Conversely, if the training input contains information that does not exist at deployment, the model may receive a falsely high score. This shows that data representation and data pipelines are not external chores outside the network, but the starting point of the entire learning chain.

After having the input, the simplest method is to perform a weighted sum:

It can form predictions based on the importance of different features, but it can only express linear or affine relationships. In classification problems, this means the decision boundary is a straight line, plane, or high-dimensional hyperplane. Faced with complex patterns such as ring-shaped boundaries, image structures, and language relationships, a linear model is usually insufficient, so it is necessary to automatically construct more useful intermediate features.

Neural networks accomplish this construction in parallel with many nodes. Each node computes a weighted sum and an activation:

The weights determine which directions in the input a node cares about, the bias adjusts the triggering condition, and the activation function determines how the response is passed on. A single node can only provide one limited response, while many nodes can form multiple feature directions at the same time; their outputs are concatenated into a new vector and become the input for the next layer.

Having nodes and layers alone is not enough. If each layer only performs affine transformations, multiple layers can still be collapsed into one, and depth will not increase the types of functions. Activation functions between layers break this linear collapse. Taking ReLU as an example, different nodes activate in different regions, causing the network to form many local pieces with different slopes; through multi-layer composition, these pieces can construct complex nonlinear mappings.

Therefore, the work of hidden layers can be summarized as representation learning. Each layer re-encodes the information from the previous layer into a coordinate system that is more useful for the final task. Early layers do not necessarily correspond to human-nameable concepts, but as long as these representations help later layers reduce task error, they have training value. Depth provides multi-level function composition, and width allows more patterns to be detected in parallel at the same level.

With the parameters fixed, forward propagation computes from input to output along the network structure:

Hidden layers gradually form internal representations, and the output layer then translates the final internal numerical values into the answers required by the task. Regression may directly output real numbers, binary classification may output a value between 0 and 1 through Sigmoid, mutually exclusive multi-class classification may form a category distribution through Softmax, and multi-label tasks may use an independent Sigmoid for each label.

Up to this point, the network can only give an answer but cannot judge whether the answer is correct. The loss function converts the prediction y-hat and the true target y into a scalar L. Regression can use mean squared error, and classification often uses cross-entropy. This scalar both makes different parameter states comparable and provides a clear objective for differentiation. Without a loss function, there is no mathematical definition of “getting better” for the optimizer to execute.

But there is only one loss number, while network parameters may be many. Backpropagation uses the chain rule to propagate sensitivity from the loss to earlier layers and obtain:

Each component of the gradient indicates how the loss changes near the current position when the corresponding parameter changes slightly. Backpropagation reuses the common downstream influence, so a single backward pass can efficiently obtain the gradients of all parameters. It is responsible for computing gradients, not for actually modifying parameters.

The optimizer reads the gradients and updates the parameters. The most basic gradient descent is:

The gradient points in the direction of steepest local ascent, and the negative gradient gives the direction of steepest local descent. The learning rate η controls the step size. Optimizers such as Adam further use historical gradient and scale information, but they still rely on the gradients provided by backpropagation.

One training step therefore forms an internal loop:

The updated parameters will change the next forward propagation; new predictions produce new losses; new losses produce new gradients. Thousands of mini-batches repeatedly pass through this loop, hidden-layer representations and output behavior change together, and training loss usually decreases gradually. This is the direct mechanism by which the network “learns.”

The training loop also depends on engineering conditions. Random initialization breaks node symmetry; appropriate weight scales, activation functions, and normalization help signals propagate stably; residual connections provide a more direct path for deep networks; gradient clipping can limit extreme updates. If the data is wrong, gradients break, the learning rate is improper, or numerical overflow occurs, actual training may still fail even if the theoretical chain is correct.

However, a decrease in training loss only proves that the optimizer has found parameters more suitable for the training data. Because the network has large capacity, it may simultaneously learn stable patterns, accidental correlations, label noise, and even directly memorize samples. The training set alone cannot determine which of these components will be useful for future data.

Therefore, the validation set must be at the end of the closed loop. The training set provides the parameter update signal, while the validation set does not participate in gradient updates but instead checks whether the current model also improves on data not used for training. If training loss decreases and validation loss also decreases, it means the current learning at least has some transfer value; if training continues to improve while validation worsens, overfitting may be occurring, and early stopping, regularization, data augmentation, capacity adjustment, or re-examining the data distribution should be considered.

The test set, after the architecture, hyperparameters, and training strategy are determined, provides a more independent evaluation of the final solution. The validation set is used for development selection, and the test set is used for final estimation. If test results are repeatedly viewed and the model is modified accordingly, the test set gradually participates in selection and loses its original independence.

The entire causal chain can be written as:

1. Real-world objects
2. Numerical inputs
3. Multi-layer nonlinear representations
4. Task outputs
5. Loss
6. Gradient
7. Parameter updates
8. Testing on unseen data

This chain also provides a troubleshooting order. If the model fails, you can ask in the reverse direction: Does the evaluation data represent the real task? Are training and validation distributions consistent and free of leakage? Does the loss match the meaning of the output? Do gradients exist and have reasonable scale? Are parameters actually updated? Do activations and inputs retain useful information? The problem may occur at any link, so you cannot directly blame the network structure just because the results are poor.

Different levels of success should also be judged separately. If the network structure can represent the target relationship, it means the expressive capacity may be sufficient; if the training loss can decrease, it means the optimization chain is at least working; good validation performance provides evidence of generalization; sustained effectiveness after deployment also requires the real environment to be sufficiently consistent with the offline data and requires continuous monitoring. These levels increase step by step, and success at the previous level cannot replace checking the next level.

Therefore, up to the point where “training loss becomes smaller,” the closed loop is still incomplete. The real goal has never been to make the model get high scores on known examples, but to make it form a mapping from known samples that is useful for unknown samples.The complete logic of neural networks is: use multi-layer nonlinear functions to gain expressive capacity, use loss and backpropagation to gain trainability, use the optimizer to form parameters from data, and finally use unseen data to judge whether what is learned is a generalizable pattern or a memorization of training samples.

13Concept Dependencies and Extended LearningPath

Neural networks involve many terms. If you learn by jumping around according to popularity, it is easy to know many names but not be able to explain the causal relationships between them. A more stable route is to divide knowledge into dependency layers: first master the mathematical and machine learning language used by the network, then understand the core computational loop, then dive into training engineering, expand to specific architectures, and finally handle trustworthiness and deployment issues.

The first layer is prerequisite knowledge. Features and labels determine how a supervised learning problem is formulated: feature x is the input information the model can see, and label y is the target the model is expected to predict during training. Understanding this distinction allows you to see why the network uses x to compute y-hat, and why it needs to use y to compute the loss.

Linear regression is the starting point for understanding weighted sums and continuous value prediction:

Logistic regression further demonstrates how to convert a linear score into a binary classification output through Sigmoid, and how to establish a training objective with cross-entropy. Neural networks can be viewed as adding multiple layers of learnable nonlinear representations before such simple models; therefore, if linear models and logistic regression are not yet clear, the meaning of hidden layers easily becomes pure memorization.

Vectors and matrices are the language for describing parallel computation across an entire layer. There is no need to master all linear algebra theory at the beginning, but you should at least understand that a vector represents a group of features, each row of a matrix can represent the weights of a single node, and matrix multiplication Wx simultaneously computes the weighted sums of many nodes and can check whether the shapes of the input, weights, and output match.

Derivatives and partial derivatives are the foundation for understanding training. A derivative describes how one quantity changes when another changes slightly; partial derivatives apply this idea to functions of multiple parameters; the gradient then assembles the partial derivatives of all parameters into a vector. You also need to understand the chain rule, because early weights do not directly affect the loss; their influence propagates step by step through pre-activations, activations, subsequent layers, and predictions.

The second layer is the core loop of this page. Nodes use weights and biases to form pre-activation values:

The activation function transforms it into a nonlinear representation:

Many nodes form hidden layers, and multiple hidden layers create function composition. Forward propagation computes from input to output along these functions; the output layer converts internal values into predictions according to the task, and the loss function converts prediction errors into a scalar. Backpropagation then uses the chain rule to compute gradients for all parameters, and the optimizer updates weights and biases based on the gradients.

This core layer should not be learned as eight unrelated definitions. You should be able to explain why each concept arises: weights are used to combine inputs, biases are used to shift response thresholds, multiple nodes are used to form different responses in parallel, activation functions prevent multiple layers from collapsing into a single layer, loss provides an optimizable objective, backpropagation efficiently computes responsibility, and gradient descent adjusts parameters based on local slope.

The third layer is training in-depth. Initialization not only gives parameters a starting point but also breaks symmetry among nodes and controls the scale of forward signals and backward gradients. Xavier and He initialization design appropriate variances for different activations and layer widths. After understanding initialization, you can explain the risks of node symmetry or gradient blockage that all-zero weights may bring: only when the relevant weights, biases, downstream connections, optimizer states, and update rules all maintain symmetry among nodes will these nodes continue to give identical responses and receive identical updates. Merely having all weights zero does not mean the nodes will permanently learn identical content; overly large or small initial values can also cause numerical issues.

Mini-batch training is a compromise between full-data gradient and single-sample stochastic gradient. Using only a small batch of samples at a time leverages hardware parallelism and updates parameters frequently. Batch size affects gradient noise, memory usage, and training speed, so it is not just a data-loading setting but also an optimization hyperparameter.

Adam, momentum, and learning rate scheduling are further developments beyond basic gradient descent. Momentum accumulates historical directions to help reduce back-and-forth oscillations in narrow valleys; Adam adjusts effective step sizes for different parameters based on gradient statistics; learning rate scheduling allows larger steps for exploration early in training and smaller steps for refinement later. When learning these methods, always keep the common thread: backpropagation provides gradients, and the optimizer decides how to use them.

Normalization, regularization, and overfitting form another group of knowledge for training stability and generalization. Batch normalization or layer normalization controls intermediate numerical scales; weight decay, Dropout, data augmentation, and early stopping limit the model from overfitting training details; train, validation, and test splits are used to distinguish parameter learning, scheme selection, and final evaluation. They answer not just “can the loss decrease?” but “is training stable, and can the decrease transfer to unseen data?”

The fourth layer is architecture extension. Convolutional neural networks use local connections and parameter sharing to process data with spatial structure, such as images. They still perform linear operations, activation, loss, and backpropagation, but replace the fully connected matrix with convolution operations that better match spatial structure.

Recurrent neural networks preserve hidden states over time for sequences, allowing early inputs to influence later outputs, but they also bring gradient problems along long paths. Attention mechanisms allow different positions to directly establish weighted connections based on content, reducing the restriction that information must pass step by step through the entire sequence. Transformers build deep architectures from components such as attention, feedforward networks, residual connections, and layer normalization.

Residual networks focus on solving the problem that deep models are difficult to optimize, by:

This makes several layers learn corrections to the input and provides a more direct path for signals and gradients. These architectures look different, but they do not depart from the principles on this page: they are still compositions of parameterized functions, still produce predictions through forward propagation, define objectives through loss, and learn parameters through backpropagation and optimizers.

Therefore, when learning a new architecture, the most effective question is not “what modules does it have?” but rather: what structural assumptions does it impose on the input; how does information flow between modules; where are parameters shared; where does nonlinearity arise; how are outputs and losses defined; which paths do gradients propagate through; and what limitations of ordinary fully connected networks does it address?

The fifth layer is trustworthiness and deployment. Interpretability studies why a model makes a certain judgment, but internal activations do not naturally correspond one-to-one with human concepts, so interpretation methods themselves also need validation. Calibration concerns whether the confidence a model gives matches the true accuracy; a model with good accuracy may still be overconfident, affecting risk decisions and threshold selection.

Distribution shift occurs when deployment inputs are no longer from the same distribution as the training data, such as changes in user groups, devices, time, or business rules. Robustness concerns whether the model is stable in the face of noise, perturbations, missing information, or adversarial inputs. These issues remind us: test set performance is only valid when the test data represents the future environment.

Inference latency, throughput, memory, energy consumption, and model size determine whether the model can actually be used. Model compression can include quantization, pruning, distillation, or structural modification, but the compressed model still needs to be revalidated for accuracy, calibration, and performance on different data slices. A model with excellent offline metrics that cannot run on the target device within the time limit is not a complete engineering solution.

This path does not require finishing all details of the previous layer before touching the next; it explains which layer to return to when encountering difficulties. If you cannot understand the tensor computations of convolutions and attention, go back to vectors, matrices, and shapes; if you do not understand optimizer differences, go back to gradients and learning rates; if you do not understand deep training problems, go back to the chain rule; if the model performs well offline but fails online, inspect data distribution, calibration, and deployment constraints.

The true core passing criterion can be checked with one complete derivation. Given:

you should be able to explain the roles of weights and biases, compute the activation a=φ(z), then forward propagate the multi-layer results to the output y-hat, choose the loss according to the task L(y-hat,y), use the chain rule to explain ∂ L/∂ W how it is obtained, and then write out the parameter update:

At the same time, you should also be able to explain why multiple layers without nonlinearity are still equivalent to a single layer, why gradients only provide local directions, and why a decrease in training loss does not equal generalization success. If you can explain these things naturally through causal relationships rather than reciting them sentence by sentence, you have crossed the threshold of neural network principles.

What this section ultimately provides is not a rigidly executed curriculum but a dependency tree:Linear models, matrices, and derivatives provide the language; nodes, activations, loss, backpropagation, and gradient descent form the core loop; initialization, optimizers, normalization, and regularization make the loop work stably; convolutional networks, recurrent networks, attention, Transformer, and residual networks organize the same principles into architectures suitable for different structures; interpretability, calibration, distribution shift, and deployment constraints determine whether the trained model can be used reliably.