Residual Connections: Preserving an Identity Information Highway for Deep Transformations

Depth degradation refers to an optimization phenomenon: when the data, training objective, and training budget are comparable, deeper models actually have higher error even on the training set than shallower models. It is different from overfitting; overfitting means training error is already low while validation performance deteriorates, whereas depth degradation indicates that the added model capacity is not effectively used by the optimization process.

Suppose a shallower network has already produced a usable intermediate representation x, and a new module is added after it. The most conservative task for this module is to output something that is still equal to x, thereby preserving the solution that the shallower network has already found; only when some change can further reduce the training loss does it need to output x + Δx. Because additional layers can in theory implement an identity mapping, the expressive capacity of the deep network includes at least the existing solution of the shallower network, so an increase in training error cannot simply be attributed to insufficient expressiveness. The real difficulty is whether the optimizer can preserve this existing solution along an easily reachable parameter path.

In ordinary stacked structures, "output x" is not an operation directly provided by the architecture; instead, it must be synthesized jointly by a chain of weights and nonlinear activations. Even if an identity mapping exists in theory, the optimizer may not be able to tune the entire chain of parameters to exactly that state; longer paths may also introduce difficulties in scale and gradients. Thus, after adding depth, although the model has more parameters, it may first lose the originally usable representation and, under given training conditions, be unable to recover it.

Residual blocks explicitly write the requirement of preserving the existing solution into the structure: a parameter-free shortcut directly passes x, while the other transformation branch only learns the correction Δx relative to the input. In this way, the output of the new module is x + Δx, and preserving the shallow network's solution only requires the new branch to be close to zero, no longer requiring multiple nonlinear layers to jointly implement an exact identity mapping. This structural change reduces the optimization difficulty of "deepening without destroying the existing solution," making the original solution easier for the training process to reach and maintain.

This mechanism improves optimizability; it does not guarantee that the newly added branch will necessarily learn useful corrections, nor does it replace data quality, regularization, or generalization evaluation. To determine whether depth degradation has occurred, the key evidence is that, under comparable conditions, the deeper model has higher training error; if only the validation error increases, this cannot be taken as the kind of depth degradation discussed here.

Suppose the complete target mapping that a module aims to implement is H(x). Ordinary parameterization makes the transformation layer directly fit H(x), whereas residual parameterization rewrites the same target as

y = x + F(x)

Therefore

F(x) = H(x) − x

Here, x is the module input, H(x) is the desired complete mapping, F(x) is the increment produced by the learnable transformation branch, and y is the module's final output. This rewriting does not change the target: as long as the branch learns H(x) − x, the result after adding the two paths is still H(x). What changes is the object the module directly fits, from “what the complete output is” to “how much the complete output should change relative to the input”.

A single forward computation contains two synchronous paths. The input x is sent through the untransformed shortcut and arrives at the addition point unchanged; the same x also enters the transformation branch to generate F(x). Finally, the two path results are added elementwise to obtain y = x + F(x). The shortcut is responsible for preserving the input, and the transformation branch only supplies the difference between the input and the target output.

This parameterization is especially natural when the target mapping is close to an identity mapping. If the existing features are already sufficient, the ideal case is H(x) ≈ x, so F(x) ≈ 0; the module only needs to make the transformation branch close to zero to preserve the input. If the existing representation only requires slight correction, H(x) = x plus a small change, the branch only needs to learn that small change. In contrast, an ordinary transformation layer, even if it does not want to change the input, still must regenerate x completely through its own parameters.

When the final layer of the branch is close to zero, F(x) is also close to zero, and the whole block initially approximately performs y ≈ x. Training can then start from this input-preserving state and gradually learn what should be added to or subtracted from the existing representation. Therefore, “learning the increment is easier” does not mean that it changes the task answer, but that it provides a more direct parameter representation for targets close to identity.

If the target mapping differs greatly from the input, F(x) = H(x) − x still holds, and the transformation branch can still learn a large residual; only the advantage of “only needing to fit a very small correction” is reduced. The residual form does not assume that all mappings must be close to identity; it simply provides the preserved term x as fixed and lets the learnable part concentrate on expressing changes relative to x.

After receiving input x, the residual block sends it to two paths simultaneously. The transformation branch computes F(x), and the identity shortcut does not pass through this new set of weights and directly carries x to the addition point. The addition point receives two tensors of the same shape and adds them element by element, yielding

y = x + F(x)

Therefore, in forward propagation there is always a direct path that does not depend on the quality of the transformation branch. It solves the following problem: in early training, the newly added branch may not yet have learned a useful transformation, or may even be temporarily very poor, but the old representation should not be forced to pass entirely through the new weights and be rewritten because of that. The identity shortcut allows input x to reach the output of the block unchanged, and then the transformation branch provides a correction.

The two paths are not an either/or choice, and there is no selection operation that turns off one of them depending on the situation. Each time the residual block is called, F(x) is still computed, and the identity shortcut also passes x; then the addition node combines the preserved term and the correction term at the same time. If F(x) = 0, the output naturally reduces to y = x; if F(x) contains useful changes, the output is updated incrementally on the basis of the original representation x. Therefore, the mechanism by which the shortcut preserves information is not skipping computation, but explicitly preserving the input term in the computation result at all times.

This path also enables existing features to continue to be reused across multiple residual blocks: each block can preserve the current representation while superimposing its own changes. However, addition requires the tensors from the two paths to have the same shape; otherwise the corresponding elements cannot be directly combined. The identity shortcut does not guarantee that input information always dominates numerically; if the scale of F(x) is much larger than x, the branch output may still overpower the preserved information. What it provides is a structurally existing preservation path, not a hard constraint on the final information proportion.

Suppose the forward relation of a residual block is y = x + F(x), the loss is L, and the gradient returned from downstream at the block output is g_y = ∂L/∂y. Backpropagation needs to compute the gradient sent to earlier layers, g_x = ∂L/∂x. The key is that the output y depends on the input x both through the shortcut term x and through the transformation branch F(x), so the chain rule must add the contributions from the two paths.

Differentiating the block output with respect to the input yields

∂y/∂x = I + J_F(x)

Here I is the identity Jacobian, coming from x appearing directly in the addition; J_F(x) is the local Jacobian of the transformation branch F with respect to the input x, describing how the branch output changes when the input changes slightly. Therefore the gradient of the loss with respect to the input is

∂L/∂x = ∂L/∂y · [I + J_F(x)]

that is,

g_x = g_y · [I + J_F(x)]

This can be understood as two gradient contributions meeting at the input: the shortcut contribution g_y · I and the transformation branch contribution g_y · J_F(x). Even if the branch's local derivative is small, the shortcut can still send one copy of the upstream gradient directly back to the input. If J_F(x) is close to zero, then a single residual block has g_x ≈ g_y, so the gradient can pass through the block approximately unchanged, rather than being forced to pass only through a very small branch derivative.

The identity term alleviates gradient attenuation in long chains, but it does not follow that "residual connections guarantee a gradient of 1". The local Jacobian of a single block is I + J_F(x), not just I; as long as J_F(x) is nonzero, it changes the gradient in different directions. When crossing multiple residual blocks, the product still involves multiplying several different I + J_F terms, and these matrices may amplify, shrink, or cancel one another. If the main path also contains normalization or shortcut projections, the local derivative of the direct path changes accordingly. Therefore, a local reading close to identity only means that the gradient passes more easily through a particular block and a particular local direction, not that the gradient throughout the entire network is always equal to 1.

Consider a simplified experiment that keeps only a single scalar direction. The input is the effective local derivative d in this direction for each block, along with the number of blocks n = 20; the output is the multiplier by which the gradient is scaled after propagating from the 20th block back to the starting point. Because the chain rule successively multiplies by each block's local derivative, if all 20 blocks take the same d, the final multiplier is d²⁰.

For ordinary stacking, if each block multiplies the gradient in this direction by 0.8, then the multiplier after 20 blocks is

0.8²⁰ ≈ 0.0115

This means the starting point receives only about 1.15% of the downstream gradient. A single-block value of 0.8 does not seem extreme, but after 20 successive multiplications, the decay is very pronounced.

For a near-identity residual block, the effective local derivative can be written as 1 plus the branch derivative. If the branch derivative is initially 0, the effective derivative of each block is 1 + 0 = 1, and after 20 blocks it is

1²⁰ = 1

In this simplified direction, the gradient multiplier remains unchanged. If the branch derivative is −0.02, the effective derivative of each block is 0.98, and after 20 successive blocks we get

0.98²⁰ ≈ 0.668

The gradient still retains about 66.8%, clearly higher than the 0.0115 after ordinary layer multiplication. If the branch derivative is +0.02, the effective derivative of each block is 1.02, then

1.02²⁰ ≈ 1.486

In this case the gradient is not attenuated but is amplified by about 1.486 times.

These numbers show that the key role of the residual structure is not to force the gradient to equal 1, but to place the effective Jacobian of each block near 1, so that deep accumulation does not suppress the signal to the order of 1% from the start. Both 0.98 and 1.02 are close to 1, but after 20 successive multiplications they still produce attenuation and amplification, respectively, so “near identity” only significantly delays harmful accumulation and does not eliminate the accumulation effect.

This experiment describes only a single scalar direction and assumes that each block shares the constant d. In a real network the Jacobian is a matrix, and the local derivatives for different layers and different directions will not all be the same. Therefore, these calculations are useful for understanding why having a single block derivative close to 1 is beneficial, but they cannot be used to assert that a residual network is stable at any depth or in any direction.

The two paths of a residual block must ultimately be added elementwise, so the shortcut output and the transformation branch output must have the same shape. As soon as the number of channels, feature width, or spatial resolution differs, corresponding elements cannot be paired directly, and y = x + F(x) cannot be used as is.

In this case, a shape-adapting mapping P can be added to the shortcut. It takes the original input x and transforms it to the same target shape as F(x), then performs

y = P(x) + F(x)

In a convolutional block, if the number of channels needs to change, a 1×1 convolution can usually complete the channel mapping; if spatial resolution also needs to be reduced, striding can be used at the same time for downsampling. In a Transformer with changing width, a linear layer can map the input to the target width. The first task of P(x) is to make the two tensors correspond elementwise at the addition point.

A projection shortcut preserves the residual interface of “computing the two paths separately and adding them at the end,” but it is no longer a strictly identity path. An identity shortcut passes x directly, has no parameters, and its backward gradients can also return along the identity mapping; a projection shortcut passes P(x), introduces learnable parameters, and backpropagation must also pass through the Jacobian of P. Therefore, projection resolves shape mismatch while losing the shortest, most direct identity backbone.

Different shortcuts have their own applicable boundaries. When shapes are the same, the strict identity shortcut x preserves a parameter-free direct path. When width or resolution genuinely changes, a linear mapping or 1×1 convolution projection P(x) can accomplish the matching. Zero padding or pooling can also adjust shapes with fewer parameters, but information and scale need to be handled carefully. Projections are best concentrated at boundaries where network stages change; if shapes are already consistent, turning every block’s shortcut into a complex network will weaken the shortcut’s meaning as the shortest path for information and gradients.

The residual formula y = x + F(x) explicitly includes x, but addition alone does not guarantee that the two paths are at similar scales. If the magnitude of F(x) is much larger than x, the output will still be dominated by the transformation branch, and the identity information may be numerically swamped; if the branch outputs of many blocks have comparable variance to the main path, successive additions may also cause activations to grow continuously with depth.

This can be observed using the norm ratio between the residual branch and the main path:

r = ‖F(x)‖ / ‖x‖

By recording the norms of F(x) and x separately on the same batch of inputs, we can obtain a scale metric that can be compared along network depth. If r is already large early in training, it indicates that branch initialization or learning rate may be too aggressive, and the result of the addition is dominated by the branch from the start; in this case, check the final-layer initialization of the branch, residual scaling, and warm-up. If the activation norm grows continuously with depth, check the accumulation of branch variance, normalization position, and depth-wise scaling. If r remains near zero for a long time, it should not be directly regarded as ideal; it may indicate insufficient branch optimization, limited capacity, or that this block indeed brings no task benefit, and it needs to be judged together with gradients, parameter update magnitudes, and actual performance.

The purpose of zero initialization is to make the network closer to an identity mapping at the start of training. You can initialize the final-layer weights or normalization scale of the residual branch near zero so that the initial F(x) is close to zero, hence y ≈ x; you can also explicitly multiply F(x) by a small residual coefficient chosen according to depth, or use depth-scaled initialization. In this way, existing representations are initially preserved through the main path, and training gradually amplifies the truly useful corrections.

Scaling is not a case of the smaller the better. An overly small coefficient suppresses the branch's output, gradients, and usable capacity at the same time, making it difficult for the branch to learn necessary changes. Therefore, when diagnosing, do not focus only on r; instead, look at the norms of the main path and branch, activation changes along depth, statistics of both paths before addition, gradients, and update magnitudes, and finally confirm whether these adjustments improve task performance. “Starting from identity” emphasizes a stable initial scale relationship, not keeping the residual branch permanently at zero.

The position of normalization in a Transformer residual block determines whether the main-path representation and main-path gradient must pass through the normalization operation. Given an input representation x, a sublayer F, and layer normalization LN, Pre-Norm and Post-Norm can be written respectively as

Pre-Norm: y = x + F(LN(x))

Post-Norm: y = LN(x + F(x))

In Pre-Norm, LN is located at the entry of the transformation branch. One path of the input x first passes through LN and then enters sublayer F, while the other path does not go through this normalization and is added directly to the sublayer output. Therefore, the x in the residual main path is closer to a strict identity pass-through; during backpropagation, gradients can also return along this main path and are less affected by normalization in the sublayer and branch. This layout usually makes very deep Transformers easier to train stably.

In Post-Norm, the shortcut term x is first added to the sublayer output F(x), and then LN is applied to the entire result. Since normalization acts on the summed overall representation, the shortcut contribution must also pass through LN, and the backward main path is likewise affected by the local derivative of LN. It can normalize the overall representation after each residual addition, but optimization often becomes more sensitive when the network is very deep.

The two layouts do not merely swap the order of one line of computation. Pre-Norm preserves a more direct x main path, while Post-Norm uniformly handles the scale after the two paths are merged, so they produce different residual outputs, gradient paths, representation scales, and convergence behavior. Post-Norm is the form of the original Transformer; Pre-Norm is often used to improve the stability of training deep networks, but the final representation properties also differ.

The choice of normalization position cannot be judged in isolation from other conditions. Model depth, initialization, residual scaling, and final normalization all jointly affect the result. That Pre-Norm provides a more direct gradient path does not mean it is necessarily better than Post-Norm under all tasks, training budgets, and settings.

In the diagram, a cross-layer connection only indicates that some information bypasses intermediate modules; you cannot judge whether it is a standard residual connection from the drawing alone. For this page, the judgment baseline that can be directly established from existing sources is: input x and the branch result F(x) after matching shapes, element-wise addition gives y=x+F(x); the identity shortcut preserves the input term, while the transformation branch still executes.

This baseline also gives a clear boundary. If a cross-layer connection does not add the shortcut term to the transformation branch element-wise, or it claims to be able to conditionally skip branch computation, then the merge semantics, execution semantics, or computational benefits cannot be inferred solely from the residual formula on this page. In that case, one should return to the architecture's own original sources for verification, rather than directly equating “there is a skip line in the diagram” with ResNet-style residual.

Therefore, the key conclusion of this section keeps only the standard residual connection itself: it performs incremental updates on same-width representations, provides short paths for information and gradients, but does not automatically skip F, nor can this be used to claim reduced inference FLOPs. The specific merge methods, design purposes, and whether modules are executed for other connection structures are not directly supported by the four existing sources on this page.

Source boundary:The table below is kept as a checklist for verification when identifying terms. Except for the ResNet residual row, the other architecture-specific descriptions require confirmation from their respective original sources and are not treated as facts already proven on this page, nor are they used to derive conclusions on this page.

To judge whether what residual connections improve is optimizability, you cannot just compare final validation scores. Score improvements may come from parameter count changes, regularization differences, or training fluctuations; therefore, you need to construct controlled comparisons and treat the training process itself as the primary evidence.

The experiment starts with two groups of equal-depth models: one group uses ordinary stacking, and the other uses residual stacking. The two groups should keep width, data, optimization method, and training budget as consistent as possible, with parameter counts matched as closely as possible. The experimental output is not just a final metric; it should include trajectories of training error over time, gradient norms and update ratios at different layers, and the scales of the residual main path and the branch. Validation scores are placed last, used to observe generalization rather than to directly substitute for optimization evidence.

First, compare training error. If the deep residual model can achieve significantly lower training error, while the ordinary deep model cannot fit the training set sufficiently, this supports that residual connections alleviate training degradation; if the difference appears only on the validation set, the benefit cannot yet be attributed to optimizability. Then perform a depth sweep, for example comparing 10, 20, 50, and 100 layers, to observe at what depth ordinary networks start to show training degradation and whether residual networks can still maintain or reduce training error as depth increases. A depth sweep can turn a one-off result into a structural pattern that varies with depth.

The training curves should also be cross-validated with internal observations. Recording the gradients and parameter update ratios at each layer can check whether the residual structure allows early layers to regain usable training signals; recording the relationship between ‖F(x)‖ and ‖x‖ can confirm that the benefit is not a superficial change caused by scale explosion in one path. If training error improves while early-layer gradients and updates also recover, and the scales of the main path and branch remain interpretable, this chain of evidence supports “the short path improves deep optimization” better than reporting the final score alone.

Ablations should also be performed on near-identity starting points. Compare zero initialization of the final layer of the branch, nonzero initialization, and different residual scalings to test whether starting with F(x) close to zero improves early stability. If the stability changes brought by zero initialization or an appropriate scaling are consistent with better training trajectories, this further supports that residual parameterization and the near-identity path do participate in optimization improvement.

Only when parameters and budgets are controlled, when ordinary networks show training degradation as depth increases while residual networks can still fit, when early-layer gradients and updates recover, and when initialization ablations show a consistent trend, is there stronger reason to rule out alternative explanations such as “just more parameters,” “just different regularization,” or random fluctuation. Even if this evidence proves improved optimizability, it does not mean performance on unseen data will necessarily be better; generalization still requires separate regularization and validation evaluation.

Ordinary deep networks, as modules are added, require each new module to reconstruct the complete representation through its own weights and nonlinearities. Forward information must pass through more local transformations in succession, and backward gradients must also propagate layer by layer along the same long chain. Even though a deeper model can theoretically preserve the solution of a shallower model, the optimizer may still struggle to make the newly added modules realize the identity mapping exactly, so increasing depth can instead cause training degradation.

A residual block rewrites the complete mapping as the sum of the input and an increment:

y = x + F(x)

The shortcut directly preserves the input x, while the transformation branch only learns the correction F(x) relative to the input. If the branch is temporarily close to zero, the forward output still has y ≈ x, so the existing representation is not forced to be rewritten just because the newly added module has not yet learned well. In this way, 'deeper but not destroying the original solution first' changes from many layers jointly reconstructing the identity mapping into keeping the newly added branch near zero.

The same addition also changes backpropagation. The Jacobian of a residual block with respect to its input contains I + J_F, where I comes from the identity shortcut and J_F from the transformation branch. Even if J_F is small, upstream gradients can still return to the input approximately unchanged through the identity term. Across many blocks, gradients still multiply by multiple I + J_F terms in succession, so residual connections are not an absolute guarantee of stability; they provide a local path closer to identity, so that depth accumulation does not have to rely entirely on the transformation branch from the start.

To make this path truly close to identity early in training, the final layer of the branch can be zero-initialized, or the residual branch can be scaled appropriately, making F(x) initially small and then allowing training to gradually amplify useful corrections. Long-term stability also depends on the scale of the branch and the trunk, the normalization position, and depth accumulation. A branch that is too strong will drown out x, while one that is too weak may fail to learn effective changes, so activations, gradients, and update magnitudes need to be monitored together.

A strictly identity shortcut requires the two paths to have the same shape. Only at stage boundaries where width or resolution changes is a projection needed to map x to a shape that can be added; although the projection preserves the interface where the two paths merge, it makes the trunk no longer strictly identity. The normalization position also changes whether the trunk must pass through additional transformations, and therefore affects the gradient path in very deep models.

Ultimately, the causal chain through which residual connections, starting from a simple addition, influence deep-network training is: the complete mapping is re-parameterized as 'preserve the input plus a learned increment'; the identity shortcut preserves forward information; the backward Jacobian gains an identity term; the near-zero branch lets many blocks start from a near-identity state; and projection, normalization, and scale control determine whether this advantage can be maintained with depth. Whether degradation is actually alleviated cannot be judged only by the final score; it must be verified together through training error trajectories, layer-wise gradient profiles, and depth ablations.

Understanding residual connections depends on several interrelated concepts. The first is vanishing gradients: only by seeing how the successive product of Jacobians along the depth direction produces exponential decay or amplification can you understand why the identity term I changes the backpropagation path, and exactly what problem “local derivatives close to 1” alleviates.

Next is normalization. How a residual block controls the scales of the two paths before and after addition affects whether the main path information can be preserved; in Transformer, Pre-Norm and Post-Norm place normalization at different positions, so whether the main path must pass through normalization and whether the gradient path is close to identity change accordingly. When continuing to study normalization, focus on the scale relationship before and after addition, not just the normalization formula itself.

Residual connections also need to be understood in the context of specific architectures. Convolutional neural networks and Transformers both form residual blocks from transformation sublayers and shortcuts, but the sublayer forms, how shapes change, and normalization positions are not the same. Drawing the two forward paths along the data flow and marking where they transform and where they merge can help identify the common structure and different implementations of standard residual blocks.

Initialization and training strategy determines whether the network can truly start from a near-identity state. Zero initialization, warmup, depth scaling, optimizers, and learning rate schedules need to be examined together: the goal is to keep the branch from being too strong early while still retaining the ability to learn effective corrections. To judge whether a configuration is appropriate, one should observe training error, gradient profiles, parameter updates, and the scale of the branch relative to the main path.

Addition is not the only way to connect across layers or across sources. When continuing to study multimodal models and other connection structures, it is necessary to distinguish what information element-wise addition, channel concatenation, and cross-modal fusion each preserve, how they change output shape, and how subsequent modules use that information. After mastering these differences, the core judgment about residual connections becomes clearer: how the two forward paths form x + F(x), why the backward Jacobian produces I + J_F, how the simplified depth multiplier is calculated, and which training observations support “optimizability is indeed improved.”