Batch Normalization: Stabilizing Channel Scale with Batch Statistics

A single forward transformation of BatchNorm takes a set of batch activations x as input and outputs y with the same shape. What it addresses is not “making each sample individually standardized,” but the problem that during training the activation scale of the same channel keeps changing, making it hard for the optimizer to maintain a stable step size over time. For the current statistical set B, BatchNorm first estimates that channel’s mean and variance, then converts each activation into a standardized position relative to this group of data, and finally applies learnable scale and shift.

Suppose B contains m activations participating in the statistics; the batch mean and batch variance are:

μ_B = (1/m)Σᵢxᵢ

σ²_B = (1/m)Σᵢ(xᵢ − μ_B)²

Here, xᵢ is the i-th input activation. First compute μ_B to obtain the center of the current set; then compute the deviation of each activation from the center, square it, and average to obtain σ²_B. Then for each activation apply:

yᵢ = γ(xᵢ − μ_B)/$\sqrt{\text{σ²\_B + ε}}$ + β

The numerator xᵢ − μ_B represents the direction and magnitude of this activation’s deviation from the current batch center; the denominator $\sqrt{\text{σ²\_B + ε}}$ uses the current batch’s scale to correct this deviation. ε is used to limit excessive amplification caused by division when the variance is very small. The standardized result is then multiplied by the per-channel learnable parameter γ and added to the per-channel learnable parameter β, allowing the network to choose a suitable output scale and center again according to the task.

Therefore, yᵢ should be interpreted as “the position of xᵢ relative to the current statistical set, followed by a channel affine transformation,” rather than an absolute score of that sample itself unaffected by the environment. When the same activation appears in different training batches, changes in the surrounding members cause μ_B and σ²_B to change, and its output changes accordingly. An element not only directly determines its own numerator, but also participates in the calculation of the mean and variance, thereby indirectly affecting the outputs of all elements in the set.

“Mean 0 and variance 1” applies only to the standardized values before γ and β, and the finite ε and finite batch will cause the actual statistics to deviate slightly. After γ and β, the mean and variance of the output y no longer need to remain 0 and 1. When the batch is small, the members are highly correlated, or the distribution of the set is abnormal, the representativeness of μ_B and σ²_B for the target distribution decreases; at this time treating the output as a stable relative standard score is no longer reliable.

The statistical unit of convolutional BatchNorm is the “channel,” not the entire tensor. For an input of shape [N,C,H,W], first fix a channel c, then gather all activations for this channel across the batch axis N, the height axis H, and the width axis W, for a total of N×H×W values. From these values compute this channel’s own mean μ_c and variance σ²_c; channels are not mixed with each other. This preserves the separate semantics and scales of different channels, without using a global mean to force them into a single statistical system.

After normalization, all positions in the same channel share one set of learnable parameters γ_c and β_c. These are broadcast along N, H, and W, applying the same scaling and shifting to every normalized activation in that channel. Thus the output shape remains [N,C,H,W], and each output position represents its deviation from the channel’s current statistical distribution, followed by the channel’s affine transformation.

Each channel needs only one γ_c and one β_c, so the total number of learnable parameters is 2C, not 2NCHW. Correspondingly, the running mean and running variance each store one value per channel, so there are C values of each. This count arises from “independent per-channel statistics and parameter sharing within a channel,” not from the number of spatial locations.

Different input types follow the same principle, but the axes used in statistics are determined by tensor semantics:

Axis choice cannot be mechanically copied from dimension indices. For example, after the data layout is changed to NHWC, the channel is in the last dimension; during statistics you must still fix the channel and aggregate across batch and spatial positions, only the corresponding axis indices change. A fully connected input has no two-dimensional spatial axes, and a one-dimensional convolution has only the length axis L, so neither can copy the [N,C,H,W] axis settings. A reliable way to determine the statistical axes is to first identify which dimension represents the feature or channel, then aggregate over the remaining batch and spatial axes that belong to the current statistical collection.

Suppose the same channel currently has only two activation values [1,3], and take affine parameters γ = 2, β = 0.5. In one BatchNorm forward pass, first compute shared batch statistics from these two values, then standardize each one, and finally use the same γ, β to adjust the channel scale. To highlight the main calculation, ε is ignored here.

The batch mean is:

μ_B = (1 + 3)/2 = 2

It gives the center of the current two activations. The batch variance averages the squared distances of the two values from the mean:

σ²_B = ((1 − 2)² + (3 − 2)²)/2 = (1 + 1)/2 = 1

Therefore the standard deviation is $\sqrt{\text{σ²\_B}}$ = 1. After subtracting the mean and dividing by the standard deviation, the two inputs become:

x̂ = (x − 2)/1 = [−1,1]

Here, −1 and 1 indicate that the two activations lie on opposite sides of the current batch mean with equal deviations. Next, apply the channel affine transformation:

y = γx̂ + β = 2x̂ + 0.5

For the first standardized value, y₁ = 2×(−1) + 0.5 = −1.5; for the second standardized value, y₂ = 2×1 + 0.5 = 2.5. The final output is [−1.5,2.5].

The standardization-stage result [−1,1] has zero mean and unit variance over the current batch, but the affine-transformed [−1.5,2.5] no longer needs to satisfy this property. γ = 2 doubles the standardized scale, and β = 0.5 then shifts it as a whole, allowing the network to reselect a suitable linear scale for this channel after standardization. If the task requires a near-identity transformation, the network can also learn γ and β that match the current statistics; however, γ and β can only adjust the standardized result and cannot remove the dependence caused by the mean and variance being jointly estimated from the current batch members.

BatchNorm follows two different data paths during training and inference. The two paths use the same input, output shapes, and channel affine parameters, but the statistics read by normalization differ: the training path uses the current batch's μ_B, σ²_B, and simultaneously updates the running mean μ_run and running variance σ²_run; the inference path no longer re-estimates statistics from the current request, but instead uses the frozen μ_run, σ²_run.

During training, the current batch is both the data to be transformed and the source of statistics. After a batch is input, BatchNorm first calculates μ_B and σ²_B for each channel in this batch and produces the training output accordingly; at the same time, it incorporates the current batch statistics into the running state, accumulating reusable scale information for future inference. The running statistics are not used to determine this training output; they are responsible for preserving state across training steps and being read by the inference path.

During inference, the input may be a single image or a request batch assembled in any way. At this point every activation is normalized by the frozen μ_run, σ²_run, and does not read the values of other samples in the current request. This switch makes the output for the same image no longer depend on which images happen to be processed together online, preventing the request batching method from becoming part of the model result. Single-image inference therefore does not need to rely on this sample itself to estimate the mean and variance.

If inference is still in training mode, after the same image is placed with a different set of neighbors, the current batch statistics change and the output also changes; when batch = 1 and the spatial dimensions are also very small, there are very few activations available for statistics, and the variance may even be close to zero. This not only causes abnormal numerical scale, but also makes online results affected by request composition. A direct diagnostic method is: in eval mode, fix the same image, change only the neighbors in the same batch, and the output should remain unchanged; if the output changes, it indicates that the implementation is still reading current batch statistics, or the training and inference modes have not been switched correctly.

The two-path protocol depends on the training phase correctly maintaining state. If the running mean and running variance are not updated, or are not correctly saved with the model after training, then even if inference switches to eval mode, the statistics state read cannot represent the scale accumulated during training, and performance may suddenly degrade. Therefore, training output, running state update, and inference state reading are three interconnected links in the same mechanism.

The running mean and running variance are state saved with the model by BatchNorm, used to compress the statistics observed batch by batch during training into a long-term reference that can be read at deployment. Deployment requests usually cannot guarantee batches that are sufficiently large and representative, so inference cannot temporarily rely on request data to estimate stable statistics; the running state accumulated during training fills this gap.

The running mean can be updated with an exponential moving average:

μ_run ← (1 − α)μ_run + αμ_B

μ_run is the running mean saved before the update, μ_B is the current batch mean, and α determines the current batch’s contribution to the new state. Each training forward pass combines the old state with the current batch’s observations according to weights, then writes the result back to μ_run. At inference, this state is read and used to center the activations. The running variance serves the corresponding scale-reference role, but the specific estimation convention should be implementation-defined.

For example, if the old running mean is 10, the current batch mean is 14, and α = 0.1, then:

New μ_run = 0.9×10 + 0.1×14 = 10.4

The new state moves only 0.4 toward the current batch, rather than directly replacing the value with 14. This recursion smooths single-batch fluctuations, but it does not guarantee an unconditionally accurate population mean. Exponential weighting makes recent batches have greater influence, so when the values are biased toward recent data, the precise meaning is that the state trusts the most recently observed distribution more.

If consecutive batches come from different classes, or use data augmentation of different strengths, the running statistics will shift toward the most recent distribution as the batches proceed. When training is too short, the state may not have accumulated sufficiently; abnormal data in the final training stage may pull the final state toward an anomalous distribution; during fine-tuning, if the state is incorrectly frozen or incorrectly updated, the saved statistics will also fail to represent the deployment data. Since inference directly consumes these values, the problem manifests as model-state mismatch, rather than merely a missing segment of training logs.

Different frameworks may use opposite perspectives when naming momentum: some use it to represent the current batch coefficient α, while others use it to represent the old state coefficient 1 − α. The same configuration number placed under different conventions produces completely different update speeds. When migrating configuration, you must check the actual update formula and align the estimation method for the running variance as well; do not just copy the momentum value or assume that the implementation conventions for the mean and variance are exactly the same.

BatchNorm keeps the shape of activation tensors unchanged, but changes the parameterization through which subsequent layers see those activations. It uses current statistics to weaken the influence of absolute channel scale, and then through learnable γ and β retains the ability to re-select scale and offset. The result is that subsequent layers usually face a more controllable numerical range, instead of having to continuously adapt to large changes in the activation scale of previous layers.

This scale reparameterization directly affects optimization difficulty. If activation scales differ greatly across channels or across training stages, parameter updates of the same size may have almost no effect in some directions while causing drastic loss changes in others, forming scale-sensitive steep regions and making the learning rate difficult to choose. After normalization converts the current activations to a relative scale, the absolute magnitude dominates subsequent computation less; γ and β then allow the model to restore the expressive range required by the task. Thus, small perturbations of parameters often correspond to smoother loss and gradient changes, making it easier for the optimizer to find effective and stable update steps.

The original paper emphasizes that BatchNorm can stabilize the distribution of layer inputs, and explains the training benefit with "reducing internal covariate shift". But "the distribution of layer inputs no longer changes" is not a guarantee BatchNorm makes about its output, nor is it sufficient alone to explain all the effects. Later understanding emphasizes mechanisms such as smoother loss and gradients with respect to parameter perturbations, and scale reparameterization making larger learning rates more usable. They describe how the optimization process benefits from changes in numerical scale, rather than claiming that intermediate activations remain static during training.

In practice, one should observe testable optimization results: whether training is faster, whether larger learning rates become usable, and whether training is more stable under different initializations. A more reliable empirical conclusion is that, in many convolutional networks and under suitable batch conditions, BatchNorm can improve optimization conditions, speed up training, and reduce initialization sensitivity.

These benefits are not theorems that hold for all models. Network architecture, optimizer, data augmentation method, and batch composition all change the statistical quality and optimization effect. Modern architectures without BatchNorm can also stabilize training with suitable initialization, residual scaling, or other normalization methods. Therefore, BatchNorm should be understood as an often effective scale-control and reparameterization mechanism whose value needs to be verified in the context of specific training conditions.

During training, the BatchNorm output for a fixed sample is determined not only by its own activation but also by the batch neighbors in the same statistics set. When placing this sample into different training batches, the sample's own input does not change, but the batch mean μ_B and batch variance σ²_B change with the neighbor composition, so the normalized representation also fluctuates slightly. This random perturbation introduced by finite batch statistics is called batch noise.

Batch noise is not an unexplained uncertain result produced by the operator under identical conditions, but a deterministic consequence of training data composition changes propagated through the statistics to the output. The causal chain can be written as: batch neighbor changes → μ_B, σ²_B changes → fixed-sample normalization scale changes → the sample's intermediate representation undergoes slight perturbation. As long as each statistics set is finite, this dependence exists.

Moderate perturbation may form implicit regularization. Because the exact activation scale of the same sample varies slightly across different batches, the network finds it harder to rely solely on a fixed value for prediction and needs to learn representations more robust to these statistical perturbations. Observing moderate representation fluctuation during training can thus be understood as training perturbation brought by the BatchNorm data path.

Reducing batch generally makes the estimates of mean and variance more susceptible to changes in members, thereby intensifying batch noise, but stronger noise does not necessarily mean better generalization. When statistical distortion exceeds what the current optimization process can tolerate, the perturbation no longer acts merely as a mild regularization signal, but first manifests as loss oscillation, training instability, and accuracy decline. Therefore, the effect of batch noise depends on the balance between perturbation strength and statistical quality.

The impact of increasing batch is not limited to improving computational throughput. A larger statistics set may weaken BatchNorm noise and change the optimization process. When comparing experimental results between large batch and small batch, it is necessary to consider four changes separately: learning rate, number of training steps, statistical quality, and implicit regularization; if they change simultaneously, the final difference cannot be attributed solely to batch size or BatchNorm noise.

To judge whether BatchNorm statistics are reliable, you cannot look only at the batch dimension N; you must look at the number of elements in each channel that actually provide approximately independent information for the mean and variance. For a convolutional tensor, a channel superficially aggregates N×H×W values, but this product is only the total number of elements involved in the computation, not the effective independent information. Whether the statistics are stable also depends on the correlation among these values.

Spatial positions can supplement statistical samples, so in early high-resolution layers, even if N is small, each channel may still have enough effective information and the batch statistics remain stable. As the network deepens, the H and W of feature maps shrink, so fewer positions can participate in the statistics; if different spatial positions are also highly correlated, the actual effective independent count will fall further below the apparent N×H×W. Therefore, even if the batch dimension N is exactly the same, the mean and variance estimates in deeper layers may be noisier than in shallower layers.

The criterion for when a small batch truly fails is therefore not some isolated N, but rather when the per-channel statistics of that layer are no longer sufficiently representative, causing the normalization scale to fluctuate significantly or deviate systematically. This judgment can be used to decide whether to synchronize statistics across devices, freeze existing BatchNorm, switch to another normalization structure, or increase the per-device batch size.

Different approaches change not only the sample count but may also change the model's inductive bias. Here inductive bias means that the normalization structure prescribes in advance which data regularities the model can more easily exploit. BatchNorm assumes that the same channel can share statistics across samples and spatial positions; GroupNorm does not depend on other samples in the same batch, but instead combines a group of channels and spatial positions within each sample to compute statistics. Because the statistical axes differ, the same activation will use different means and scales under the two methods. Directly replacing an existing BatchNorm with GroupNorm cannot guarantee that the original convolution weights and γ, β reproduce the original function, and usually requires retraining or fine-tuning.

When choosing an approach, you should consider the effective statistics and data correlations of the specific layers, rather than judging whether BatchNorm is usable based only on the global batch number.

In distributed training, “batch size” must distinguish at least three notions: the total number of samples whose gradients are accumulated for one parameter update, the per-device mini-batch size in a single forward pass, and the statistical batch that BatchNorm actually uses in a forward pass to estimate the mean and variance. They may differ numerically and also act at different stages of the training process.

Suppose training uses 8 GPUs, each with a mini-batch of 16, and performs 2 steps of gradient accumulation. The total number of samples ultimately gathered for one optimizer update is:

8×16×2 = 256

Therefore, from the optimizer's perspective, the global optimization batch is 256. However, ordinary BatchNorm must compute the statistics and output activations immediately on every forward pass. It can only see the 16 images and their spatial positions in the current mini-batch on the local GPU, so its statistical batch corresponds to 16 images, not 256 images.

The reason is that aggregation occurs at different times. Ordinary distributed training usually synchronizes gradients across devices during the backward phase, and gradient accumulation adds over time across multiple forward–backward steps; meanwhile, BatchNorm normalization has already been completed during each forward pass. Later gradient synchronization or temporal accumulation cannot trace back and rewrite the μ_B and σ²_B that were already used to generate activations. Therefore, “the optimizer sees 256 images” does not imply “BatchNorm also uses 256 images for statistics.”

If SyncBatchNorm is used, in the current forward step each device aggregates the statistics for this layer across GPUs. Under the above configuration, the set of statistics in the same step can cover:

8×16 = 128 images

But the two gradient accumulation steps occur at different times, and SyncBatchNorm still does not normally merge them into one statistical batch, so BatchNorm sees 128 images instead of 256.

When reproducing an experiment, you must separately record the global optimization batch, the per-GPU mini-batch, the number of devices, whether statistical synchronization is enabled, and the relevant spatial dimensions of each layer. Writing only “batch size = 256” can describe only one of these notions; it cannot reconstruct which data BatchNorm actually used during the forward pass, and therefore cannot accurately reproduce the normalization behavior in the experiment.

BatchNorm's running mean and running variance record the center and scale of activations for each channel in the training source domain. After the model shifts from training photos to medical images, a new camera, a new institution, or a different preprocessing pipeline, even if the network weights themselves have not immediately lost expressive power, the activation distribution produced by earlier layers can already change. Eval mode still uses the frozen source-domain μ_run and σ²_run to center and scale the target-domain activations, creating a statistical mismatch.

For example, changes in brightness, contrast, or sensor characteristics may cause many channels' activations to deviate overall from the original μ_run. If the target-domain activation center shifts, subtracting the old mean leaves a systematic bias; if the target-domain scale changes, dividing by the scale corresponding to the old variance may over-amplify or compress the representation. These biases propagate layer by layer to later representations, causing the eval-mode output to drift overall. Therefore, the failure point in domain shift may first appear as a mismatch between the BatchNorm state and the target domain, rather than the final classifier or all weights failing simultaneously.

The standardized deviation of pre-BN activations relative to running statistics can serve as direct evidence. For a given channel, one can observe how far the target-domain activations are from μ_run relative to the running scale; if many channels persistently show an overall shift, it indicates the model is interpreting target-domain data using source-domain references. This signal is closer to the start of the causal chain than merely seeing worse final predictions: input domain changes → intermediate activation distribution changes → frozen running statistics mismatch → normalized representation is shifted or scaled → subsequent prediction degradation.

If the application conditions cannot provide a reliable batch over the long term, one can also choose an architecture that does not depend on batch statistics.

Online monitoring should not retain only the final average accuracy. One can simultaneously compare the standardized deviation of pre-BN activations relative to the running mean, prediction confidence, and metrics on different data slices. Statistical drift often appears before overall accuracy drops noticeably; jointly observing intermediate statistical signals and prediction performance can earlier distinguish running-state mismatch from other types of model degradation.

Convolution–BatchNorm fusion exploits the condition that the statistics are already frozen at inference time. The convolution first produces z = Wx + b, and BatchNorm in inference mode then applies a per-channel affine transformation to z using the fixed γ, β, μ_run, σ²_run, and ε. Since the entire BN transformation no longer depends on the current input batch, it can be merged with the linear parameters of the preceding convolution, producing new W′ and b′ and thereby eliminating the separate BN operator.

Under fixed statistics, the BN computation on the convolution output is:

y = γ(z − μ_run)/$\sqrt{\text{σ²\_run + ε}}$ + β

Substituting z = Wx + b:

y = γ(Wx + b − μ_run)/$\sqrt{\text{σ²\_run + ε}}$ + β

Combining the part multiplied by x and the constant part separately gives:

W′ = γW/$\sqrt{\text{σ²\_run + ε}}$

b′ = γ(b − μ_run)/$\sqrt{\text{σ²\_run + ε}}$ + β

Thus y = W′x + b′. Here, γ, β, μ_run, and σ²_run all act per output channel: the scaling factor γ/$\sqrt{\text{σ²\_run + ε}}$is multiplied onto the convolution weights of the corresponding channel; the convolution bias b first subtracts the running mean, then is scaled by the same factor, and finally β is added. If the original convolution has no explicit bias, the fusion computation must still treat the bias as zero and produce a fused b′.

The equivalence before and after fusion holds only when the eval state, running statistics, and ε match exactly. In that case, comparing layer by layer on the same input, the output of the fused convolution should be consistent with the output of the original convolution followed by BatchNorm, allowing only acceptable floating-point rounding differences due to the implementation. Comparing only the final predicted class is not sufficient for acceptance: even if intermediate values have diverged, the final class may coincidentally be the same, masking parameter or broadcast errors.

After fusion, the separate normalization operator is omitted, but the new W′ and b′ have baked the frozen statistics into the convolution parameters, so they cannot be treated as the original training structure for continued training. Omitting the convolution bias, applying channel scaling along the wrong axis, using a different ε, or reading mismatched running statistics during fusion will all break equivalence. After entering the quantization pipeline, the channel weight scales have been redistributed, so it is also necessary to re-check the quantization scale and calibration results; one cannot simply assume that the quantized model is still equivalent based on the floating-point fusion formula.

BatchNorm's acceptance object is not a tensor operator that passes as long as its shape is correct, but a protocol composed of numerical transformation, training and inference mode switching, and persistent running state. Test inputs should cover hand-computable activations, different batch neighbors, model states before and after saving, multi-GPU training configurations, and deployment domain slices; outputs should not only look at final task metrics but also record per-layer numerical errors and state consistency. Only in this way can implementations with 'correct output shape but wrong statistics axes, modes, or states' be discovered.

Validation should start from the smallest computable case. For the same-channel input [1,3], set γ = 2 and β = 0.5, and ignore ε as agreed, the correct output should be [−1.5,2.5]. This example can establish the basic formulas for mean, variance, normalization, and affine transformation. Then check the statistical axes: compute the mean, variance, and output separately for each channel, and compare item by item with a reference implementation to avoid incorrectly aggregating across channels or broadcasting along the wrong dimension.

Mode switching should be verified through batch-neighbor experiments. Fix one sample; when changing the same-batch neighbors in train mode, the current batch statistics change, and the output may change accordingly; after switching to eval mode, the output for the same sample must not be affected by the neighbors. Any eval neighbor dependence indicates that the inference path still reads current batch statistics or that the mode switch has not taken effect, and should be regarded as a blocking fault.

Running state needs to be audited as part of the model. After the model is saved and reloaded, the running mean, running variance, and batch count should be exactly consistent. Comparing only weights or final predictions is not sufficient to prove correct serialization, because state differences may only appear under specific data distributions. After loading, as long as these values change unexpectedly, it means the statistical reference that inference depends on has been altered, which is also a blocking issue.

Statistical quality should be observed layer by layer, not just by recording a single global batch number. You can record, layer by layer, the N×H×W that nominally participates in the statistics, the statistical noise, and task metrics, to determine which layers become unstable after feature maps shrink or correlations increase. In a multi-GPU environment, you should also separately run the single-GPU, ordinary multi-GPU, and SyncBatchNorm paths to clarify the statistical scope actually used in each configuration, and avoid mistaking the gradient aggregation scale for the BatchNorm statistics scale.

Deployment optimization requires separately verifying convolution-BN fusion. Use multiple batches of inputs to compare the per-layer maximum error before and after fusion, and also check the final task metrics; the per-layer results should only differ by permitted floating-point rounding differences. Final class consistency cannot replace per-layer equivalence, because intermediate deviations may not yet change the class of the current test sample but may already break functional equivalence.

Domain stability should be observed through meaningful data slices. For new devices, different augmentations, seasonal changes, or different institutions, separately inspect intermediate statistical drift and task performance, rather than only aggregating them into a single average metric. The complete acceptance order starts from the minimum formulas and statistical axes, then covers train/eval, multi-GPU statistics, state serialization, fusion equivalence, and deployment domain slices in turn, so that each type of failure can be localized to the corresponding data path or model state.

The relationships between BatchNorm and other topics can be developed along five directions: “statistical dependencies, tensor semantics, network structure, training noise, and distributed execution.” Each direction changes or explains a key condition in the BatchNorm mechanism, among which the normalization denominator $\sqrt{\text{σ\_B² + ε}}$ The statistical source and usage of this run through these connections.

The single-sample statistics direction can be extended to LayerNorm and RMSNorm. The focus is on which scale-control capabilities normalization retains after the participation of other samples in the same batch is removed from the statistics, and which properties of cross-sample, cross-space shared statistics are lost. This contrast helps to understand that “whether it depends on the batch” is not an implementation detail, but a choice by the normalization structure about how information is aggregated.

The visual backbone direction requires connecting to convolutional neural networks. The channel axis and spatial axes in convolutional activations come from the network's organization of features and positions; only then can BatchNorm fix the channel dimension and compute statistics across batches and spatial positions. Understanding the semantics of convolutional tensors is the prerequisite for correctly deriving the statistical axes from the shape.

The deep shortcuts direction can continue with residual connections. Placing BatchNorm at different positions in a residual block changes the numerical path of the main branch and the shortcut branch before and after addition, thereby affecting training behavior. Here normalization needs to be viewed as part of the computational order of the residual block, rather than as an independent layer that can be moved arbitrarily while its function remains unchanged.

The noise and generalization direction corresponds to regularization. BatchNorm's batch noise comes from the finite-batch statistics changing with batch membership, while explicit regularization is applied directly by the training configuration. When analyzing the two, one should focus on how they jointly change training perturbations and generalization, and cannot attribute all regularization effects to BatchNorm or to a specific explicit term.

The multi-device perspective direction connects to distributed training. The key distinction is that gradient synchronization and statistics synchronization occur at different stages: the former aggregates backward updates, while the latter determines which activations are used to compute the mean and variance in the current forward pass. Only by separately unpacking the number of devices, micro-batches per device, gradient accumulation, and statistics synchronization can the behavior of BatchNorm in a distributed environment be accurately explained.

After mastering these dependencies, one should be able to determine the statistical axes from tensor shapes and semantics, compute the standardization and affine outputs in the training state, describe how batch statistics are updated to the running state and how inference reads that state, and be able to locate deployment drift through batch-neighbor dependencies, state saving and loading, multi-device statistics perspectives, and layer-wise equivalence before and after fusion.