Transformer

Attention solves "how different positions exchange information": a given position can, based on relevance, read and aggregate content from other positions. But this information-passing mechanism alone is not sufficient to constitute a deep model capable of learning and processing language layer by layer. The model also needs a structurally stable, repeatably stackable standard module, allowing each layer to receive the previous layer's representation, complete one round of information interaction and transformation, and then pass the result to the next layer.

Transformer is just such a standard module. It combines attention, feedforward networks, residual connections, and normalization into a layer: attention is responsible for passing information between positions, the feedforward network continues to process the representation obtained at each position, residual connections preserve and pass on existing information, and normalization helps each layer work at a relatively stable numerical scale. Because these components are encapsulated into a unified structure, the same kind of layer can be stacked repeatedly, causing representations to change gradually through layer-by-layer processing.

Therefore, attention and Transformer are not concepts at the same level. Attention is more like the core information exchange mechanism; Transformer is then the complete building block that packages this mechanism into a form that can be batch-assembled and trained. After the input passes through the first layer, it produces new representations, which become the input to the next layer, and so on, ultimately forming a large model composed of dozens to hundreds of standard modules.

This "stackability" does not mean that the number of layers can increase without limit. Deeper models are still constrained by optimization difficulty, GPU memory capacity, inter-device communication cost, and inference latency. Transformer provides a structural path that can systematically extend depth, not a removal of all engineering and training boundaries.

A typical Transformer layer can be divided into four functional categories based on data flow: multi-head attention, feed-forward network, residual connection, and layer normalization. In the common structure, input first passes through multi-head attention, then through the feed-forward network; the attention and feed-forward sublayers each have their own residual connection and layer normalization. Stacking such complete layers N times with the same structure yields a multilayer Transformer.

Multi-head attention handles information exchange across positions. For each position in the sequence, it gathers information from other positions based on relevance and aggregates context; “multi-head” allows this information aggregation to occur in parallel across multiple projection spaces. Its output still corresponds to the original positions, but each position’s representation now incorporates information from the context.

The feed-forward network then applies a nonlinear feature transformation independently to each position. “Independently” means that different positions use the same set of transformation parameters, but a given position does not directly read other positions at this step; cross-position information has already been incorporated into that position’s representation by attention.

Residual connections send the sublayer input directly to the output via a bypass and then add it to the sublayer’s computed result. If the sublayer function is denoted F, the basic relationship can be written as y = x + F(x). In this way, the existing representation x does not have to fully pass through the complex transformation to continue propagating backward, and gradients also get a more direct path. Residual connections are therefore an important prerequisite for stacking networks very deep.

Layer normalization keeps the numerical scale of each layer’s internal representations within a relatively stable range, reducing the risk that the numerical distribution drifts and causes training divergence as layers are stacked. Together with residual connections, it supports deep optimization: residual connections provide a direct path for information and gradients, while normalization provides a controlled numerical scale.

From a functional perspective, multi-head attention and the feed-forward network are the two main computation modules: the former passes information across positions, while the latter processes features position by position; residual connections and normalization provide structural conditions that allow these computations to be stacked repeatedly in a stable way. However, the “four essential components” describe a functional map, not a recipe that is identical word-for-word in all models. Modern implementations may use RMSNorm instead of LayerNorm, change the position of normalization before or after a sublayer, or add gating structures; when evaluating a specific implementation, focus on how these functions are implemented and the order in which data flows.

The Pre-Norm Transformer block places normalization before each sublayer; its basic form is:

r = x + Attention(Norm(x))

y = r + FFN(Norm(r))

The first equation performs the attention sublayer and the first residual addition, and the second equation performs the feedforward sublayer and the second residual addition. Both residual paths preserve the backbone representation before entering the corresponding sublayer, while the sublayer only needs to write a new change onto the backbone.

You can hand-calculate this data flow using the two-dimensional toy vector x = [1, 2]. Here we ignore ε in the normalization and set the affine parameters γ = 1 and β = 0. After centering the two components of x around their mean and normalizing by scale, we get Norm(x) = [−1, 1]. This example only illustrates the structure and does not represent real model parameters or weights.

First process the attention sublayer. Suppose that after multi-head attention aggregates information based on Norm(x) and the context, it produces the result A = [0.4, −0.2] for the current token. The first residual addition is:

r = x + A = [1, 2] + [0.4, −0.2] = [1.4, 1.8]

A represents the information aggregated from other positions and written into the current token; the resulting r after addition retains the original input x and also superimposes the change introduced by attention.

Next, normalize r again. Under the same simplified setting, Norm(r) = [−1, 1]. Suppose the feedforward network performs a position-wise transformation on this normalized representation, giving:

F(Norm(r)) = [0.3, 0.5]

The second residual addition then gives the final output:

y = r + F(Norm(r)) = [1.4, 1.8] + [0.3, 0.5] = [1.7, 2.3]

The final vector [1.7, 2.3] can be understood as the result of three parts gradually accumulating along the backbone: the original representation is first preserved, attention writes cross-position aggregated information, and the feedforward network then independently processes the existing representation at the current position. A real network performs this computation on all tokens in parallel, and multi-head attention actually produces A for each position.

When reading structure diagrams, you must confirm the position of normalization. This example uses Pre-Norm, namely x + Sublayer(Norm(x)); the original paper often writes Post-Norm, namely Norm(x + Sublayer(x)). The input and output tensor shapes of the two structures can be the same, but their gradient paths are different, so you cannot regard them as equivalent based only on shape, nor can you freely swap them on already trained weights.

The training of deep Transformers depends not only on what attention and feed-forward networks can do, but also on whether information and gradients can pass through many layers while remaining controllable. When many nonlinear layers are directly connected in series, the gradient in backpropagation must be continuously multiplied through the derivatives of each layer; these products may rapidly become smaller or may also continuously grow. At the same time, the scale of activations in forward propagation may also drift with depth, becoming too large or too small layer by layer. Residual connections and normalization address these two types of conditions respectively.

A residual connection writes a sublayer as y = x + F(x). Here, F(x) is the change computed by the attention or feed-forward sublayer, and x is added unchanged to the output along the bypass path. During backpropagation, the derivative of y with respect to x includes a direct path from the identity mapping, so the gradient does not have to rely entirely on the successive product of derivatives of all nonlinear transformations to propagate to shallower layers. This 'direct' path reduces the risk of gradient vanishing in deep networks caused by repeated multiplication, and also allows each sublayer to learn an increment on top of the existing representation, rather than rebuilding all information from scratch.

Normalization deals with numerical scale. It pulls the distribution of each layer's representation back to a relatively controlled range, reducing the tendency for activations to become larger or smaller during continuous propagation. In this way, the input scale received by subsequent sublayers is more stable, and training is less likely to diverge due to numerical loss of control. Residual connections focus on the information and gradient path, while normalization focuses on the representation scale; the two have different effects but together determine whether a deep stack is easy to optimize.

These effects come from the complete structure and training recipe, rather than from a particular component name that can never be replaced. Some architectures implement the normalization function with RMSNorm, or use gated residual connections, specialized residual scaling, and other designs. They can assume similar responsibilities, but the specific gradient paths and numerical behavior may still differ.

Therefore, removing residual connections or changing the position of normalization is not a local, inconsequential replacement. Such changes significantly alter the deep optimization conditions, and often require re-designing together with initialization, learning rate, and residual scaling. After a validated deep training recipe has been modified, one cannot assume the new structure is equivalent to the original merely because tensor shapes still match.

Consider fully connected self-attention without positional representations, causal masks, or other order signals: every position can read all positions, and information aggregation depends only on token content. Under these conditions, self-attention is equivariant to input permutations: if you rearrange input tokens in some order, the output will be rearranged in the same way. The mechanism moves computed results along with the permutation, but cannot determine the original order based on content alone.

Under the above no-order-signal conditions, “Xiaoming hits Xiaohong” and “Xiaohong hits Xiaoming” contain the same set of words. Attention aggregates the same content for the same word, and the output only moves with the word; but who hits whom in the sentence has changed. To make these representations reflect the order of words, an order signal must be introduced.

The position mechanism receives each token’s position index or the relative distance between tokens and then produces an order signal. This signal can be added to the input representations or applied to the attention scores. After adding position information, the model no longer computes only “whether the content of two tokens is related”; it can also use information such as “where they are located, who comes before whom, and how far apart they are.” Thus, even if two inputs contain the same set of words, different permutations can form different relational representations.

The word-order example in Figure 2, the text in the figure, and the caption all apply only to the fully connected self-attention described above without any order signal. “Same result” should be understood as the output for the same word being rearranged along with it, not that the output values at each fixed position remain unchanged. In this example, adding positional encoding to the input embeddings allows the representation to use the position of the word.

The causal decoder in Section 6 does not satisfy the above premise. The fixed causal mask restricts the visible range by position: if tokens are rearranged while the mask remains unchanged, the prefix that the same token can read usually changes, and the output generally will not just be the same rearrangement of the original output. Therefore, one cannot infer from “no explicit positional encoding” that a causal decoder cannot distinguish word order; the mask itself already imposes order constraints.

What positional mechanisms provide is an order signal available for learning, not a guarantee that the model automatically masters all order relationships. Whether the model can correctly use long-distance positional information still depends on training and evaluation. The design of positional encoding also affects whether the model can extrapolate to longer sequences unseen during training, so it is also one of the technical challenges that long-context models need to address.

The original Transformer consists of an encoder stack and a decoder stack connected together. The encoder transforms the input symbol sequence into a sequence of continuous representations; the decoder reads these representations and the already generated output prefix, and gradually produces subsequent outputs. When understanding this connection scheme, the key is to trace two information paths: which positions each sublayer is allowed to see, and how the decoder reads the encoder's results.

The encoder receives embeddings and positional representations of the input tokens. Encoder self-attention does not have the causal masking used in the decoder, so each input position in a layer can aggregate information from other positions in the input sequence; after attention, feed-forward, residual connections, and normalization, it outputs a sequence of continuous representations of the same length, to be read by the next layer or the decoder.

The decoder first applies masked self-attention to the right-shifted output sequence. The mask prevents a position from attending to outputs to its right that have not yet been produced, so during training the prediction at that position cannot peek at future tokens; during generation it can only continue based on the existing prefix. After a linear mapping and softmax, the decoder gives a probability distribution over the next output token, then incorporates the new token into the prefix and continues computing.

The two paths are connected through encoder-decoder attention: the decoder uses its current representations to form queries, and uses the encoder outputs to form keys and values, thereby selectively reading the input sequence when producing each output position. Thus, the encoder is responsible for forming input context, the masked decoder is responsible for maintaining the output prefix, and cross-attention is responsible for connecting the two into a transformation process from input sequence to output sequence.

When checking the original architecture, the data flow can be summarized as: input sequence → encoder continuous representation; right-shifted output prefix → masked decoder representation; encoder-decoder attention → writes input information into the output process. This description does not need to rely on the names or usage categories of later model families.

The original Transformer does not directly compute on text strings; instead, it first maps input and output tokens to vectors with dimension d_model, and then adds positional encoding. What is fed into the first layer is a sequence of vectors carrying order signals; what is passed between layers also remains a sequence representation of the same width.

Once inside a standard layer, self-attention aggregates information across sequence positions, the feed-forward network then transforms each position independently, and the residual connection adds the sublayer input and output. Because these operations are all organized around a unified sequence shape, the output of one layer can directly become the input of the next, which is exactly the interface condition that allows standard layers to be stacked repeatedly.

Here it is necessary to strictly distinguish between “core layers receive vector sequences” and “arbitrary raw data has been proven to be directly usable with the same architecture.” The sources available on this page only support token embedding, positional encoding, and encoder–decoder data flow in the original Transformer; they cannot be used to infer later approaches such as image patching, ViT, audio, video, proteins, or cross-modal attention.

If you want to connect the Transformer to another kind of raw object, you still need to explain how the object is segmented, how it is projected to d_model, how positional or structural signals are injected, and how the output is interpreted; these designs must be supported separately by materials from the corresponding modality. Merely saying “they can all be turned into vectors” is not enough to justify the adaptation method, parameter sharing approach, or task performance.

The historical significance of the Transformer is that it makes large-scale sequence models more suitable for training on modern accelerator hardware and provides a viable structural path for continuously scaling model size.

Classic RNNs process sequences by time steps: each later time step depends on the state of the previous time step, so many computations within the same sequence must be performed sequentially. During training, the Transformer organizes computations for multiple sequence positions into large-scale matrix operations, allowing these positions to be processed more fully in parallel. Matrix operations match the computation style of GPUs and TPUs more closely, so hardware utilization improves significantly.

This computational structure also makes it easier to combine parallelism strategies at different levels. Training can distribute different samples along the data dimension, split large computations along the tensor dimension, or arrange different layers into pipelines. Together with the stable deep stacking supported by residual connections and normalization, and a unified sequence interface that can be adopted across different modalities, scaling up parameter counts, training data, and compute investment becomes more feasible. The Transformer is therefore not merely replacing a sequence model, but opening an important path toward large-scale training.

Scalability does not mean no cost. Attention incurs significant overhead as sequence length grows; inter-device communication and GPU memory capacity constrain model and context scale. In autoregressive generation, the next token still depends on the already generated prefix, and inference cannot compute all output positions at once in parallel the way training does, so latency remains a hard constraint.

The effects of larger model scale cannot all be credited to the Transformer alone as a single module. Scaling performance also depends on training objectives, data quality, optimization methods, and hardware systems. The Transformer's key contribution is combining parallelizable training, deep stacking, and unified sequence processing to provide a foundational structure more suitable for scaling; the ultimate capabilities come from the joint effect of this structure with data, algorithms, and systems engineering.

Transformer organizes the process from token vectors to a deep trainable model into a complete causal chain. Attention first solves how information is transmitted between positions, but attention alone is not yet a complete large-model structure; it is also necessary to encapsulate the components required for information interaction, feature transformation, and deep optimization into standard layers that can be stacked repeatedly.

Each layer contains two main types of computation and two types of structural components. Multi-head attention reads and aggregates content across different positions, and the feed-forward network then performs a nonlinear transformation independently at each position; residual connections preserve the backbone representation from before entering the sublayer, and layer normalization controls the numerical scale of the sublayer input. In a single forward pass, attention first writes cross-position content into the current token, the feed-forward network then processes the features already present at this position, and the two residual additions both allow the original information to continue along the backbone.

These four types of components can form a closed loop because the model must not only be expressive but also trainable. Attention and the feed-forward network provide the actual information-processing capability; residual connections provide shorter gradient paths for backpropagation, and layer normalization reduces the risk that the input scale becomes uncontrolled with depth. The first two answer “what one layer can compute,” while the latter two answer “how such layers can be stacked deeper while remaining optimizable.”

How the order of tokens enters the computational chain also depends on the visibility range of attention. Fully connected self-attention without sequential signals such as position representations or causal masks is permutation equivariant; after adding position representations, the model can exploit sequential relations in addition to content relations. A fixed causal mask also restricts the visible prefix by position, so this equivariance conclusion cannot be generalized to all decoders that lack explicit positional encoding.

The same standard layer can form different task skeletons by changing the information visibility range and connection patterns. The encoder allows positions to use a more complete context; the decoder uses a restricted visibility range to support step-by-step generation; when the encoder and decoder are combined, the decoder can also read the encoded input through cross-attention. The architecture thus adapts to understanding, generation, and input-to-output transformation tasks.

This structure is not limited to text. As long as objects can be represented as token sequences, Transformer can process the relationships among these vectors and therefore can be extended to modalities such as images and audio. At the same time, sequence-position computation can be organized into large-scale matrix operations suited to hardware, and the deep structure also has residual connections and normalization to provide optimization conditions; together, these improve the model's scalability.

The entire chain can be summarized as: raw objects → token vectors and position representations → multi-head attention performs cross-position aggregation → the feed-forward network processes position by position → residual connections preserve the backbone, normalization controls scale → standard layers are repeatedly stacked → combined into different architectures according to visibility range. What Transformer does beyond the attention mechanism alone is precisely to package information transfer into a complete, trainable, scalable deep module. It provides key structural conditions for large-model scaling, but actual scale remains constrained by computing power, data, and system conditions.

Understanding Transformer can be expanded along the hierarchy of "prerequisite fundamentals → core structure → adjacent topics → further engineering problems." The layers are not isolated from each other; instead, they move step by step from basic representation and training mechanisms toward model scaling and practical application.

Prerequisite concepts include attention mechanism, neural network, residual connection, vanishing gradient, and embedding. Embedding explains how raw symbols become computable vectors; neural networks provide the basic framework for parameterized transformations; the attention mechanism explains how different positions select and aggregate information; residual connections and vanishing gradients help understand why deeper networks need more direct information and gradient paths. These concepts together form the foundation for understanding Transformer data flow and training conditions.

Transformer's core knowledge centers on five aspects: stackable standard layers explain how the model deepens layer by layer from uniform building blocks; the "four essential components" explains how attention, feedforward, residual connections, and normalization divide the work; positional encoding supplements sequence order information; encoder, decoder, and their combinations show how the same building blocks form different information visibility ranges and task structures; modality-agnostic nature explains why this architecture can handle token sequences from different sources.

After mastering the core structure, you can then learn Large Language Models, Scaling Laws, Multimodal Models, and Context Window. Large Language Models use Transformer for large-scale language modeling; Scaling Laws focus on the overall performance when model, data, and compute scale up; Multimodal Models extend unified sequence processing to different information types; Context Window focuses on the context range a model can handle at one time. These topics directly extend Transformer's structural capability and scale boundary.

Further learning includes Mixture of Experts (MoE), LLM Inference Optimization, Pre-training, and Fine-tuning. MoE explores different parameter organization and activation approaches; LLM Inference Optimization focuses on efficiency during the model running phase; Pre-training and Fine-tuning correspond to the training processes by which the model acquires general capabilities and adapts to specific tasks. By learning in this order, you can first establish Transformer's structural causal chain, then move into system-level issues at the scale, modality, training, and deployment layers.