Multimodal Models

The problem that multimodal models aim to solve can be summarized in one sentence: let the same model process multiple forms of information such as images, text, and sound at the same time, rather than training a dedicated model for each form. This goal is not driven by engineering curiosity, but by two practical reasons.

The first reason is that real-world tasks themselves are often cross-modal. Image captioning requires converting visual content into language; identifying objects by sound requires linking sound signals to categorical knowledge about objects; reading illustrated documents requires understanding both text and images and clarifying their correspondence. If these modalities are completely separated and each is processed by an independent model, the “connection” that the task depends on is lost: a text-only model cannot exploit structural information in illustrations, and a sound-only model cannot use text to name what it hears. The value of multimodal models lies precisely in keeping this connection within the same computational process.

The second reason is at the capability level: after connecting different modalities into the same model system, the model can achieve joint conditioning and cross-modal reasoning. Joint conditioning means that generation or judgment is based on inputs from multiple modalities at the same time—for example, when answering “where is this cat lying,” the model uses both the cat’s location in the image and the reference “cat” in the question. Cross-modal reasoning means that something observed in one modality can trigger inference or correction of content in another modality. This capability emerges only after information channels between modalities are established.

It is important to note that “the same system” is not a fixed architecture. It can be an end-to-end unified model in which all modalities share one set of parameters from the start; it can also be a system composed of specialized encoders, projectors, adapters, and a language model, among other modules. The key point is that, regardless of internal organization, from the outside the input of a multimodal system is data in one or more modalities plus a task instruction, and the output is text, images, sound, actions, or some cross-modal judgment. Its processing flow can be decomposed into two steps: first encode each modality into a computable representation, then establish cross-modal information channels between these representations so that information can flow across modalities.

Interpretation of the output must return to the specific task. For example, a visual question answering result of “the cat is lying on the keyboard” only shows that the model correctly associated the image evidence with the question on this task, and does not mean that all of the model’s visual capabilities are reliable in any scenario. Conversely, if a task does not require cross-modal connections—such as classifying a piece of plain text or single-modal recognition of an image—a dedicated model is usually simpler, cheaper to train, and easier to debug. Multimodal is not unconditionally better; it shows irreplaceable value only when the task genuinely spans modalities.

Therefore, the essence of multimodal models is not “putting several models together,” but enabling one model to simultaneously accommodate and relate different modalities in a unified representation. Only when information is genuinely connected in a unified representation space can it produce an effect of 1 + 1 > 2; if the outputs of several models are merely mechanically concatenated, cross-modal connections will not appear automatically.

Transformer was originally designed for language modeling, but it can be used directly to process images and sound because it has a strong “agnostic” characteristic with respect to its input: it does not care what the input actually is, only whether the input can be cut into a sequence of tokens, that is, a sequence of vectors. As long as this condition is met, data from any source can go through the same computational pipeline.

The specific approach is to force different modalities into the same shape. An image is cut into small squares, and each small square is encoded as if it were a “word”; an audio clip is cut into frames, and each frame is likewise encoded into a vector; text is already made up of tokens. Thus, information from completely different sensory channels is unified into a single data structure, “a sequence of tokens,” and the same Transformer can process them all.

This solves the first hurdle of multimodality—whether they can be computed together. At this layer, the input is a sequence of vectors obtained after encoding image patches, audio frames, or text, and the output is the updated token representation from the Transformer. The update is effective because of the self-attention mechanism: every token in the sequence can exchange information with every other token, and the output at each position integrates context from the entire sequence. Precisely because this mechanism depends only on the abstract relationship of “pairwise interactions between positions” and not on the specific physical meaning of the input, the same computational backbone can process sequences from different sources without modification.

But we must be clear about the boundary of this layer: the output representation only indicates that the model has completed a numerical transformation. Without paired training or other alignment signals, tokens that are close in position across the two modalities cannot be interpreted as the same concept. In other words, “can be computed together” does not mean “understands that they are the same thing.”

After you cut an image of a cat into tokens and turn the word “cat” into a token, put them into the same model, at this moment they are just two strings of numbers speaking different languages. The language of image tokens comes from pixel patches, and the language of text tokens comes from the vocabulary; there is no natural correspondence between the two. How is the model supposed to know that this sequence of image tokens and that word refer to the same thing? Transformer itself cannot answer this question. It is only responsible for continuing the computation over the sequence, while knowledge like “this string of numbers and that string of numbers have the same semantics” must be injected by an additional alignment mechanism. This is exactly the problem the next cornerstone will solve.

The previous cornerstone left an unresolved question: although image tokens and text tokens can enter the same Transformer and be computed together, the model does not know which string of numbers and which word refer to the same thing. Cross-modal alignment aims to solve this problem—making the “image of a cat” and the word “cat” correspond in the model's view.

One important and direct approach is to learn comparable cross-modal representations: pull paired image-text vectors closer and push unpaired ones apart. This is the CLIP route. It uses massive paired data of “image + corresponding text caption” to train two encoders simultaneously—one for images, one for text—and then applies contrastive training: paired image and text vectors move toward each other, unpaired ones push apart. This objective is exactly the contrastive learning from the “embedding” concept, except the objects of comparison cross modalities. After training, images and texts share the same semantic space: different modalities have their own encoders, but all are mapped into the same space, so that the “image of a cat” and the word “cat” land close to each other. With alignment, image tokens and text tokens truly “speak the same language”; this shared space is also the foundation for text-to-image generation to “find corresponding pictures from text.”

It should be noted, however, that multimodal systems do not necessarily permanently compress all tokens into a single shared space. Another approach is to keep visual features unchanged and connect them to a language model through projection layers or cross-attention. Whichever route is taken, the common goal is the same: to establish learnable modality alignment and information channels.

The effect of alignment can be seen clearly through a numerical example. Let the cat image vector v = (0.8, 0.6), the text vector for “cat” t₁ = (1, 0), and the text vector for “keyboard” t₂ = (0.6, 0.8), all three with length 1. After length normalization, cosine similarity equals the dot product. Computing one by one: v · t₁ = 0.8 × 1 + 0.6 × 0 = 0.8, whereas v · t₂ = 0.8 × 0.6 + 0.6 × 0.8 = 0.48 + 0.48 = 0.96. So the similarity of this cat image vector to “keyboard” is 0.96, which is actually higher than its similarity of 0.8 to “cat.” This result is not contradictory: if the image shows a scene of “a cat lying on a keyboard,” the keyboard pixels occupy a considerable portion of its visual representation, so the local representation matching “keyboard” more closely is completely reasonable. It precisely illustrates that similarity scores must be interpreted in the context of a specific task.

The input to alignment training is paired or corresponding image-text samples; the output is comparable representations or features that can be queried across modalities. The training objective is clear: increase the similarity of correct image-text pairs and decrease the similarity of incorrect pairs. In retrieval scenarios, the model ranks by relative scores, where a higher score means “a better match within the current candidate set,” but it is not a factual probability. A single similarity value cannot guarantee that the model can count keycaps or understand spatial relations; if the paired data itself is biased, the candidate set changes, or the task requires counting and spatial reasoning, global alignment alone is not enough.

Looking at the two cornerstones together: Transformer enables different modalities to be computed together, and alignment enables them to speak the same language. Without alignment, multimodality is just several strings of unrelated tokens forced together; alignment is the soul that truly connects modalities in a multimodal system.

The previous two sections discussed “can they be calculated together” and “how to align them”; now we arrive at a more concrete scenario: we already have a very articulate large language model, how do we give it “eyes” so that it can look at images while answering questions.

A mainstream approach is very direct and can be viewed as a three-stage connection chain. The first stage is the visual encoder: it turns the input image into a sequence of visual tokens, that is, a vector sequence of image features; this step completes the conversion from pixels to a computable representation. The second stage is the projection layer: it aligns these visual tokens into the large model's word embedding space, making them look like “words” the large model recognizes. The third stage is concatenation and generation: treat the projected visual tokens as some special “words”, place them before the text tokens, and feed them together into the large model. Thus the large model can “read” the image just as it “reads” text, and then generate an answer autoregressively as usual—when it predicts the next token at each step, it can see both the image content and the question text, and the two interact within the same sequence.

The input to this connection chain is an image and a text question, and the output is the answer generated by the language model. The division of labor among the parts is clear: the visual encoder is responsible for extracting image features, the projection layer is responsible for turning the features into visual tokens that the language model can accept, and the language model then combines the text tokens to gradually generate the answer. The whole process is feasible precisely because the first two cornerstones are already in place: visual tokens and text tokens are the same data structure, so they can enter the same sequence; the alignment completed by the projection layer lets the model know that these visual tokens express image semantics rather than a string of unrelated numbers.

When interpreting the output, we need to stay clear-headed: the words in the answer are the result of conditional generation, not direct readings from a visual sensor. What the model “sees” is actually the encoded and compressed image representation; fine text, spatial localization relationships, and details in ultra-high-resolution images may be lost during this compression process. Therefore, for any task that involves reading small text in an image, judging precise positions, or relying on high-resolution details, you should separately verify the model's answer using the corresponding image crop or enlarged region, rather than assuming it has “seen clearly” the whole image.

Once text, images, and sound are connected into the same model, things that were previously impossible or done poorly begin to become possible. These new capabilities do not appear out of thin air; each one corresponds to the mechanisms established earlier:

The input to this capability table is a specific cross-modal business task, and the output is the conversion direction to adopt and the acceptance result: image-text question answering outputs an answer, text-to-image outputs an image, and a screen Agent outputs controlled actions. The thinking order when selecting a capability should be: first determine which modalities provide conditions for the task and which modality carries the final result, then verify segment by segment along the “alignment–generation” chain whether the model truly possesses the required capability for that segment. When interpreting results, also limit the scope: a single successful image-text question answering only shows that the model is qualified on that task slice; “supports image input” does not imply that OCR, chart reasoning, spatial localization, and video understanding all meet the standard.

From a more macroscopic perspective, multimodal large models can be understood as adding input interfaces and alignment layers for processing other modalities to a large language model, while the core “understanding and generation” still follows the same mechanism as the language model. It is not a completely new system built from scratch, but a natural extension of language model capabilities. This also explains why today’s mainstream models are almost all natively multimodal: since the cost of alignment and interfaces has been continuously driven down, while the benefits cover the entire table above, incorporating multiple modalities into the same system becomes a natural choice.

Multimodal systems introduce new failure sources that text-only models do not have, which can be grouped into three categories: alignment, computation, and generation. To diagnose a failure, you cannot look only at the final answer—looking at the answer alone cannot distinguish whether the model failed to see clearly, failed to align, or whether the language model itself made up the content. The correct approach is to put the failure sample, the original modality source, and the intermediate representation together, and inspect layer by layer along a causal chain: whether the original input is clear; whether the encoding preserved key information; whether the cross-modal correspondence is correct; whether the generation is faithful to the evidence. Every layer can be the first place where an error occurs, and the location of the first error determines the remediation to take.

The first type of cost lies in alignment. Alignment quality sets the upper limit: if image-text alignment is trained poorly, the model will "see wrong"—recognize things in the image as something else, or establish incorrect correspondences between image and text. The success or failure of multimodal systems depends to a large extent on the volume and quality of alignment training data, and flaws in this step will be amplified across all downstream tasks.

The second type of cost lies in computation. The information density of different modalities varies greatly: a high-resolution image carries far more information than a sentence, and after being split into tokens, the sequence is both long and expensive; processing long videos is especially compute-hungry. Multimodal models have to pay encoding and attention costs far exceeding those of text-only models for an input that has "few words but a lot of information".

The third type of cost lies in generation. Hallucination still exists under image input: the model may describe things that are not in the image at all in vivid detail, for example asserting "there is a dog in the image" when there is actually none. The root cause is that what it describes is still the "most likely statement" rather than the "observed fact"—the image merely adds a condition and changes the probability distribution, but does not turn the generation process into an objective transcription of the image.

Therefore, do not treat multimodal models as "objective eyes". The essence of "image captioning" is still generating the most likely text, just with an image as an additional condition. It will see wrong and will make things up; in critical scenarios, its descriptions still need to be checked against the original image, and you cannot assume by default that its output is a faithful reading of the image. Only by understanding this can you make good use of the capabilities brought by multimodality while not pushing it into a position it cannot bear.

Taken together with the previous sections, the whole causal chain of multimodal models is interlocking. The starting point is reality: many tasks are inherently cross-modal—image captioning, reading documents with illustrations, identifying objects by sound. Separating modalities loses the very connections the tasks depend on; when a single model handles them in a unified way, different modalities can corroborate each other—for example, image evidence correcting linguistic ambiguity. Therefore, multimodal models are needed; this is the first link in the whole chain.

The first cornerstone then appears: Transformer is agnostic to modality. It does not care whether the input is pixel patches, audio frames, or text; as long as they can be cut into tokens, they can be computed together. This solves the problem of 'whether they can be computed together'—but only that half: being computed together does not mean understanding they are the same thing. The image token of a cat and the word token 'cat' are still two strings of numbers each speaking its own language; semantic identity does not automatically appear just because they enter the same sequence. It is precisely this gap that pushes the reasoning to the second cornerstone.

The second cornerstone is cross-modal alignment. Its form can be CLIP-style learning of shared representations, pulling paired vectors closer together, or it can preserve visual features and connect to the language model through a projector or cross-attention, but the goal is the same: to establish learnable modal alignment and information channels. With alignment, image tokens and word tokens can speak the same language, and cross-modal queries such as 'find the image from text' become possible.

The engineering implementation puts this chain into practice: a visual encoder converts images into visual tokens, which are then projected into the word-embedding space, concatenated with text tokens, and fed into the large model. The model can then generate answers autoregressively while looking at the image. Once modalities are connected, capabilities such as image-text question answering, text-to-image generation, screen AI Agents, and accessibility applications are unlocked in succession, each corresponding to the mechanisms accumulated earlier.

But at the end of the causal chain is cost: alignment quality determines the ceiling; poor alignment leads to 'seeing wrong.' Information density varies greatly across modalities, and high-resolution images and long videos mean long and expensive token sequences. Moreover, image conditioning cannot eliminate hallucination—the model may still be describing the most likely statement rather than the fact it sees. So the whole chain ultimately returns to a passing standard: being able to clearly explain that multimodal models are built on the two cornerstones of 'Transformer is modality-agnostic' and 'cross-modal alignment', especially explaining why 'aligning to the same semantic space' is the soul—being able to compute together is only an admission ticket; speaking the same language is where multimodal models truly begin.

The knowledge points on this page can be divided into four levels according to their dependencies; proceeding level by level will make learning smoother:

The prerequisite layer provides the foundation for all mechanisms on this page: Transformer explains why the same computational skeleton can process arbitrary token sequences; embeddings explain vector representations and contrastive training; Token and Tokenization explains the text-side version of “cutting different modalities into unified units”; large language models are the most commonly integrated generation core in multimodal systems. If the prerequisites are not solid, every later building block will be left hanging.

The four core concepts on this page happen to form a causal chain: modality-agnostic processing allows sequences from different sources to enter the same model; shared semantic space and modality alignment let these sequences “recognize” each other; and visual encoder integration turns this chain into a concrete engineering solution. These four concepts are the basis for judging whether “a system counts as truly multimodal”.

Immediate extensions unfold in two directions: first, CLIP, which makes the alignment idea into a benchmark model, and image generation premised on “finding the picture from text”; second, using multimodality for action—Computer Use and AI Agent, where an agent understands interface screenshots and operates them. A further layer moves toward the temporal dimension and generation capabilities: video generation requires alignment to further cover dynamic sequences, speech recognition and synthesis brings the auditory modality into the same chain, and world models try to give models the ability to predict the world on top of multimodal perception. Each layer is not isolated; instead, it is a natural extension of the preceding causal chain.