CLIP: Using image–text contrast to align two modalities into the same space

Images and text are originally two completely different types of input: images are pixel matrices, text is a token sequence, and the two cannot be directly compared. CLIP uses two independent encoders to map them to a single shared vector space of the same dimension. For the i-th image Iᵢ, first the image encoder f_img obtains features, and then L2 normalization is performed:

vᵢ = f_img(Iᵢ) / ‖f_img(Iᵢ)‖

For the j-th text Tⱼ, the text encoder f_txt performs the same process:

tⱼ = f_txt(Tⱼ) / ‖f_txt(Tⱼ)‖

where ‖·‖ denotes the vector length. After dividing by its own length, both vᵢ and tⱼ become unit vectors, so the dot product vᵢ·tⱼ equals their cosine similarity. The larger the score, the closer the directions of the image and text in this shared space, that is, the more semantically matched the model considers them.

For example, a batch contains two pairs of data: I₁ is a cat image, T₁ is “a cat”; I₂ is a dog image, T₂ is “a dog”. After comparing the normalized vectors pairwise, the similarity matrix is [[0.8, 0.2], [0.3, 0.7]]. The first row represents the similarity of the cat image to “a cat” and “a dog”, with 0.8 higher than 0.2; the second row represents the similarity of the dog image to the two texts, with 0.7 higher than 0.3. The two correct pairs therefore lie on the diagonal of the matrix and achieve the higher score in their respective row or column.

The immediate benefit of the dual-encoder structure is that images and text can be encoded independently and offline in advance. During actual retrieval, there is no need to rerun the joint computation of the two encoders; only the already obtained vectors need to be compared. Its limitation also comes from the same design: before scoring, there is no deep interaction between individual tokens and image regions, so the final judgment depends on whether the two global vectors can retain the information needed for matching.

If a batch contains N real image-text pairs, it simultaneously produces N×N candidate combinations. The unit vector vᵢ of the i-th image and the unit vector tⱼ of the j-th text segment are dot-multiplied pairwise, then scaled by temperature τ, yielding the matching score before entering softmax:

Sᵢⱼ = vᵢᵀtⱼ / τ

The superscript ᵀ denotes transpose, so vᵢᵀtⱼ is the dot product of two unit vectors. S is an N-row, N-column matrix: each row fixes one image and compares N text segments, and each column fixes one text segment and compares N images. In the collected data, the i-th image and the i-th text segment belong to the same real pair, so the diagonal Sᵢᵢ are positive pairs; off-diagonal combinations are treated as negative pairs in the standard CLIP objective. In this way, by providing only N real pairs, a large number of candidates for comparison naturally form within the batch, without needing to construct negative samples for each sample one by one.

Taking two groups of data as an example, I₁ is a cat image, T₁ is cat text, I₂ is a dog image, T₂ is dog text. The original similarity matrix is [[0.8, 0.2], [0.3, 0.7]]: 0.8 and 0.7 lie on the diagonal and are the cat–cat and dog–dog positive pairs, respectively; 0.2 and 0.3 lie off the diagonal and are the cat–dog and dog–cat negative pairs. When τ = 0.1, each similarity is divided by 0.1, giving logits [[8, 2], [3, 7]]. Scaling does not change the ranking of candidates, but it widens the numerical differences seen by softmax, making the probability distribution sharper; a larger or smaller τ will correspondingly change this sharpness.

Increasing the batch size makes each image or text face more candidates, thus providing richer contrastive signals, but off-diagonal entries are not necessarily all true negative examples semantically. If two samples with similar content and actually identical semantics are not collected as the same pair, they will still be treated as negative pairs by the objective, and this is a false negative sample. As the number of candidates grows with the batch, such conflicts are also more likely to increase, so the efficient N×N construction simultaneously brings stronger contrast and a higher risk of false negative samples.

The same similarity matrix can support retrieval in both directions, but the two directions correspond to different competition sets. In image-to-text retrieval, one image is the query and all texts in the batch are candidates, so softmax must be applied along each row of the matrix; in text-to-image retrieval, one text is the query and all images in the batch are candidates, so softmax must be applied along each column. CLIP optimizes both directions simultaneously:

L = (CE_image→text + CE_text→image) / 2

CE_image→text represents the average cross-entropy for each image to select its paired text among all texts, and CE_text→image represents the average cross-entropy for each text to select its paired image among all images. The final loss L is the arithmetic mean of the two. Cross-entropy takes the negative logarithm of the predicted probability of the correct candidate, so the closer the probability of the correct pairing is to 1, the smaller the corresponding loss; conversely, if the model assigns a higher probability to an incorrect candidate, it incurs a larger penalty.

Continuing with the logits matrix [[8, 2], [3, 7]], compute the image-to-text probabilities row by row. For I₁, the probability of T₁ is e⁸ / (e⁸ + e²) ≈ 0.9975; for I₂, the probability of T₂ is e⁷ / (e³ + e⁷) ≈ 0.9820. The probabilities of both correct texts are close to 1, so the image-to-text loss is small. When computing column by column, T₁ must retrieve I₁ among I₁ and I₂, and T₂ must retrieve I₂ among I₁ and I₂; the loss in this direction constrains the text vectors to correctly distinguish images as well. The bidirectional averaging prevents training from favoring only one query mode; if only a unidirectional objective is optimized, the retrieval structure formed by the other side may be weaker.

Temperature affects the intensity of candidate competition in both directions. The lower the temperature, the sharper the softmax distribution, and high-scoring incorrect candidates or hard negative pairs close to the positive pair have a greater impact. This can strengthen learning on hard-to-distinguish candidates, provided the pairing signal is sufficiently reliable; if there are many incorrect pairings in the data, an overly sharp distribution will also force the model to reinforce these erroneous relationships.

Figure 1 shows the relationship between image–text dual encoders and a 2×2 similarity matrix. Images first pass through the image encoder to obtain image vectors, and texts first pass through the text encoder to obtain text vectors; the two processing paths are independent until these vectors meet when compared pairwise in the similarity matrix. The rows of the matrix correspond to images, and the columns correspond to texts; each cell represents the logit of a pair of image and text vectors after a single temperature scaling. This matrix is denoted S, consistent with the definition in Section 2; the 8, 2, 3, and 7 in the figure are already the values of vᵢᵀtⱼ divided by τ, and should not be divided by τ again.

The true image–text pairs in the training data lie on the diagonal of the matrix. The training signal targets these diagonal positions and is sent back to both the image encoder and the text encoder, so what is adjusted is not just the vectors on one side, but the way both sides produce vectors. As a result, the image vectors and text vectors of correct pairs gradually move closer in the shared space, making the matches on the diagonal more prominent.

This figure also reveals a key boundary of the dual-encoder structure: images and texts do not directly exchange information during their respective encoding processes; the two meet only at the final vector comparison. Therefore, the shared space is not an additional joint representation, but a comparable coordinate system jointly shaped by the two encoders through the same training signal.

Zero-shot classification does not require training a new classification head for target categories such as “cat” and “dog”; instead, each category is converted into text, and the text vector serves as the class prototype. For example, when the candidate categories are cat and dog, “a photo of a cat” and “a photo of a dog” can be constructed separately. After these texts pass through the text encoder and are normalized, they become candidate representations that can be directly compared with image vectors; the same category can also use multiple templates, and the corresponding vectors are averaged into the text prototype for that category.

At prediction time, the image to be tested passes through the image encoder to obtain a normalized vector, and then its similarity to all pre-declared category prototypes is computed. The system can directly select the category with the highest similarity, or apply softmax to these candidate scores to obtain a probability distribution over the current candidate set. Thus, classification is converted into an image-to-category-text retrieval: whichever text prototype the image is closest to is predicted as the category.

Here, “zero-shot” means that no new classification head is trained for these categories; it does not mean that the model can discover all possible labels out of thin air. Results are valid only within the pre-given closed set of candidates; if the true category is not included in the candidate set, the model will still select one from the existing categories. The wording of category names, the language used, prompt templates, temperature, and the candidate set itself can all change the relative scores of the categories. To prevent test results from being influenced by temporarily selected templates, templates should first be selected on the validation set, frozen, and then used for final testing.

Dual encoders encode retrieval objects and queries independently into global vectors, so image vectors in a large image gallery can be computed in advance and an ANN (approximate nearest neighbor) index can be built. After receiving a text query, the system only needs to encode the text once and then quickly find a small batch of images whose vectors are closest through the index, without having to perform a complete exact scan of the query against every image in the gallery. This division of labor—"gallery processed offline, query encoded once online"—is key to CLIP achieving high throughput in large-scale retrieval.

The cost of this efficiency is limited interaction depth. During the comparison stage, CLIP faces global vectors compressed separately from images and text, and during scoring it does not let text tokens interact further with specific image regions. For complex queries that depend on detailed relational judgments, the global vector may be insufficient to express the local correspondences that determine whether a match is made, so it is easy for recalled results to be broadly relevant while the details do not match the query.

Cross-encoder takes the opposite trade-off: it lets text tokens interact with image regions in the same model, enabling finer relational judgments, but every query–candidate combination requires a joint computation, making it difficult to pre-index gallery representations as dual encoders do. Real systems often chain the two: first use CLIP to recall a small batch of candidates from a large gallery with high recall, then have a cross-modal cross-encoder rerank those candidates. In this way, the dual encoder is responsible for narrowing the search space, and the cross-encoder confines the more expensive fine-grained computation to a small number of candidates. CLIP is more suitable for recall and zero-shot matching, while the cross-encoder is more suitable for fine-grained ranking within a small candidate set.

“Dog chases person” and “person chases dog” contain the same object and action words, but the subject and object roles are exactly reversed. If the model’s global vector mainly retains the overall topic “dog, person, chasing” without stably encoding who performs the action on whom, the two sentences may be mapped to nearby positions. Web image-text pairs often come from relatively coarse captions, and the global contrastive objective only requires the correct topic’s score to be higher than other candidates in the batch, which does not necessarily force the model to parse the role and order of each word.

The same issue also appears with quantity, spatial relations, and negation. “No dog,” “two dogs,” and “the dog is under the table” all contain the conspicuous object word “dog”; global similarity may be dominated by object co-occurrence and ignore the key constraints introduced by “no,” “two,” or “under the table.” Therefore, high image-text similarity only indicates that the overall representations are close; it cannot be interpreted as the image entailing the entire sentence, nor can it prove that every predicate in the description is supported by the image.

When evaluating these abilities, minimal contrasts should be used to isolate the factor that actually needs to be judged. One can swap only the subject and object, change only the quantity, change only the spatial relation, or add only negation, while keeping other words and image statistics as consistent as possible. Then check whether the model can consistently rank the correct description ahead of the incorrect description that differs only in one key relation, and report pairwise accuracy. Overall Recall may be inflated by a large number of samples whose topics are easy to match, whereas pairwise accuracy more directly reveals whether the model truly understands roles, counting, spatial relations, or negation.

Alt-text, titles, and nearby text on web pages can link large numbers of images with natural language, providing the model with large-scale, open-vocabulary supervisory signals. Their value is that rich expressions can be covered without creating precise annotations anew for each image; but this image-text relationship is only weak pairing, and the text may describe the entire web page, advertisement, or context rather than the image itself. Data scale therefore grows together with supervisory noise, and what the model learns may be useful visual-language correspondences or mismatched relationships.

Noise does not only manifest as inaccurate descriptions. Web text may also contain stereotypes, mislabeled identities, and personal information; near-duplicate content in images or text may cross into training and evaluation sets, causing evaluation results to be affected by data leakage. For mismatches and weak descriptions, manual sampling can be used to understand actual noise, and noise-robust training can reduce the impact of incorrect supervision. For social bias, one should not only look at overall metrics; differences should be checked by slices such as population × occupation × language. For duplication and contamination, near-duplicate deduplication should be performed separately on images and text. For privacy and licensing, the source and authorization status should be recorded, and an executable deletion chain should be retained.

Data filtering itself also has boundaries. Filtering too weakly retains more mismatches, harmful content, and sensitive information, while filtering too strongly may delete minority languages and specific cultural contexts along with it, further unbalancing data coverage. Therefore, data governance cannot only pursue 'the cleaner the better'; it must simultaneously evaluate the quality of the retained supervision, coverage across different groups and languages, evaluation contamination risk, and the source, licensing, and deletability of content.

CLIP score comes from the global vector similarity between image and text, answering 'how close this image is to this prompt in the shared representation.' Based on the mechanism introduced earlier, this value is suitable for comparing how close the current prompt is to the image representation; it does not itself output independent judgments about anatomy, facts, aesthetics, copyright, or safety. Therefore, a high score only indicates that the model leans more toward this image on this matching axis, and cannot serve as proof that these other attributes have been satisfied.

When using it, first clarify the decision problem: if the question is which candidate image is closer to the prompt, CLIP similarity can be used as a ranking signal; if the question is whether an image is real, safe, or suitable for a specific population, then metrics and checks need to be designed separately for that attribute. This limited use comes from the input-output boundaries of the score itself, and does not rely on interpreting it as a universal quality score.

Generative evaluation should measure prompt alignment, visual quality, diversity, relationship and text accuracy, safety, and human preference separately. Each dimension needs evidence matching its meaning, and should be validated with independent models and human sampling. Results across different populations should also be reported in slices; especially in high-impact scenarios affecting people, rights, or safety, one cannot rely solely on a single embedding score as the final decision.

From raw data to practical applications, CLIP’s mechanism can be connected into an inspectable causal chain:

Collecting and governing image-text pairs → generating normalized vectors with dual encoders → constructing an image-text similarity matrix within the batch → aligning the shared space with bidirectional cross-entropy → completing task transfer via text prototypes or ANN → constraining usage with minimal contrasts and subgroup slices.

The starting point of the chain is image-text pair data. After governance is applied to these pairs, images and texts separately enter the dual encoders and output normalized vectors. Normalization allows the representations of the two modalities to be compared on the same scale; combining the image vectors and text vectors in the batch pairwise yields a similarity matrix. Training then simultaneously constrains image-to-text and text-to-image matching through bidirectional cross-entropy, giving true pairs more suitable relative positions in the shared space. The data provides the supervisory relation, the encoders transform inputs into representations, the similarity matrix creates candidate competition, and the loss function feeds the competition results back to the encoders on both sides; together these steps produce a transferable aligned space.

After obtaining aligned representations, we can write categories as text prototypes to perform transfer based on candidate categories, or organize vectors into an ANN index to support retrieval. Application effectiveness must still return to verifiable tests: minimal contrasts are used to check whether the model can distinguish candidates that change only one key factor, and subgroup slices are used to observe performance differences across different groups. Only by considering these verification results together with specific usage scenarios can we determine what tasks the model is suitable for and within which boundaries it should be used.