In-context Learning

In-context learning addresses a seemingly paradoxical question: without any additional training, how can the model "learn" a new task within a single request that it has never specifically practiced? The method is surprisingly simple—write a few examples of the task directly into the prompt, and the model follows them after reading them.

First, look at inputs and outputs. The input of an in-context learning request consists of three parts: a task description, zero or more "input → output" examples, and a new query. The output is the answer generated by the model according to the pattern temporarily presented in the prompt. Throughout the process, the model only performs a single forward computation, using the examples to constrain generation, and parameters are not updated. A direct consequence of this is: the result only indicates that this particular request successfully followed the mapping given by the examples; if the examples are deleted or the request changes, this convention is not automatically retained.

Based on how many examples are given in the prompt, it can be divided into three levels:

A few-shot prompt consists of several examples plus one query. Its skeleton is: the first few lines use “input → output” pairs to demonstrate the task, and the last line gives only the input and leaves the output blank. For example, the first two lines give “Great → positive” and “Expensive and unpalatable → negative”, and the last line is “Service is very attentive → ”. What the model needs to do is still the only thing it does—predict the next token, and at that moment “the most likely next token” is exactly “positive”.

Working example: even the meaning of labels is temporarily defined by context

Replacing the familiar “positive/negative” with two arbitrary labels zorp / blip makes it clearer what the model actually recovers from the prompt. Here is such a prompt:

• “Great” → zorp
• “Expensive and unpalatable” → blip
• “Service is very attentive” → ?

The model can extract from the first line that zorp is associated with positive sentiment, and from the second line that blip is associated with negative sentiment; following this temporary mapping, the third line should output zorp.

It should be noted that this does not permanently write the new meaning of zorp into the weights; rather, it establishes a temporary mapping in the current token sequence by using the examples. After deleting the first two lines, the model has no reason to continue answering according to this convention in the next request: this mapping exists only in the reasoning process of this sequence and will not settle into any reusable parameter change.

Therefore, the word “learning” should be used with caution. It does not “remember” knowledge as humans do: the “learning” here means following the examples on the spot, without changing a single weight throughout; once the conversation ends, it forgets, and next time you have to provide the examples again. This feature of “temporary convention, no parameter changes, discarded after use” is the fundamental dividing line between it and traditional training methods such as fine-tuning, and later chapters will explain this meaning thoroughly.

'Letting the model learn a new task'—isn't fine-tuning already available for that? What's the difference between the two? It comes down to one fundamental thing: whether weights are changed. Fine-tuning permanently reshapes the model's parameters during training; in-context learning temporarily adapts during use via a prompt, with parameters not moving at all.

Along this main line, the two approaches diverge point by point in inputs, mechanisms, and applicable scenarios. The inputs being compared are task stability, sample size, latency, call volume, and maintenance constraints; the output is one of two approaches: temporary prompt adaptation, or a parameter training approach. In-context learning includes examples as tokens in forward computation each time, while fine-tuning updates weights via loss and gradients. The former suits quick experimentation, the latter suits long-term consolidation, and the two are not mutually exclusive—they can also be used together.

Fine-tuning is a one-time 'model modification' engineering effort: its results are written into the weights and take effect for all subsequent requests. In-context learning is a one-time 'temporary borrowing': it is effective only within this prompt and disappears as soon as the request ends. Let's put several key dimensions together:

Condense the division of labor into one sentence: in-context learning is temporary adaptation at runtime; fine-tuning is permanent modification at training time. When there are few tasks, you need them immediately, and they change frequently, choose the former; when the task is fixed, samples are plentiful, and you need long-term stability and cost savings, choose the latter.

This is the part that requires the most rigor: a model whose training objective is just “predicting the next word”—why would it follow a few examples without anyone ever teaching it to “learn”? First, lay out the doubts: the model has never been explicitly trained on the procedure of “reading examples, inducing patterns, and applying them to new inputs.” In-context learning is an ability that was not directly taught but emerged on its own. Where does it come from?

The clue is hidden in the pre-training data. Internet text is full of repeated patterns: a consistently formatted list, a sequence of “Question: … Answer: …”, a question-and-answer dialogue, an “English–Chinese” lookup table. To guess the next word in such text accurately, the model has no choice but to learn one thing—recognize the pattern currently in progress and continue writing in that pattern. When it sees “A → X, B → Y” and needs to continue “C → ___”, the most likely next token is, of course, the one that corresponds to C according to the same mapping rule. A few-shot prompt artificially creates exactly this pattern: by giving several “input → output” pairs, the model recognizes “oh, we are now doing an X → Y mapping,” and then does the same for new inputs.

So in-context learning is not a new ability that emerged out of thin air, but a by-product forced by the training objective of “predicting the next word”—just like reasoning and translation. The model has already seen countless texts of the sentiment-classification type; the real role of few-shot examples may not be “teaching you this task from scratch,” but telling the model: among the thousands of patterns you learned during pre-training, which one to invoke now. The example “tunes” it to that channel.

From a mechanistic perspective, the input is a token sequence with a repeated format and a mapping relationship, and the output is a distribution over the next token that matches that local pattern: pre-training has let the model practice completing many lists, Q&A passages, and lookup tables, so examples can help it locate the current task pattern. This “task location” perspective can also explain the counterintuitive finding in the next section.

But to be honest: what exactly happens inside in-context learning is still an active research topic, with multiple explanations—task location, implicitly performing gradient-descent-like updates during forward propagation, and so on—and no definitive conclusion. What was given above is a working model with strong explanatory power, one that is tenable and sufficient, not the only internal algorithm that has been proven, and even less a final conclusion.

Since the model is "learning" from examples, must the example labels all be correct? The researchers' approach was to run controlled experiments: take the original few-shot prompt, then create versions with shuffled labels, disrupted formatting, replaced label sets, and changed input distributions, have the model run the task on each, and compare the performance drop caused by each change. A small drop indicates that this factor has weak influence in the current task—note that this is only a conclusion for the current task and does not mean it is unimportant in all tasks; specialized tasks and new knowledge mappings may still depend heavily on correct examples.

The experimental results are surprising. Deliberately shuffling and randomly pairing the labels of few-shot examples—for example, giving a mismatch like "Great → Negative"—often causes model performance to drop only slightly. But changing the other three things causes performance to drop substantially:

Taken together, these results lead to the conclusion that few-shot examples mainly demonstrate what the task "looks like"—what the input looks like, what format the answer should be, and what possible labels exist—rather than feeding the model item-by-item correct knowledge. This exactly corroborates the "task location" perspective: examples tune the model to the right channel, and the model already had the correct judgment from Pre-training.

But don't misread this conclusion as "examples can be written carelessly." It says that individual label mismatches have little impact, not that you are encouraged to provide wrong examples: format, label set, and input distribution must all be correct and consistent, and on harder, more specialized tasks, the correctness of the examples themselves remains important. A more robust understanding is to treat examples as "providing a template" rather than "giving them casually."

Since the effect is almost entirely concentrated in these few examples, what is the most effective way to provide them? The task of example selection takes candidate examples, the current query, and a token budget as input, and outputs a prompt with a moderate number of examples, adequate class coverage, stable order, and consistent format. You can first ensure correctness and consistency, then prioritize samples that are similar to the query and cover boundary cases, and retest with multiple orderings; when judging results, look at stable gains on the target slice rather than just one particular arrangement. Specifically, break it down into four things.

Quantity: Going from 0 to 1, then to a few, gives the most obvious improvement; beyond that marginal returns diminish, and each example consumes context window budget. Examples are not necessarily better with more; 0 → 1 → 2 usually yields the largest improvement, then it plateaus with more. Choosing accurately matters more than piling on many.

Selection: Choose examples that are similar to the current query and high in quality; for classification tasks, try to cover all classes so the model does not mistakenly think there is only one possible answer.

Order: Models are sensitive to example order and often exhibit a recency bias where the closer to the query, the greater the influence. Do not group similar examples together; interleaving them makes the arrangement more stable.

Format: Use consistent, clear separators and layout, such as unified “Input: … Output: …”. The controlled experiments in Section 4 have already shown that consistent format often affects results more than individual label correctness.

These four things belong to a larger layer: deciding which examples to put into the context window and how to arrange them is essentially part of context engineering; equipping a single prompt with examples is one of the most common moves in prompt engineering. In-context learning is the principle behind why this move works.

The previous examples only gave “input → answer.” What happens if the examples also write out “how to think of the answer step by step”? The model will answer new questions by following the pattern of “reason first, then give the answer”—this is few-shot chain-of-thought.

The input to few-shot chain-of-thought is examples containing an intermediate reasoning format plus a new question, and the output is the steps and answer that the model generates by imitating that format. Its mechanism is pattern continuation: you are not teaching the model to reason; you are merely using examples to set the pattern that “the output should include reasoning steps,” and the model follows suit, and this continuation induces longer intermediate computation. Note that steps that look reasonable are not necessarily factually correct; acceptance should be based on the final answer, verifiable intermediate results, and task boundaries together.

It is still in-context learning. Chain-of-thought can be triggered by a few examples precisely because in-context learning makes the model “continue in the manner of the examples”: when the examples look like “solutions with reasoning processes,” what the model continues to produce is also solutions with reasoning processes.

In-context learning is so convenient and requires no training, so why is fine-tuning still needed? Where is its ceiling? When making this judgment, the inputs are window capacity, prompt sensitivity, knowledge freshness, request scale, and stability requirements; the output is a decision to continue with in-context learning, shift to fine-tuning, or use a combination of both. A practical signal is: if adding examples only increases token cost without improving bucketing quality, or changing the order crosses the business threshold, then temporary adaptation is already approaching its practical limit. Its limitations center on four areas.

Limited by the window: examples are all crammed into the context window, so you can't fit too many; as examples increase, it gets slower and more expensive, and it may trigger “lost in the middle”—mid-context information gets ignored.

Unstable: sensitive to example wording, order, and format; change the arrangement and results change, making it hard to reproduce and guarantee.

Not persistent and doesn't update knowledge: nothing is written into weights; every request has to replay examples; it also doesn't really “remember” new knowledge from this.

Not equal to true mastery: once a query exceeds the range covered by the examples, it can easily break down. When tasks are complex, proprietary, and have many samples, fine-tuning is usually more stable, and in the long run each request also costs fewer tokens.

Put the trade-offs of the two scenarios side by side:

The pragmatic approach is to view the two as a ladder, not opposites: first use in-context learning to quickly verify whether the model can do the task; once it works, volume grows, and you need long-term stability and cost savings, then consider solidifying it with fine-tuning.

String the entire page's content into a chain, and check link by link where each conclusion comes from.

The starting point is: write a few examples into the prompt, and after reading them the model follows suit—this is in-context learning, divided by the number of examples into zero-shot, one-shot, and few-shot. Its fundamental difference from fine-tuning lies in 'whether the weights are changed': in-context learning is temporary adaptation at runtime, while fine-tuning is permanent modification at training time.

It holds because the training objective of 'predicting the next word' forces the model to learn to 'recognize the current pattern and continue writing'; few-shot examples artificially create a pattern that tunes the model to the right task channel. From this follows the finding in Section 4: examples mainly demonstrate 'what the task looks like'—format, label set, input distribution—rather than instilling individual pieces of correct knowledge, so individual label mismatches have little effect.

Following this conclusion, providing examples requires care: moderate quantity, selecting ones similar to the query and covering all categories, shuffled order, and consistent format. And if you replace the 'form' of the examples with solutions that include a reasoning process, you readily get few-shot chain-of-thought—it is still in-context learning, just with the pattern to be continued changed to 'reason first, then give the answer'.

The final link in the chain is boundaries: limited by the context window, sensitive to wording and order, not persistent, and prone to breaking down when the examples' coverage is exceeded. These costs mark the boundary between it and fine-tuning: first use examples to validate quickly, then use fine-tuning to consolidate as needed. If you can clearly explain 'why a model that only predicts the next word can learn a new task from a few examples' and say 'examples mainly demonstrate what the task looks like rather than instill knowledge,' you have grasped the core of in-context learning.

Viewed along the learning hierarchy, the concepts on this page each have their dependencies and destinations.

The prerequisite concepts lay the mechanistic foundation of this page: In-context Learning is built on the objective of autoregressive generation, “predicting the next token,” and few-shot examples are precisely how this objective is used. The immediate extensions are the direct destinations of this page’s concepts—adding examples to prompts belongs to prompt engineering, arranging examples within the window belongs to context engineering, few-shot chain of thought advances from “what the examples demonstrate” to “demonstrating how to think,” and the capacity of the context window further defines the boundary on the number of examples. One level further out, whether examples can be replaced by retrieved material points to retrieval-augmented generation, and the capability that In-context Learning exhibits with model scale is connected to scaling laws and the emergent abilities debate.