Quantization: Compressing Continuous Parameters onto a Finite Lattice

The problem that quantization aims to solve is straightforward: weights, activations, and KV cache in models are all stored as floating-point numbers, each number often taking up as many as 32 bits or even 16 bits, and there are a great many such numbers, so bandwidth and memory are fully occupied by them during inference. What quantization does is to map values that can originally take a large number of continuous decimals onto a finite number of discrete grid points, replacing floating-point numbers with shorter integer encodings.

Information loss first comes from the number of grid points itself. Taking INT4 as an example, each encoding uses only 4 bits, so there are only 2⁴ = 16 encodings in total. Which 16 integers these 16 encodings correspond to—whether −8 to 7, −7 to 7, or some other range—depends on the signedness convention and whether special values are reserved; you cannot infer the value range just by seeing the three characters "INT4". But no matter how it is agreed, using 16 grid points to carry an originally continuous interval inevitably produces two types of errors: continuous values that fall near the same grid point within the interval become indistinguishable from each other, which is rounding error; values outside the interval are truncated to the boundary grid point, which is saturation error.

Here you can see the causal chain between bit width and error: lower bit width reduces the number of bytes the model reads each time, thereby saving bandwidth and memory; the cost is fewer grid points and coarser spacing, so rounding or saturation error increases accordingly. The input to the entire process is weights, activations, or KV cache in floating-point form, and the output contains three things: the integer encodings themselves, the scale metadata used to recover the values, and the approximated values recovered during computation—what the model actually uses in computation is the approximated values, not the original floating-point numbers.

Therefore quantization is not "lossless decompression after compressing a file." Throughout inference, approximate weights or approximate activations are used, and errors propagate layer by layer through the network. Fortunately, neural networks have redundancy, and with appropriate scales, many tasks can tolerate these errors; but the errors do not distribute evenly, and values on critical decision boundaries may still be pushed to the wrong grid points, causing some individual outputs to flip.

Finally, it is necessary to ask clearly what is being quantized, because the consequences of loss are completely different for different objects: weight quantization saves model bandwidth, activation quantization affects the numerical flow of intermediate operators, and KV quantization determines the capacity limit for long contexts. Each of the three has its own trade-offs, and they cannot be generalized with the single word "INT4".

Mapping floating-point values to integers must answer a specific question: How is a continuous floating-point range put into one-to-one correspondence with a finite set of integer grid points? The correspondence given by linear quantization is completely determined by two parameters: the scale s and the zero-point z. Given the integer bit width, the available integers are restricted to between q_min and q_max; for example, an 8-bit signed integer corresponds to −128 through 127. Quantization first performs a linear transform, then rounds, and finally clips:

q = clip(round(x ÷ s) + z, q_min, q_max)

Dequantization uses the same set of parameters to convert integers back to floating-point approximations:

x̂ = s × (q − z)

In the formula, each symbol has a clear responsibility. x is the original floating-point value; s is the floating-point step size between adjacent integer grid points, meaning that whenever the quantization code changes by 1, the dequantized result changes by s; round takes x ÷ s to the nearest integer; z is the zero-point, specifying which integer grid point corresponds exactly to floating-point zero, and its role is to shift the integer axis to a position aligned with the floating-point distribution; clip limits out-of-range intermediate results to between q_min and q_max. The final code is q, and the dequantized x̂ is usually no longer exactly equal to x; the difference between the two is the quantization error introduced in this step.

The division of labor between the two parameters can be understood as follows: s determines the grid-point spacing, that is, the resolution; z determines the shifted position of the grid-point axis, that is, which integer the floating-point zero falls on. Together they determine how many grid points a floating-point range is cut into and how large a floating-point interval each grid point represents. Therefore, the output error should be explained as the result of jointly choosing the range and the step size, not solely determined by the bit width: with the same bit width, if the range is chosen narrow and the step size fine, the error for the bulk of the data is small; if the range is chosen wide and the step size coarse, more values can be covered, but the error for each value becomes correspondingly larger.

Depending on whether z is zero, linear quantization is divided into symmetric and asymmetric types.

Symmetric quantization sets z = 0; the integer range is symmetric around zero, and floating-point zero is encoded as zero. The advantage is that the zero-point is simple and multiply-add operations are efficient, and dequantization does not require an additional zero-point correction. The cost is that when the data distribution as a whole deviates from zero, grid points on one side are barely used, the effective range is wasted, and at the same bit width the step size is forced to become larger.

Asymmetric quantization places q_min and q_max according to the data's actual min and max; z is usually nonzero, so it can completely cover a shifted distribution. The advantage is that it fits arbitrary min/max and has high range utilization. The cost is that zero-point correction is more complex, and operations must additionally handle the shift introduced by z.

There is another fork in range selection: when the data has extreme values far from the main body, whether to let the quantization range cover them.

Truncate extremes: q_min and q_max only cover the main body of the data, and a small number of outliers that exceed them are clipped to the boundary and become saturated. The advantage is that the main grid points are finer and the step size is smaller, so the error for most data is lower. The cost is that outliers are forcibly saturated, and the information they carry is lost.

Cover all extremes: the range extends from the minimum value all the way to the maximum value. The advantage is that nothing is truncated, and every value has a corresponding code. The cost is that the extremes enlarge the range and coarsen the step size, making quantization of the main data rough.

These two choices together determine a set of s and z parameters. With a fixed bit width, the larger the range, the coarser the step size and the greater the main-body error; the smaller the range, the finer the step size and the more values that fall into the saturation region. Whether the zero-point is zero determines the complexity of the kernel. Looking at a quantization table is essentially looking at which range and step size it has chosen, and using that to anticipate how the error will be distributed.

To see how much error symmetric INT3 quantization actually introduces, it is most intuitive to go through a specific set of numbers. Assume the integer grid points are 3-bit signed values in the range −4…3, and the maximum absolute value of the floating-point weights is 1.2. To make the grid cover 1.2, use the largest positive grid point 3 to represent it, so the scale is s = 1.2 ÷ 3 = 0.4, meaning adjacent integers differ by 0.4 in floating-point value. Now quantize and then dequantize the four weights in turn:

Each column's operation corresponds to one step in the formula. Take −1.10 as an example: first divide by the scale, −1.10 ÷ 0.4 = −2.75; round to the nearest integer gives −3, which is the quantized code q; dequantization multiplies back by the scale, x̂ = 0.4 × (−3) = −1.20; the absolute error from the original value is 0.10. −0.35 and 0.31 likewise are just small rounding errors, at 0.05 and 0.09 respectively. As long as values are within the grid-point coverage range, the error is at most half a step size, i.e. 0.2.

1.25 is the exception. 1.25 ÷ 0.4 = 3.125, rounding would give 3, and 3 is already the largest grid point in the positive direction: x̂ = 1.20, which is 0.05 smaller than the original value. This 0.05 may look smaller than the error for −1.10, but it comes from clipping—1.25 exceeds the upper bound 1.2 and is saturated to the boundary, so the excess information is lost; no matter how large it is, it will be squeezed onto the same grid point.

If you do not want to clip and want 3 to represent exactly 1.25, the scale must be adjusted to s = 1.25 ÷ 3 ≈ 0.4167. The cost is that all grid points become coarser: −1.10 dequantizes to about −1.25, and its error rises from 0.10 to 0.15; 0.31 dequantizes to about 0.4167, and its error rises from 0.09 to about 0.11. The loss of main-body precision from covering the extreme value is greater than the cost of clipping a single outlier.

This four-number example shows the essence of real calibration: the range and scale are a choice between bulk error and extreme-value clipping. If you clip the extreme value, the main-body grid points are fine and most errors are small, at the cost of saturating outliers; if you cover the extreme value, the outlier is preserved, at the cost of making every grid point coarser and raising the overall main-body error. The magnitude of the error is ultimately determined by this set of choices, not by “how many bits are used” alone.

The value distribution within a matrix is often not uniform. A few channels may be shifted significantly as a whole or contain large outlier values, while most channels' values are crowded near zero. If the entire tensor uses only one scale, that scale must cover the widest channel range, so most channels can only use a small segment of the sparse grid points, and quantization error is indirectly increased by a few anomalous channels. This is exactly why a single scale for the whole matrix is often dominated by a few anomalous channels: the range is determined by extremes, but the cost is borne by everyone.

The solution is to subdivide the quantization range so that each group of data uses its own suitable scale. By granularity from coarse to fine, there are three choices: per-tensor uses one scale for the entire tensor; per-channel sets a separate scale for each output channel; group-wise divides a number of consecutive weights into one group, one scale per group. The finer the granularity, the narrower the range within each group, the smaller the step size can be, and quantization error usually decreases accordingly. The cost is equally straightforward: more scales need to be stored, indexing is more complex, and computation requires specialized kernels that support this grouped layout.

The growth in the amount of metadata can be estimated with a 4096 × 4096 weight matrix. It has a total of 4096 × 4096 = 16777216 weights. Per-tensor quantization requires only 1 scale; per-channel divides by output channel and requires 4096; if every 128 consecutive weights form a group, then 16777216 ÷ 128 = 131072 scales are needed. The choice of granularity is essentially a trade-off between error and metadata: the number of scales can span six orders of magnitude.

From a process perspective, the input to this step is the weight matrix, the grouping axis, and the group size, and the output is the integer encoding and scale metadata for each group. The volume of scale metadata relative to the weights is usually still very small — even if there are 130,000 scales, compared with over 16 million weights, they account for less than 1% — so the main conflict for precision is not storage but computation: during dequantization, each group must read its own scale and convert according to the layout, and actual speed is determined jointly by dequantization overhead and layout.

Fine-grained quantization has another boundary to note: different libraries make different choices for group size, grouping axis, and packing format. Model files that are also labeled “4-bit” may internally be per-channel, or group-wise with different group sizes, or have different weight packing order, so you cannot directly compare two model files solely based on the bit width.

Finally, finer granularity and lower error do not necessarily mean faster execution on the target hardware. If the inference kernel does not support a particular grouping layout, the additional overhead of metadata reads and dequantization may completely offset the benefits from precision. When choosing granularity, you must consider hardware support for the layout together, rather than only looking at error metrics.

If an entire layer uses only one scale, the scale must accommodate the activation with the largest magnitude. A few unusually large activations stretch the range, coarsening the step size, and the vast majority of normal values can only crowd onto a few grid points near zero, indistinguishable from one another. A small number of large activations cause ordinary values to lose discriminative ability; this is what "outliers determine the range" means.

Several mainstream methods address different aspects of this problem, and their optimization objectives differ.

LLM.int8 directly separates out the outlier dimensions: it detects outlier feature dimensions in the hidden states, that is, columns of the activation matrix with unusually large magnitudes, corresponding to rows on the input side of the weight matrix, rather than the weight output channels referred to by per-channel quantization; the multiply-accumulate operations involving these dimensions are kept in FP16 high precision, while the remaining parts undergo INT8 quantization as usual. The inputs are the layer to be quantized and representative calibration activations, and the outputs are a high-precision decomposition of the outlier feature dimensions and the quantization ranges for the remaining parts.

The idea behind SmoothQuant is to move part of the scale onto the weights. If the outlier scales in activations are borne entirely by the activations, the activation-side step size becomes coarse; by migrating part of the scale to the weight side, the weights are amplified accordingly, and the quantization difficulty on both sides becomes more balanced.

GPTQ starts from the impact of weight errors on the output, using approximate curvature to measure how errors introduced by different weights propagate to the layer output, and prioritizes correcting weights with large influence at low cost. AWQ uses real activations to identify which weights are more important and compensates those important weights. GPTQ is concerned with the global impact of weight errors, while AWQ is concerned with the importance of weights for real inputs; both operate at a different stage than LLM.int8, which handles outlier feature dimensions, and SmoothQuant, which balances activations and weights.

These methods share a common premise: calibration data must represent the true distribution. Blindly clipping outliers may harm the performance on rare tokens, long contexts, or specific languages, because the so-called "outlier" values may actually carry important features—a rare but critical grammatical structure, positional information in long sequences—all of which may be expressed by those outlier activations. The calibration set must therefore cover real lengths, domains, and hard examples, and cannot rely solely on short English encyclopedia corpora.

The last boundary belongs to evaluation rather than quantization itself: a small average error does not mean the behavior is safe. The model outputs logits for each token, and final behavior is determined by their ranking. A ranking flip among a few critical logits is enough to change tool-calling actions or refusal behavior. Quantization evaluation must focus on this kind of behavior-level degradation, not just average error.

Quantization can be performed after training, during training, or even without touching the original model at all. These three routes correspond to different costs and applicable scenarios: PTQ (post-training quantization), QAT (quantization-aware training), and QLoRA-style quantized base training. Their outputs are a quantized model, a new model adapted to quantization error, and a mountable adapter, respectively—these three are not the same thing.

PTQ is performed after model training is complete: the weights are no longer changed, and only a small batch of representative data is used to estimate the scale of each layer, with weight compensation applied when necessary. It is low-cost and serves as the starting point for compressing weights to 8-bit or 4-bit. However, PTQ does not mean zero data—scale estimation still relies on calibration data, or on layer-wise optimization to correct errors; when there is no representative data at all, the estimated scales are not trustworthy.

QAT moves quantization into training: rounding and truncation are simulated in the forward pass, and backpropagation uses straight-through estimation to let gradients pass through the quantization nodes, allowing the weights to gradually adapt to quantization error during training. It is high-cost and suitable for low-bit-width or activation-error-sensitive models. The trade-off is that it reintroduces dependence on training data and overfitting risk, and the performance after quantization is affected by the training data.

QLoRA takes a different path: it keeps the low-bit base model frozen and trains only high-precision low-rank adapters (LoRA). During fine-tuning, the main GPU memory consumption comes from the base model, so keeping the base model at low bit width can significantly save fine-tuning GPU memory; the adapters have few parameters and high precision, and are responsible for absorbing domain knowledge. Note that during QLoRA training, the base model must first be dequantized to high precision to participate in forward computation, so training and final deployment are not the same graph: at deployment, whether to merge the adapters into the base model and whether to requantize after merging need separate evaluation and cannot be taken for granted.

The applicable conditions of the three can be summarized as follows: PTQ is low-cost and suitable as the starting point for 8/4-bit weight quantization; QAT is high-cost and used for low-bit-width or activation-sensitive scenarios; QLoRA is moderate-cost and used to save fine-tuning GPU memory. In practice, one should start with a low-cost PTQ baseline, and only when critical slices cannot meet the target should one take on the training cost of QAT or introduce other compensation methods.

Between a smaller file and faster execution lies an execution chain. An INT4 weight file is only one quarter the size of FP16, so transfer and storage pressure does drop, but end-to-end throughput may not improve. The reason is that at compute time weights must return to a usable form to participate in multiply-add: quantization must have matching packed formats, low-bit matrix multiplication, and fused kernels on the target GPU or CPU in order to perform low-bit multiply-add directly. Otherwise each step would first dequantize, do format conversion, and additionally issue a kernel launch—starting a compute kernel itself has fixed overhead—these overheads offset the bandwidth benefits from low bit width; this is especially obvious at small batch sizes, because the amortizable fixed overhead is spread over fewer computations.

It is also necessary to distinguish between "weights becoming smaller" and "the entire memory becoming smaller." A reduction in weights does not equal a reduction in KV cache, activations, and framework overhead: the KV for long context, intermediate activations, and runtime overhead remain unchanged, and the savings from weights may be drowned out.

The bottlenecks for throughput and latency are not the same in the two stages of inference. Prefill processes the entire input prompt at once, has a large amount of computation, and is usually more compute-bound, so low-bit weights provide limited help; decode generates token by token, and every generated token must read the weights once, so it is usually more weight-read bandwidth-bound, and the smaller the weights, the more direct the benefit. The bottlenecks of the two stages differ, and therefore the optimization targets also differ.

Therefore, the inputs for evaluating quantization are not only the quantized artifact itself, but also the target hardware, batch, prompt and output length; the outputs should cover first-token latency, per-token latency, throughput, peak GPU memory, power consumption, and load time. Looking at a single number cannot determine the benefit.

Finally, there are service-level boundaries. SLO is a service-committed target for latency, availability, etc. "Being able to fit into GPU memory" only passes the capacity gate, does not equal latency meeting the SLO: the model may fit in GPU memory, but each token may be too slow to interact. CPU offload moves part of the state to host memory, allowing the model to run, but the round-trip latency of PCIe transfers may still make interaction unusable. Capacity, throughput, and latency are three independent gates, and quantization is considered compliant only when it passes all of them simultaneously.

Quantized models are rarely isolated when deployed: they often still have optimizations such as speculative decoding, LoRA merging, and KV compression stacked on top. Each of these optimizations has its own test conclusions, but adding up individual test results does not yield the combined conclusion, because quantization error stacks with subsequent optimizations and they amplify each other.

Take speculative decoding as an example: it relies on output consistency between the draft model and the target model to achieve acceleration. The quantization schemes for the draft and target change how well their outputs match, which in turn changes the acceptance rate; once the acceptance rate changes, the speedup ratio changes, and the original acceleration conclusion no longer holds.

LoRA merging is a more direct coupling. After merging, the weight range changes, but the scale was estimated based on the weight distribution before merging, so the old scale may no longer match the new distribution, and quantization error is amplified again.

KV quantization is coupled with attention error. Attention over long contexts is inherently sensitive to KV precision; KV quantization error stacks with long-context attention error, and conclusions from testing KV compression alone or long context alone cannot be directly extrapolated to the scenario where both are enabled simultaneously.

Under tensor parallelism there is also a layout problem: the quantization layout affects communication, and the partitioning scheme and quantization granularity must be considered together.

The correct approach is to re-quantize or recalibrate based on the actual final artifact, and to perform end-to-end ablation on the complete system after stacking optimizations, rather than adding up results from separate isolated tests.

This leads to an engineering requirement: the quantization recipe must be fully preserved. The base model hash, weight/activation/KV bit widths, symmetry, granularity, group size, calibration set, algorithm, compute dtype, kernel, and hardware each change the final error. If any one of these is missing, results cannot be reproduced even if both models are labeled 'INT4'.

The final boundary is at the evaluation level: do not hide compression loss with more retries. Giving the model more retry opportunities may indeed improve the final success rate, but this is achieved at the expense of cost, latency, and selection bias—selecting good results from multiple samples is itself a form of selection bias, and it masks the degradation of single-shot behavior. When evaluating quantization loss, look at single-shot behavior, not the best result after retries.

A perplexity increase of only 0.2% and degraded tool calling can occur simultaneously, because the model's final behavior is determined not by average error but by local logit ordering. At each generation step, probabilities of candidate tokens are compared; a small quantization error falling on a token near the decision boundary is enough to swap two probabilities that should be adjacent. One flip changes one word: if it falls on a tool name or parameter, it changes the action; if it falls on a safety boundary, it changes whether to refuse. Average perplexity measures the overall closeness of token predictions, and it is insensitive to these few critical flips.

Therefore evaluation cannot look only at a single overall score; it must compare per-sample differences by slice: general capability, target task, rare tokens, language, long context, structured output, calibration, refusal, and adversarial slices. Each slice answers a question: whether this type of behavior changed before and after quantization.

System metrics and quality metrics also need to be precise. TTFT is the time from request to first token; TPOT is the average time per subsequent token during generation; p50, p95, and p99 are latency percentiles, respectively indicating that 50%, 95%, and 99% of requests do not exceed that latency. These metrics are comparable only when inputs are identical: same prompt, same sampling, same kernel, same load. Output should include quality difference before and after quantization, latency interval, throughput, GPU memory, energy consumption, and cost per successful task—quality difference answers "did it change?", while cost-type metrics answer "is it worth it?".

A small change in average perplexity only indicates that overall token predictions are close; it cannot rule out ranking flips in a few critical actions. To distinguish random fluctuation from systematic long-tail damage, per-sample failures must be saved and examined to see which slices are degrading: if failures concentrate in rare language or long-context slices, it is systematic damage; if they are scattered everywhere, it may be random fluctuation.

The release threshold is therefore two hard conditions: hard-risk slices do not degrade, and the system benefit holds on the target hardware. Only when both are satisfied is the quantization configuration a viable solution.

From "why quantize" to "can this quantization go live", there is a complete causal chain in between, and each step takes the output of the previous step as input.

The first step is to determine the quantization target and target hardware. The target determines whether to quantize weights, activations, or KV; the hardware determines which packing formats and low-bit kernels are available, and also determines the environment in which all subsequent system metrics are measured.

The second step is to use representative data to select range and granularity. Calibration data provides the real numerical distribution; the choice of range and granularity (per-tensor, per-channel, or group-wise; symmetric or asymmetric) is settled in this step, and the trade-off between main error and outlier clipping is also completed here.

The third step is to generate candidates using PTQ or QAT. In low-cost scenarios, start with PTQ; only consider the training cost of QAT if critical slices cannot meet the target. The output of this step is a candidate quantized model with a complete recipe.

The fourth step is to inspect errors and critical logits flips. Average error only reflects the overall level; behavior depends on local ordering; check by slice whether hard-risk slices such as rare tokens, languages, long context, structured output, and refusal exhibit ranking flips.

The fifth step is to measure system metrics under target load. With the same prompts, sampling, kernels, and load, measure GPU memory, TTFT, TPOT, latency percentiles, and throughput. Saving GPU memory does not guarantee it is faster; capacity, latency, and throughput are three independent gates.

The sixth step is joint gating: release only when both quality risk and cost per successful task meet the bar. Quality risk answers "will it make mistakes"; cost per successful task answers "whether speed and savings really hold." If either one does not meet the bar, the candidate should be rolled back or abandoned rather than launched with risk.

Each link in this chain may roll back to the previous link when a problem is found: if error is large, reselect the range or granularity; if latency is poor, change hardware or kernels; if risk slices degrade, return to candidate generation. Quantization is not a one-time compression action but a gating loop around error, behavior, and cost.