Quantizing Small Language Models for Maximum Speed

Small language models (SLMs) — models in the 1B to 8B parameter range — have become the workhorses of edge AI, on-device assistants, and cost-sensitive inference pipelines. But even a "small" 7B model in float16 requires roughly 14 GB of memory and can be painfully slow on consumer hardware. Quantization is the single most effective technique for shrinking these models and accelerating inference without sacrificing meaningful accuracy. This tutorial walks through what quantization is, why it matters for SLMs specifically, and how to apply it in practice.

What Is Quantization?

Quantization is the process of reducing the numerical precision of a model's weights and, optionally, its activations. Most language models are trained in float16 or bfloat16, where each parameter occupies 16 bits. Quantization compresses these values into lower-bit representations — typically 8-bit integers (int8), 4-bit integers (int4), or even sub-4-bit formats like 3-bit and 2-bit in aggressive setups.

There are two main flavors of quantization:

Post-Training Quantization (PTQ): Applied after training is complete. Fast, requires no retraining, and is the standard approach for deploying SLMs.
Quantization-Aware Training (QAT): Simulates quantization during training so the model learns to compensate for precision loss. More accurate but more expensive.

For SLMs, PTQ is almost always the right choice. The models are small enough that the accuracy hit from PTQ is minimal, and the speed gains are substantial.

Why Quantization Matters for Small Models

For large frontier models, quantization is often a trade-off: you save memory but may lose benchmark points. For small models, the calculus is different. SLMs are typically deployed in latency-sensitive or memory-constrained environments where raw speed and footprint matter more than squeezing out the last fraction of a percent on MMLU.

The benefits compound:

Memory reduction: A 7B model goes from ~14 GB in fp16 to ~3.5 GB in int4, fitting comfortably on a single consumer GPU or even in CPU RAM.
Inference speed: Lower-bit weights mean less memory bandwidth consumed per token. Memory bandwidth, not compute, is usually the bottleneck for autoregressive generation.
Energy efficiency: Critical for laptops, phones, and embedded devices.
Higher batch throughput: Smaller models allow larger batch sizes within the same memory budget.

The key insight is that autoregressive decoding is memory-bound. Each generated token requires reading the entire model weight matrix from memory. Halving the bit-width roughly doubles the effective bandwidth available for those weights, which translates directly into faster token generation.

Popular Quantization Formats

Several quantization schemes have emerged as de facto standards:

GGUF (GPT-Generated Unified Format): Used by llama.cpp. Supports CPU and GPU inference with mixed precision. Excellent for CPU-only or hybrid setups.
GPTQ: A PTQ method that uses second-order error compensation. Produces 4-bit models with minimal accuracy loss. Best for GPU inference.
AWQ (Activation-aware Weight Quantization): Protects the most salient weight channels based on activation magnitudes. Often outperforms GPTQ at 4-bit.
BitsAndBytes (NF4): The NormalFloat 4-bit format used in QLoRA. Simple to apply, good for fine-tuning workflows.
EXL2: A variable-bit format that allows fractional bitrates (e.g., 4.5 bits per weight) for fine-grained size/quality trade-offs.

How to Quantize a Small Language Model

Let's walk through three practical approaches: loading a pre-quantized model with BitsAndBytes, quantizing a model with AWQ, and converting a model to GGUF for CPU inference.

Approach 1: Loading with BitsAndBytes NF4

The simplest way to get a quantized model running is to load a standard HuggingFace checkpoint with 4-bit quantization on the fly. This requires no pre-processing and works well for prototyping.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "meta-llama/Llama-3.2-3B"

# Configure 4-bit NF4 quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

# Quick generation test
inputs = tokenizer("Explain quantization in one sentence:", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The bnb_4bit_use_double_quant=True flag applies a second round of quantization to the quantization constants themselves, squeezing out a bit more memory savings. The bnb_4bit_compute_dtype controls the dtype used during the actual matrix multiplications — float16 or bfloat16 are standard choices.

Approach 2: Quantizing with AWQ

For production GPU inference, AWQ typically gives the best quality-to-size ratio at 4-bit. The autoawq library makes it straightforward to quantize a model and save it for later use.

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Llama-3.2-3B"
quant_path = "llama-3.2-3b-awq"

# AWQ quantization configuration
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",
}

# Load model and tokenizer
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Quantize — this uses calibration data to identify salient weights
model.quantize(
    tokenizer,
    quant_config=quant_config,
)

# Save the quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
print(f"Quantized model saved to {quant_path}")

The q_group_size parameter controls the granularity of per-group scaling factors. A group size of 128 is a common default — smaller groups improve accuracy but increase overhead. The version field selects the kernel implementation; GEMM is optimized for batched inference on GPUs.

To load and run the saved AWQ model later:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

quant_path = "llama-3.2-3b-awq"

model = AutoAWQForCausalLM.from_quantized(
    quant_path,
    fuse_layers=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(quant_path)

inputs = tokenizer("Write a haiku about inference speed:", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The fuse_layers=True option merges adjacent operations (like attention and layer norm) into fused kernels, which significantly reduces kernel launch overhead and improves throughput.

Approach 3: Converting to GGUF for CPU and Hybrid Inference

If you are targeting CPU inference — for laptops, servers without GPUs, or edge devices — GGUF is the format of choice. The llama.cpp project provides conversion scripts that take a HuggingFace model and produce a quantized GGUF file.

# Clone llama.cpp and build the conversion tools
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# Download a model in fp16 GGUF format first
# (or convert from safetensors)
python convert_hf_to_gguf.py /path/to/llama-3.2-3b --outfile llama-3.2-3b-fp16.gguf

# Quantize to 4-bit (Q4_K_M is the recommended balance)
./llama-quantize llama-3.2-3b-fp16.gguf llama-3.2-3b-q4_k_m.gguf Q4_K_M

# Run inference
./llama-cli -m llama-3.2-3b-q4_k_m.gguf -p "What is quantization?" -n 100

The Q4_K_M quantization type is widely recommended: it uses 4-bit quantization for most layers but keeps attention and feed-forward layers at slightly higher precision via a "medium" importance weighting. Other common types include Q4_0 (fastest, lowest quality), Q5_K_M (better quality, slightly larger), and Q8_0 (near-lossless, 8-bit).

Measuring the Impact

Always benchmark before and after quantization. The following script measures tokens-per-second for a quantized model using HuggingFace's generate with a fixed prompt.

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def benchmark(model, tokenizer, prompt, num_tokens=128):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    # Warmup
    _ = model.generate(**inputs, max_new_tokens=10)
    
    torch.cuda.synchronize() if torch.cuda.is_available() else None
    start = time.time()
    outputs = model.generate(**inputs, max_new_tokens=num_tokens, do_sample=False)
    torch.cuda.synchronize() if torch.cuda.is_available() else None
    elapsed = time.time() - start
    
    tps = num_tokens / elapsed
    print(f"Generated {num_tokens} tokens in {elapsed:.2f}s")
    print(f"Throughput: {tps:.1f} tokens/second")
    return tps

prompt = "The quick brown fox jumps over the lazy dog. " * 20
benchmark(model, tokenizer, prompt)

Compare the fp16 baseline against your quantized version. For a 3B model on a consumer GPU, you should expect roughly 1.5x to 2.5x speedup going from fp16 to int4, with the exact factor depending on your hardware's memory bandwidth and the quantization kernel efficiency.

Best Practices

Start with 4-bit. Int4 is the sweet spot for SLMs. Going to 8-bit gives speed gains but leaves memory savings on the table; going below 4-bit (3-bit, 2-bit) starts to degrade quality noticeably on smaller models.
Prefer AWQ or GPTQ for GPU, GGUF for CPU. Match the format to your target hardware. Using the wrong format can negate the speed benefits.
Use calibration data that matches your use case. AWQ and GPTQ both use calibration samples to determine which weights are most important. If your model will be used for code generation, calibrate with code snippets rather than generic text.
Keep a fp16 reference model for evaluation. Run your evaluation suite against both the quantized and full-precision models. If the quantized model drops more than 1-2 points on your key metrics, consider a higher bit-width or a different quantization method.
Enable fused kernels. Layer fusion, KV cache quantization, and flash attention all stack with weight quantization. Combine them for maximum throughput.
Watch out for tokenizer and embedding overhead. In very small models (under 1B parameters), the embedding and LM head layers can dominate inference time. Some quantization tools skip these layers by default, which is usually the right call.
Consider EXL2 for fine-grained control. If you need to hit a specific memory budget (e.g., fitting in 4 GB VRAM), EXL2's variable bitrate lets you dial in the exact size you need.
Re-test after every change. Quantization interacts with batching, sequence length, and sampling parameters. A configuration that is fast for short prompts may behave differently for long-context generation.

Conclusion

Quantizing small language models is one of the highest-leverage optimizations available to developers. By moving from 16-bit to 4-bit precision, you can cut memory usage by roughly 4x, double token generation throughput, and unlock deployment on hardware that could never run the original model. The ecosystem has matured to the point where quantization requires only a few lines of code — whether you choose BitsAndBytes for quick prototyping, AWQ for production GPU inference, or GGUF for CPU and edge deployment. The key is to treat quantization as part of your evaluation pipeline: benchmark both speed and quality, calibrate with representative data, and choose the format that matches your target hardware. Done well, quantization lets small models punch far above their weight class.