AWQ vs GPTQ: Comparing Quantization Techniques for Production

Introduction to Model Quantization

As large language models (LLMs) grow in size, deploying them in production environments becomes increasingly challenging. A 70B parameter model in FP16 precision requires roughly 140GB of VRAM — well beyond what most single-GPU setups can handle. Quantization addresses this by reducing the precision of model weights from 16-bit floating point to 4-bit or 8-bit integers, dramatically shrinking memory footprint and improving inference throughput with minimal accuracy loss.

Two of the most popular weight-only quantization techniques in the LLM ecosystem are GPTQ and AWQ. Both compress models to 4-bit (or 3-bit) representations, but they take fundamentally different approaches to deciding which weights matter most. This tutorial walks through how each works, when to use them, and how to apply them in practice.

What Is GPTQ?

GPTQ (Generative Pre-trained Transformer Quantization) is a post-training quantization method based on approximate second-order information. Introduced in the 2022 paper "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers," it builds on the earlier OBQ (Optimal Brain Quantization) framework but scales it efficiently to models with billions of parameters.

The core idea is to quantize weights column by column, using the Hessian matrix of the layer's input activations to estimate the impact of each quantization error. When a weight is quantized, the resulting error is compensated by adjusting the remaining unquantized weights in the same row. This "error compensation" step is what allows GPTQ to maintain accuracy even at aggressive bit widths like 3-bit.

Key Characteristics of GPTQ

• Calibration-based: Requires a small calibration dataset (typically 128-1024 samples) to compute Hessian statistics.
• Layer-wise: Processes one transformer layer at a time, making memory usage predictable.
• Group quantization: Supports per-channel and group-wise quantization (e.g., group size 128) for finer granularity.
• Accuracy: Excellent retention at 4-bit; noticeable degradation below 3-bit on smaller models.

What Is AWQ?

AWQ (Activation-aware Weight Quantization) is a more recent technique introduced in the 2023 paper "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." Rather than using second-order error compensation, AWQ takes a simpler but surprisingly effective approach: it observes that not all weights are equally important, and the important ones can be identified by looking at the magnitude of activations that flow through them.

The key insight is that a small fraction of weights (around 0.1-1%) are "salient" — they interact with large activation magnitudes and thus have outsized impact on model output. AWQ protects these salient channels by applying a per-channel scaling factor before quantization. The scaling pushes important weights into a range where quantization error is smaller, while an inverse scaling on the activations keeps the overall computation equivalent.

Key Characteristics of AWQ

• Activation-aware: Uses activation statistics to identify and protect salient weight channels.
• No weight adjustment: Does not modify unquantized weights — only applies scaling factors.
• Hardware-friendly: The scaling approach maps well to efficient GPU kernels, enabling faster inference.
• Accuracy: Often slightly better than GPTQ at 4-bit, especially on smaller models; strong at 3-bit.

Why Quantization Matters in Production

In production deployments, quantization is rarely optional for large models. The benefits extend beyond just fitting a model onto available hardware:

• Memory reduction: 4-bit quantization reduces weight memory by ~4x compared to FP16, enabling a 70B model to run on a single 80GB GPU.
• Inference speed: Lower memory bandwidth requirements translate to faster token generation, especially in memory-bound autoregressive decoding.
• Cost savings: Smaller GPU footprints mean lower cloud compute bills and the ability to serve more models per node.
• Edge deployment: Quantization is often the only way to run capable models on consumer GPUs or edge devices.

The choice between GPTQ and AWQ can meaningfully affect both the quality of model outputs and the throughput of your serving stack. Understanding the tradeoffs is essential for making the right production decision.

Comparing AWQ and GPTQ

Accuracy

Both techniques preserve most of the model's capability at 4-bit, but AWQ generally has a slight edge, particularly on smaller models (7B and below) and at lower bit widths (3-bit). GPTQ's error compensation is powerful but can accumulate errors in layers with unusual activation patterns. AWQ's scaling approach is more robust because it directly targets the weights that matter most.

On larger models (30B+), the difference narrows considerably — both techniques produce near-lossless 4-bit models. For 3-bit quantization, AWQ is typically the safer choice.

Inference Speed

AWQ was designed with kernel efficiency in mind. The AWQ authors provide optimized CUDA kernels that fuse the dequantization and matrix multiplication steps, resulting in faster inference than most GPTQ implementations. In practice, AWQ models served through vLLM or the AWQ-native inference engine often achieve 1.2-2x higher throughput than equivalent GPTQ models.

GPTQ has broader kernel support (including through ExLlamaV2 and AutoGPTQ), and on some hardware configurations the speed difference is negligible. But if raw throughput is your priority, AWQ has the advantage.

Quantization Speed

AWQ quantization is significantly faster than GPTQ. The GPTQ process involves computing the inverse Hessian and performing sequential weight updates, which is computationally expensive. AWQ only needs to search for the optimal scaling factor per layer, which is a much lighter operation. Quantizing a 7B model with AWQ might take 10-20 minutes, while GPTQ can take 30-60 minutes depending on calibration set size and group settings.

Ecosystem and Tooling

GPTQ has been around longer and has a more mature ecosystem. AutoGPTQ provides a clean Python API, and pre-quantized GPTQ models are widely available on Hugging Face. ExLlamaV2 offers some of the fastest GPTQ inference kernels available.

AWQ's ecosystem is catching up rapidly. The official AWQ library, vLLM integration, and Hugging Face Transformers support make it straightforward to use. Pre-quantized AWQ models from providers like TheBloke and casperhansen are readily available.

How to Quantize a Model with GPTQ

Below is a practical example using AutoGPTQ to quantize a Hugging Face model. You'll need to install the required packages first:

pip install auto-gptq transformers optimum accelerate

Here is a complete script for quantizing a model with GPTQ:

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer
from datasets import load_dataset
import torch

# Model to quantize
model_id = "meta-llama/Llama-2-7b-hf"

# Configure quantization parameters
quantize_config = BaseQuantizeConfig(
    bits=4,              # Target bit width
    group_size=128,      # Group size for group-wise quantization
    desc_act=False,      # Whether to use activation order reordering
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Prepare calibration data
# Using a small subset of a general text dataset
calibration_dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")

def format_examples(dataset, tokenizer, num_samples=128, max_length=512):
    examples = []
    for i, example in enumerate(dataset):
        if i >= num_samples:
            break
        text = example["text"].strip()
        if len(text) < 50:
            continue
        tokenized = tokenizer(
            text,
            return_tensors="pt",
            max_length=max_length,
            truncation=True,
            padding="max_length",
        )
        examples.append({
            "input_ids": tokenized["input_ids"],
            "attention_mask": tokenized["attention_mask"],
        })
    return examples

calibration_data = format_examples(calibration_dataset, tokenizer)

# Load the model in FP16
model = AutoGPTQForCausalLM.from_pretrained(
    model_id,
    quantize_config,
    trust_remote_code=True,
)

# Run quantization
model.quantize(calibration_data)

# Save the quantized model
output_dir = "./llama-2-7b-gptq-4bit"
model.save_quantized(output_dir)
tokenizer.save_pretrained(output_dir)

print(f"Quantized model saved to {output_dir}")

To load and run inference with the quantized GPTQ model:

from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer

model_path = "./llama-2-7b-gptq-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoGPTQForCausalLM.from_quantized(
    model_path,
    device_map="auto",
    use_safetensors=True,
)

prompt = "Explain the concept of quantization in machine learning."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

How to Quantize a Model with AWQ

For AWQ, we'll use the official llm-awq package or the more commonly used autoawq library. Install it first:

pip install autoawq transformers accelerate

Here is a complete AWQ quantization script:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
import torch

# Model to quantize
model_id = "meta-llama/Llama-2-7b-hf"
output_dir = "./llama-2-7b-awq-4bit"

# AWQ quantization configuration
quant_config = {
    "zero_point": True,       # Use zero points for asymmetric quantization
    "q_group_size": 128,      # Group size, similar to GPTQ
    "w_bit": 4,               # Weight bit width
    "version": "GEMM",        # Kernel version: GEMM or GEMV
}

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Load the model
model = AutoAWQForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    trust_remote_code=True,
)

# Define calibration data
# AWQ works well with diverse text; a few hundred samples is sufficient
calibration_text = [
    "The quick brown fox jumps over the lazy dog.",
    "Machine learning is a subset of artificial intelligence.",
    "Quantization reduces the precision of neural network weights.",
    # In practice, load 128-256 diverse samples from a dataset
]

# Format calibration data for the tokenizer
def get_calibration_samples(tokenizer, texts, max_length=512):
    samples = []
    for text in texts:
        tokenized = tokenizer(
            text,
            return_tensors="pt",
            max_length=max_length,
            truncation=True,
        )
        samples.append(tokenized["input_ids"])
    return samples

calibration_samples = get_calibration_samples(tokenizer, calibration_text)

# Run quantization
model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data=calibration_text,
)

# Save the quantized model
model.save_quantized(output_dir)
tokenizer.save_pretrained(output_dir)

print(f"AWQ quantized model saved to {output_dir}")

To load and run inference with the AWQ model:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "./llama-2-7b-awq-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoAWQForCausalLM.from_quantized(
    model_path,
    device_map="auto",
    fuse_layers=True,  # Fuse layers for faster inference
)

prompt = "Explain the concept of quantization in machine learning."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Serving Quantized Models with vLLM

For production deployment, vLLM is one of the best serving frameworks and supports both GPTQ and AWQ models natively. Here's how to serve each:

Serving an AWQ Model

# Start the vLLM server with an AWQ model
python -m vllm.entrypoints.openai.api_server \
    --model casperhansen/llama-3-8b-instruct-awq \
    --quantization awq \
    --dtype float16 \
    --max-model-len 4096 \
    --port 8000

Serving a GPTQ Model

# Start the vLLM server with a GPTQ model
python -m vllm.entrypoints.openai.api_server \
    --model TheBloke/Llama-2-7B-Chat-GPTQ \
    --quantization gptq \
    --dtype float16 \
    --max-model-len 4096 \
    --port 8000

Once the server is running, you can send requests using the OpenAI-compatible API:

import requests

response = requests.post(
    "http://localhost:8000/v1/chat/completions",
    json={
        "model": "casperhansen/llama-3-8b-instruct-awq",
        "messages": [
            {"role": "user", "content": "What are the benefits of model quantization?"}
        ],
        "max_tokens": 300,
        "temperature": 0.7,
    },
)

print(response.json()["choices"][0]["message"]["content"])

Best Practices

Choose the Right Bit Width

4-bit quantization is the sweet spot for both GPTQ and AWQ. It provides roughly 4x memory reduction with negligible accuracy loss on most models. Reserve 3-bit for situations where memory is extremely constrained, and be prepared to evaluate accuracy carefully. 8-bit quantization is rarely worth the effort compared to simply using FP16, as the memory savings are modest.

Use a Representative Calibration Set

Both techniques rely on calibration data to understand activation patterns. Use 128-256 diverse samples that are representative of your actual inference workload. For general-purpose models, datasets like WikiText, C4, or RedPajama work well. For domain-specific models, use in-domain text. Avoid using too many samples — diminishing returns set in quickly and quantization time increases.

Set an Appropriate Group Size

The group size parameter controls how many weights share a single quantization scale. A smaller group size (e.g., 64) provides better accuracy but slightly larger model files and slower inference. A larger group size (e.g., 256) is more efficient but may lose more accuracy. The default of 128 is a good starting point for most use cases.

Always Evaluate on Your Task

Never assume that quantization is lossless. After quantizing, run your model through your actual evaluation pipeline — whether that's perplexity on a held-out set, accuracy on a benchmark, or human evaluation on production queries. The accuracy impact of quantization varies by model architecture, task, and even fine-tuning history.

Consider Pre-Quantized Models

Before quantizing yourself, check if a high-quality pre-quantized model already exists on Hugging Face. Community contributors have already quantized most popular models with both GPTQ and AWQ, often with carefully tuned configurations. This saves time and ensures you benefit from others' calibration efforts.

Enable Kernel Fusion for AWQ

When serving AWQ models, always enable layer fusion (fuse_layers=True in AutoAWQ or the equivalent in vLLM). This fuses attention and MLP operations, significantly improving inference speed. Without fusion, AWQ models may actually be slower than their GPTQ counterparts.

Monitor for Degradation in Edge Cases

Quantization can disproportionately affect rare or out-of-distribution inputs. If your application handles diverse user queries, test specifically on edge cases — long contexts, unusual languages, code generation, and mathematical reasoning. These are the areas where quantization artifacts are most likely to surface.

Conclusion

Both GPTQ and AWQ are mature, production-ready quantization techniques that can reduce LLM memory footprint by 4x with minimal accuracy loss. GPTQ offers a longer track record, broader kernel support, and a well-established ecosystem, making it a safe choice for teams already invested in that toolchain. AWQ delivers faster quantization, slightly better accuracy at low bit widths, and superior inference throughput thanks to its hardware-friendly scaling approach, making it the preferred choice for new deployments where inference speed is critical. For most production scenarios starting today, AWQ is the recommended default — but the best choice ultimately depends on your specific model, hardware, and accuracy requirements. Whichever you choose, always validate with proper evaluation, use representative calibration data, and leverage optimized serving frameworks like vLLM to get the most out of your quantized models.