← Back to DevBytes
Developer Tutorial August 18, 2026 9 min read

Deploying Quantized Models with Hugging Face TGI

Deploying Quantized Models with Hugging Face TGI

Large language models (LLMs) have transformed how developers build AI-powered applications, but their sheer size makes production deployment a serious engineering challenge. A 70-billion-parameter model in FP16 precision requires roughly 140 GB of GPU memory — well beyond what a single consumer or mid-tier datacenter GPU can offer. Quantization, combined with Hugging Face's Text Generation Inference (TGI) server, offers a practical path to running these models efficiently on accessible hardware without sacrificing meaningful quality.

What Is Hugging Face TGI?

Text Generation Inference (TGI) is a Rust- and Python-based inference server developed by Hugging Face specifically for serving LLMs in production. Unlike generic serving frameworks, TGI is purpose-built for text generation workloads and ships with optimizations that make it a popular choice for enterprise deployments.

Key capabilities of TGI include:

What Is Model Quantization?

Quantization is the process of reducing the numerical precision used to represent a model's weights (and sometimes activations). Instead of storing each weight as a 16-bit or 32-bit floating-point number, quantization represents them using 8-bit, 4-bit, or even 2-bit integers. This dramatically reduces memory footprint and often improves inference speed because lower-precision math is faster on modern GPUs.

The main quantization formats you will encounter when working with TGI are:

Why Quantization Matters in Production

For most teams, the choice is not between "quantized" and "unquantized" — it is between "quantized and deployable" and "unquantized and unaffordable." Quantization matters for several concrete reasons:

The tradeoff is a small degradation in output quality. For most conversational and summarization tasks, 4-bit quantization is virtually indistinguishable from full precision. For math, code, or highly precise reasoning tasks, you may want to benchmark carefully before committing.

Prerequisites

Before deploying a quantized model with TGI, ensure you have the following:

Verify your GPU is visible to Docker:

docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

Deploying a bitsandbytes-Quantized Model

The simplest way to get started is to let TGI quantize a model on the fly using bitsandbytes. This requires no pre-quantized weights — you point TGI at any standard model repository and add a single flag.

docker run --gpus all --shm-size 1g -p 8080:80 \
  -e HF_TOKEN=hf_your_token_here \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --quantize bitsandbytes \
  --max-input-length 4096 \
  --max-total-tokens 8192

The --quantize bitsandbytes flag instructs TGI to load weights in 4-bit NF4 format. This is ideal for quick prototyping and works with almost any model on the Hub. The downside is that on-the-fly quantization adds startup time and the bitsandbytes kernel is generally slower than GPTQ or AWQ for inference.

Deploying a Pre-Quantized GPTQ Model

For better inference performance, use a model that has already been quantized using GPTQ. The Hugging Face Hub hosts many community-quantized checkpoints. TGI detects the quantization config from the model repository automatically.

docker run --gpus all --shm-size 1g -p 8080:80 \
  -e HF_TOKEN=hf_your_token_here \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
  --quantize gptq \
  --max-input-length 4096 \
  --max-total-tokens 8192

Notice the --quantize gptq flag. TGI will load the pre-quantized safetensors and use optimized GPTQ kernels during inference. Startup is faster than bitsandbytes because no quantization happens at load time.

Deploying an AWQ-Quantized Model

AWQ is often the best choice for production because it tends to offer higher throughput than GPTQ while maintaining comparable accuracy. Deployment is nearly identical:

docker run --gpus all --shm-size 1g -p 8080:80 \
  -e HF_TOKEN=hf_your_token_here \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model casperhansen/llama-3-70b-instruct-awq \
  --quantize awq \
  --max-input-length 4096 \
  --max-total-tokens 8192

A 70B AWQ model typically fits in around 40 GB of GPU memory, making it deployable on a single A100 80GB or two A6000 GPUs with tensor parallelism.

Using EETQ for 8-Bit Inference

If you want a middle ground between full precision and aggressive 4-bit quantization, EETQ provides high-throughput 8-bit weight-only quantization with negligible quality loss. It is especially effective on Ampere and Hopper GPUs.

docker run --gpus all --shm-size 1g -p 8080:80 \
  -e HF_TOKEN=hf_your_token_here \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --quantize eetq \
  --max-input-length 4096 \
  --max-total-tokens 8192

EETQ quantizes weights on the fly at load time, so startup is slower than using a pre-quantized checkpoint but faster than bitsandbytes for actual inference.

Multi-GPU Deployment with Tensor Parallelism

For models that exceed the memory of a single GPU, TGI supports tensor parallelism via the --num-shard flag. This splits the model across GPUs and is fully compatible with quantized weights.

docker run --gpus all --shm-size 1g -p 8080:80 \
  -e HF_TOKEN=hf_your_token_here \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model casperhansen/llama-3-70b-instruct-awq \
  --quantize awq \
  --num-shard 2 \
  --max-input-length 4096 \
  --max-total-tokens 8192

This command splits the 70B AWQ model across two GPUs. Make sure both GPUs have identical memory capacity and architecture for best performance.

Querying the TGI Server

Once TGI is running, you can interact with it using a simple HTTP client. The server exposes both a native API and an OpenAI-compatible endpoint at /v1/chat/completions.

Using the OpenAI-compatible endpoint with Python:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed-for-local"
)

response = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain what a KV cache is in two sentences."}
    ],
    max_tokens=200,
    temperature=0.7,
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

Using the native TGI API with curl:

curl -X POST http://localhost:8080/generate \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": "What are the benefits of model quantization?",
    "parameters": {
      "max_new_tokens": 256,
      "temperature": 0.7,
      "top_p": 0.9,
      "do_sample": true
    }
  }'

Monitoring and Health Checks

TGI exposes a health endpoint and a metrics endpoint that you should integrate into your deployment pipeline. The metrics endpoint is Prometheus-compatible and exposes counters for request latency, queue depth, and GPU utilization.

# Health check
curl http://localhost:8080/health

# Prometheus metrics
curl http://localhost:8080/metrics

Useful metrics to watch include tgi_request_duration_seconds, tgi_queue_size, and tgi_request_count. These help you decide when to scale horizontally by adding more TGI replicas behind a load balancer.

Best Practices for Production Deployments

Based on real-world experience deploying quantized models with TGI, the following practices consistently lead to better outcomes:

Common Pitfalls and How to Avoid Them

Even with a solid understanding of TGI and quantization, a few issues recur frequently:

A Complete Docker Compose Example

For production deployments, Docker Compose provides a clean way to manage configuration, environment variables, and restart policies. Below is a complete example you can adapt:

version: "3.9"

services:
  tgi:
    image: ghcr.io/huggingface/text-generation-inference:2.3.0
    container_name: tgi-server
    restart: unless-stopped
    ports:
      - "8080:80"
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - HUGGING_FACE_HUB_CACHE=/data
    volumes:
      - ./hf-cache:/data
    shm_size: 1g
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command:
      - --model=casperhansen/llama-3-70b-instruct-awq
      - --quantize=awq
      - --num-shard=2
      - --max-input-length=4096
      - --max-total-tokens=8192
      - --max-batch-prefill-tokens=8192
      - --prefix-cache-len=4096
      - --log-level=info

Start the stack with:

export HF_TOKEN=hf_your_token_here
docker compose up -d

Conclusion

Deploying quantized models with Hugging Face TGI is one of the most effective ways to bring large language models into production without breaking your hardware budget. By understanding the tradeoffs between bitsandbytes, GPTQ, AWQ, and EETQ, you can choose the right quantization strategy for your specific latency, throughput, and quality requirements. Combined with TGI's continuous batching, PagedAttention, and tensor parallelism, even 70-billion-parameter models become practical to serve on commodity GPUs. Start with a pre-quantized AWQ checkpoint, benchmark it against your real workloads, and iterate on configuration parameters like --max-total-tokens and prefix caching until you hit your target performance. With careful tuning and the practices outlined above, you can build inference infrastructure that is both cost-efficient and production-grade.

— Ad —

Google AdSense will appear here after approval

← Back to all articles