How to Scale LLM Inference with Ray Serve

How to Scale LLM Inference with Ray Serve

Large Language Models (LLMs) have become the backbone of modern AI applications, but serving them at scale is a formidable engineering challenge. A single inference request can consume gigabytes of GPU memory and take seconds to complete. When traffic spikes, naive deployment setups crumble under latency, memory pressure, and throughput bottlenecks. Ray Serve, the scalable serving library built on top of the Ray distributed computing framework, offers a production-grade solution for deploying LLMs across clusters of machines with fine-grained control over resources, autoscaling, and routing.

What Is Ray Serve?

Ray Serve is a model serving library that lets you deploy machine learning models and Python business logic as distributed services. Unlike traditional web frameworks, Ray Serve is designed specifically for ML workloads. It runs on top of Ray, which means it inherits Ray's ability to distribute computation across thousands of nodes, manage GPU and CPU resources, and recover from failures automatically.

For LLM inference specifically, Ray Serve provides several key capabilities:

• Resource-aware scheduling: Pin replicas to specific GPU types and control how many requests each replica handles concurrently.
• Autoscaling: Scale the number of model replicas up and down based on request queue length or custom metrics.
• Composability: Chain multiple models or preprocessing steps together in a single deployment graph.
• Multi-node deployment: Serve models that are too large for a single GPU by leveraging tensor parallelism across nodes.
• Unified API: The same code runs on your laptop for development and on a production cluster for serving.

Why Scaling LLM Inference Matters

LLM inference is fundamentally different from serving a typical REST API. A single request might generate hundreds of tokens, each requiring a forward pass through a multi-billion-parameter model. This creates several scaling challenges:

• GPU memory pressure: Model weights, KV cache, and activation tensors all compete for limited GPU memory.
• Variable latency: A short prompt might return in 200 milliseconds, while a long generation could take 30 seconds.
• Batching complexity: Throughput improves dramatically when requests are batched, but naive batching delays individual responses.
• Cost sensitivity: GPUs are expensive, so underutilized replicas waste money while insufficient replicas lose users.

Ray Serve addresses these challenges by giving you explicit control over replica count, resource allocation, and request routing, while handling the distributed infrastructure plumbing for you.

Getting Started: A Basic LLM Deployment

Let's start with a simple example. We will deploy a Hugging Face transformer model using Ray Serve. First, install the required packages:

pip install "ray[serve]" transformers torch fastapi uvicorn


Now create a file called serve_llm.py with the following deployment:

from ray import serve
from fastapi import FastAPI
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

app = FastAPI()

@serve.deployment(
    num_replicas=2,
    ray_actor_options={"num_gpus": 1},
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 4,
        "target_num_ongoing_requests_per_replica": 2,
    },
)
@serve.ingress(app)
class LLMDeployment:
    def __init__(self, model_name: str = "meta-llama/Llama-3.2-1B-Instruct"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.float16,
            device_map="auto",
        )

    @app.post("/generate")
    async def generate(self, prompt: str, max_new_tokens: int = 128) -> str:
        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.model.device)
        with torch.no_grad():
            outputs = self.model.generate(
                **inputs,
                max_new_tokens=max_new_tokens,
                do_sample=True,
                temperature=0.7,
            )
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

deployment = LLMDeployment.bind()


Let's break down what is happening here. The @serve.deployment decorator wraps the class as a Ray Serve deployment. The num_replicas=2 argument starts two replicas of the model. The ray_actor_options={"num_gpus": 1} tells Ray to schedule each replica on an actor with one GPU. The autoscaling_config enables dynamic scaling between 1 and 4 replicas based on the number of in-flight requests per replica.

To launch this deployment, run:

serve run serve_llm:deployment


This starts a local Ray Serve instance and binds the deployment to an HTTP endpoint. You can test it with curl:

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain quantum computing in one sentence.", "max_new_tokens": 64}'


Scaling Strategies for LLM Inference

Replica-Based Horizontal Scaling

The simplest scaling strategy is horizontal replication. Each replica loads a full copy of the model and handles requests independently. Ray Serve's autoscaler monitors the request queue and spins up new replicas when demand increases. This works well for models that fit comfortably on a single GPU, such as 7B or 8B parameter models on an A10G or L4 GPU.

The key tuning parameter is target_num_ongoing_requests_per_replica. Setting it too low causes aggressive scaling and idle GPUs. Setting it too high causes queue buildup and latency spikes. For LLM inference, a value between 2 and 8 is typically a good starting point, depending on your model size and latency budget.

Continuous Batching for Higher Throughput

Naive replica scaling has a limitation: each replica processes one request at a time. For higher throughput, you need continuous batching, which dynamically inserts and evicts requests from a batch as they arrive and complete. Ray Serve integrates with high-performance inference engines like vLLM that implement continuous batching natively.

Here is an example deploying a model with vLLM through Ray Serve:

from ray import serve
from fastapi import FastAPI
from vllm import LLM, SamplingParams

app = FastAPI()

@serve.deployment(
    num_replicas=1,
    ray_actor_options={"num_gpus": 1},
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 8,
        "target_num_ongoing_requests_per_replica": 16,
        "upscale_delay_s": 10,
        "downscale_delay_s": 60,
    },
)
@serve.ingress(app)
class VLLMDeployment:
    def __init__(self):
        self.llm = LLM(
            model="meta-llama/Meta-Llama-3-8B-Instruct",
            tensor_parallel_size=1,
            gpu_memory_utilization=0.9,
            max_model_len=4096,
        )

    @app.post("/generate")
    async def generate(self, prompt: str, max_tokens: int = 256, temperature: float = 0.7) -> str:
        sampling_params = SamplingParams(
            max_tokens=max_tokens,
            temperature=temperature,
        )
        outputs = self.llm.generate([prompt], sampling_params)
        return outputs[0].outputs[0].text

deployment = VLLMDeployment.bind()


With vLLM, a single replica can handle dozens of concurrent requests efficiently because vLLM manages the KV cache and batches requests dynamically. The target_num_ongoing_requests_per_replica is set higher at 16 because each replica can absorb more work. The upscale_delay_s and downscale_delay_s parameters prevent thrashing by requiring sustained load changes before scaling actions occur.

Tensor Parallelism for Large Models

Models with 70 billion or more parameters cannot fit on a single GPU. Ray Serve supports tensor parallelism, where model layers are split across multiple GPUs. You configure this by requesting multiple GPUs per replica and telling the inference engine to shard accordingly.

from ray import serve
from fastapi import FastAPI
from vllm import LLM, SamplingParams

app = FastAPI()

@serve.deployment(
    num_replicas=1,
    ray_actor_options={"num_gpus": 4},
)
@serve.ingress(app)
class LargeModelDeployment:
    def __init__(self):
        self.llm = LLM(
            model="meta-llama/Meta-Llama-3-70B-Instruct",
            tensor_parallel_size=4,
            gpu_memory_utilization=0.92,
            max_model_len=8192,
        )

    @app.post("/generate")
    async def generate(self, prompt: str, max_tokens: int = 512) -> str:
        sampling_params = SamplingParams(max_tokens=max_tokens, temperature=0.7)
        outputs = self.llm.generate([prompt], sampling_params)
        return outputs[0].outputs[0].text

deployment = LargeModelDeployment.bind()


In this configuration, each replica consumes four GPUs and vLLM shards the model across them. You can combine tensor parallelism with replica scaling: for example, run two replicas of a 4-way tensor-parallel model on an 8-GPU node. Just be mindful that each replica requires tensor_parallel_size GPUs, so your cluster must have enough free GPUs to satisfy the autoscaler's maximum.

Deploying to a Multi-Node Cluster

For production workloads, you will run Ray Serve on a multi-node cluster. Start by launching a Ray cluster using the Ray CLI:

# On the head node
ray start --head --port=6379

# On each worker node
ray start --address=<head-node-ip>:6379 --num-gpus=8


Then deploy your application to the cluster using serve deploy with a config file. Create config.yaml:

applications:
  - name: llm-service
    import_path: serve_llm:deployment
    route_prefix: /llm
    deployments:
      - name: VLLMDeployment
        num_replicas: 1
        ray_actor_options:
          num_gpus: 1
        autoscaling_config:
          min_replicas: 1
          max_replicas: 8
          target_num_ongoing_requests_per_replica: 16


Deploy with:

serve deploy config.yaml


This command pushes your application to all nodes in the Ray cluster. Ray Serve handles scheduling replicas onto nodes with available GPUs, routing HTTP traffic to the correct replicas, and restarting failed replicas automatically.

Best Practices


  
Right-size your replicas. Match the model size and GPU memory to the instance type. A 7B model in float16 needs roughly 14 GB for weights, so an L4 (24 GB) or A10G (24 GB) is a good fit with room for the KV cache.
  
Use continuous batching engines. For any production LLM workload, prefer vLLM, TGI, or TensorRT-LLM over raw Hugging Face generate. The throughput difference is often 5x to 20x.
  
Tune autoscaling delays. Set upscale_delay_s to avoid reacting to transient bursts, and set downscale_delay_s higher than your typical inter-request gap to avoid premature scale-down.
  
Monitor GPU utilization. Use Ray Dashboard at http://<head-node>:8265 to inspect GPU memory, utilization, and replica health. If GPU memory utilization is consistently below 60 percent, you may be over-provisioned.
  
Set max model length explicitly. In vLLM, max_model_len controls the pre-allocated KV cache size. Setting it to your actual maximum context length prevents wasted GPU memory.
  
Handle cold starts. Model loading can take minutes for large models. Use graceful_shutdown_wait_loop_s in your deployment config to give in-flight requests time to finish before a replica is torn down.
  
Separate preprocessing from inference. Use Ray Serve's deployment graph to put tokenization or input validation on a CPU-only deployment, reserving GPU replicas exclusively for model forward passes.
  
Pin to specific GPU types. In heterogeneous clusters, use Ray's placement group constraints or ray_actor_options with resource labels to ensure replicas land on the right hardware.


Conclusion

Scaling LLM inference is as much an infrastructure problem as it is a model problem. Ray Serve gives you a unified framework to manage replicas, GPUs, autoscaling, and multi-node deployment without rewriting your application logic. By combining Ray Serve's scheduling capabilities with high-performance inference engines like vLLM, you can serve large language models with low latency, high throughput, and efficient GPU utilization. Start with a single replica on your local machine, validate your pipeline, then scale out to a production cluster with confidence that the same deployment code will work across environments. As your traffic grows, tune your autoscaling parameters and batching strategy iteratively, guided by the metrics available in the Ray Dashboard, to find the sweet spot between cost and performance.