Introduction to Infrastructure as Code for LLM Deployment
Large Language Models (LLMs) have become a cornerstone of modern AI applications, but deploying them at scale introduces significant infrastructure challenges. From GPU-provisioned compute instances to load balancers, object storage for model weights, and networking configurations, the surface area of required infrastructure is substantial. Managing this manually through cloud consoles is error-prone, non-reproducible, and difficult to audit.
Infrastructure as Code (IaC) solves these problems by allowing you to define your infrastructure declaratively using configuration files. Terraform, developed by HashiCorp, is the most widely adopted IaC tool and supports all major cloud providers including AWS, GCP, and Azure. This tutorial walks you through using Terraform to deploy production-grade infrastructure for hosting LLMs.
Why Terraform for LLM Workloads
LLM deployments have unique characteristics that make IaC especially valuable:
- GPU scarcity and cost: GPU instances (e.g., NVIDIA A100, H100) are expensive and often capacity-constrained. Terraform lets you spin instances up and down predictably, reducing idle costs.
- Complex networking: LLM inference servers need low-latency internal networking, private subnets, and controlled egress for downloading model artifacts from Hugging Face or S3.
- Reproducibility: Training and inference environments must be identical across development, staging, and production to avoid subtle bugs.
- Multi-region failover: Serving LLMs globally often requires deployments across multiple regions, which Terraform handles through reusable modules.
- Security posture: IAM roles, security groups, and secrets management can be version-controlled and peer-reviewed.
Prerequisites
Before you begin, ensure you have the following installed and configured:
- Terraform CLI (version 1.5 or later)
- An AWS account with programmatic access (access key and secret key)
- AWS CLI configured with your credentials
- Basic familiarity with Terraform syntax and AWS services
Project Structure
A well-organized Terraform project improves maintainability as your LLM platform grows. Here is the structure we will build:
llm-terraform/
├── main.tf
├── variables.tf
├── outputs.tf
├── providers.tf
├── vpc.tf
├── storage.tf
├── compute.tf
├── loadbalancer.tf
├── scripts/
│ └── user_data.sh
└── environments/
├── dev.tfvars
└── prod.tfvars
Configuring the Provider
Start by defining the AWS provider and the backend for storing Terraform state. State management is critical for team collaboration — never store state locally in production.
# providers.tf
terraform {
required_version = ">= 1.5.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
backend "s3" {
bucket = "my-llm-terraform-state"
key = "llm-infra/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-locks"
encrypt = true
}
}
provider "aws" {
region = var.aws_region
default_tags {
tags = {
Project = "llm-deployment"
Environment = var.environment
ManagedBy = "terraform"
}
}
}
Defining Variables
Parameterize your configuration so the same code can target different environments. For LLM workloads, the most important variables relate to instance type, model size, and scaling thresholds.
# variables.tf
variable "aws_region" {
description = "AWS region for deployment"
type = string
default = "us-east-1"
}
variable "environment" {
description = "Deployment environment (dev, staging, prod)"
type = string
validation {
condition = contains(["dev", "staging", "prod"], var.environment)
error_message = "Environment must be dev, staging, or prod."
}
}
variable "instance_type" {
description = "EC2 instance type for LLM inference"
type = string
default = "g5.2xlarge"
}
variable "min_instances" {
description = "Minimum number of inference instances"
type = number
default = 1
}
variable "max_instances" {
description = "Maximum number of inference instances"
type = number
default = 4
}
variable "model_bucket_name" {
description = "S3 bucket for storing model weights"
type = string
}
variable "vpc_cidr" {
description = "CIDR block for the VPC"
type = string
default = "10.0.0.0/16"
}
Networking Layer
The VPC configuration creates isolated network segments. LLM inference instances live in private subnets with no direct internet access, while a NAT gateway allows them to download model weights and dependencies. An Application Load Balancer sits in public subnets to receive client traffic.
# vpc.tf
resource "aws_vpc" "llm_vpc" {
cidr_block = var.vpc_cidr
enable_dns_support = true
enable_dns_hostnames = true
}
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.llm_vpc.id
}
# Public subnets for the load balancer
resource "aws_subnet" "public" {
count = 2
vpc_id = aws_vpc.llm_vpc.id
cidr_block = cidrsubnet(var.vpc_cidr, 8, count.index)
availability_zone = data.aws_availability_zones.available.names[count.index]
map_public_ip_on_launch = true
tags = {
Name = "llm-public-${count.index + 1}"
}
}
# Private subnets for inference instances
resource "aws_subnet" "private" {
count = 2
vpc_id = aws_vpc.llm_vpc.id
cidr_block = cidrsubnet(var.vpc_cidr, 8, count.index + 10)
availability_zone = data.aws_availability_zones.available.names[count.index]
tags = {
Name = "llm-private-${count.index + 1}"
}
}
# NAT gateway for outbound traffic from private subnets
resource "aws_eip" "nat" {
count = 2
domain = "vpc"
}
resource "aws_nat_gateway" "nat" {
count = 2
allocation_id = aws_eip.nat[count.index].id
subnet_id = aws_subnet.public[count.index].id
depends_on = [aws_internet_gateway.igw]
}
resource "aws_route_table" "private" {
count = 2
vpc_id = aws_vpc.llm_vpc.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway[count.index].id
}
}
resource "aws_route_table_association" "private" {
count = 2
subnet_id = aws_subnet.private[count.index].id
route_table_id = aws_route_table.private[count.index].id
}
resource "aws_route_table" "public" {
vpc_id = aws_vpc.llm_vpc.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
}
resource "aws_route_table_association" "public" {
count = 2
subnet_id = aws_subnet.public[count.index].id
route_table_id = aws_route_table.public.id
}
data "aws_availability_zones" "available" {
state = "available"
}
Storage for Model Weights
Model weights for large models can be tens or hundreds of gigabytes. Storing them in S3 with appropriate lifecycle policies keeps costs manageable while providing fast access during instance startup.
# storage.tf
resource "aws_s3_bucket" "model_weights" {
bucket = var.model_bucket_name
}
resource "aws_s3_bucket_versioning" "model_weights" {
bucket = aws_s3_bucket.model_weights.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "model_weights" {
bucket = aws_s3_bucket.model_weights.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
}
}
}
resource "aws_s3_bucket_public_access_block" "model_weights" {
bucket = aws_s3_bucket.model_weights.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
Compute Resources with GPU Instances
This is the core of the deployment. We use an Auto Scaling Group of GPU instances running a user-data script that installs the inference server (for example, vLLM or TGI) and downloads model weights from S3.
# compute.tf
data "aws_ami" "nvidia_gpu" {
most_recent = true
owners = ["amazon"]
filter {
name = "name"
values = ["amzn2-ami-ecs-gpu-hvm-*"]
}
}
resource "aws_security_group" "inference" {
name = "llm-inference-sg"
description = "Security group for LLM inference instances"
vpc_id = aws_vpc.llm_vpc.id
ingress {
from_port = 8000
to_port = 8000
protocol = "tcp"
security_groups = [aws_security_group.alb.id]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_iam_role" "inference" {
name = "llm-inference-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}
]
})
}
resource "aws_iam_role_policy" "inference_s3" {
name = "llm-inference-s3-access"
role = aws_iam_role.inference.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"s3:GetObject",
"s3:ListBucket"
]
Resource = [
aws_s3_bucket.model_weights.arn,
"${aws_s3_bucket.model_weights.arn}/*"
]
}
]
})
}
resource "aws_iam_instance_profile" "inference" {
name = "llm-inference-profile"
role = aws_iam_role.inference.id
}
resource "aws_launch_template" "inference" {
name = "llm-inference-template"
image_id = data.aws_ami.nvidia_gpu.id
instance_type = var.instance_type
vpc_security_group_ids = [aws_security_group.inference.id]
iam_instance_profile {
arn = aws_iam_instance_profile.inference.arn
}
user_data = base64encode(templatefile("${path.module}/scripts/user_data.sh", {
model_bucket = var.model_bucket_name
model_name = "meta-llama/Llama-2-7b-chat-hf"
}))
tag_specifications {
resource_type = "instance"
tags = {
Name = "llm-inference"
}
}
}
resource "aws_autoscaling_group" "inference" {
name = "llm-inference-asg"
vpc_zone_identifier = aws_subnet.private[*].id
min_size = var.min_instances
max_size = var.max_instances
desired_capacity = var.min_instances
launch_template {
id = aws_launch_template.inference.id
version = "$Latest"
}
target_group_arns = [aws_lb_target_group.inference.arn]
tag {
key = "Name"
value = "llm-inference"
propagate_at_launch = true
}
}
User Data Script
The user-data script runs on first boot. It installs dependencies, pulls model weights from S3, and starts the inference server.
#!/bin/bash
set -euo pipefail
# Install Docker
amazon-linux-extras install docker -y
systemctl enable docker
systemctl start docker
# Install AWS CLI v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install
# Download model weights from S3
mkdir -p /opt/models
aws s3 sync s3://${model_bucket}/models/ /opt/models/
# Run vLLM inference server
docker run -d \
--gpus all \
--shm-size 8g \
-p 8000:8000 \
-v /opt/models:/models \
--name vllm-server \
vllm/vllm-openai:latest \
--model /models/${model_name} \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 1
Load Balancer Configuration
An Application Load Balancer distributes requests across inference instances and performs health checks. This ensures traffic only reaches healthy nodes.
# loadbalancer.tf
resource "aws_security_group" "alb" {
name = "llm-alb-sg"
description = "Security group for the LLM load balancer"
vpc_id = aws_vpc.llm_vpc.id
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_lb" "inference" {
name = "llm-inference-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.alb.id]
subnets = aws_subnet.public[*].id
}
resource "aws_lb_target_group" "inference" {
name = "llm-inference-tg"
port = 8000
protocol = "HTTP"
vpc_id = aws_vpc.llm_vpc.id
health_check {
enabled = true
path = "/health"
port = "8000"
protocol = "HTTP"
interval = 30
healthy_threshold = 2
unhealthy_threshold = 3
timeout = 10
matcher = "200"
}
}
resource "aws_lb_listener" "https" {
load_balancer_arn = aws_lb.inference.arn
port = 443
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06"
certificate_arn = aws_acm_certificate_validation.cert.certificate_arn
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.inference.arn
}
}
Auto Scaling Based on GPU Utilization
LLM inference is GPU-bound. Scaling based on GPU utilization ensures you add capacity when inference requests queue up and remove capacity when idle. Because CloudWatch does not natively report GPU metrics, you need the NVIDIA DCGM exporter running on each instance to push metrics to CloudWatch.
# Add to compute.tf
resource "aws_cloudwatch_metric_alarm" "gpu_high" {
alarm_name = "llm-gpu-utilization-high"
comparison_operator = "GreaterThanOrEqualToThreshold"
evaluation_periods = 2
metric_name = "GPUUtilization"
namespace = "LLM/Inference"
period = 60
statistic = "Average"
threshold = 80
alarm_description = "Scale up when GPU utilization is high"
dimensions = {
AutoScalingGroupName = aws_autoscaling_group.inference.name
}
alarm_actions = [aws_autoscaling_policy.scale_up.arn]
}
resource "aws_cloudwatch_metric_alarm" "gpu_low" {
alarm_name = "llm-gpu-utilization-low"
comparison_operator = "LessThanOrEqualToThreshold"
evaluation_periods = 5
metric_name = "GPUUtilization"
namespace = "LLM/Inference"
period = 60
statistic = "Average"
threshold = 30
alarm_description = "Scale down when GPU utilization is low"
dimensions = {
AutoScalingGroupName = aws_autoscaling_group.inference.name
}
alarm_actions = [aws_autoscaling_policy.scale_down.arn]
}
resource "aws_autoscaling_policy" "scale_up" {
name = "llm-scale-up"
scaling_adjustment = 1
adjustment_type = "ChangeInCapacity"
cooldown = 300
autoscaling_group_name = aws_autoscaling_group.inference.name
}
resource "aws_autoscaling_policy" "scale_down" {
name = "llm-scale-down"
scaling_adjustment = -1
adjustment_type = "ChangeInCapacity"
cooldown = 300
autoscaling_group_name = aws_autoscaling_group.inference.name
}
Outputs
Outputs expose useful information after deployment, such as the load balancer DNS name and the S3 bucket ARN.
# outputs.tf
output "load_balancer_dns" {
description = "DNS name of the LLM inference load balancer"
value = aws_lb.inference.dns_name
}
output "model_bucket_arn" {
description = "ARN of the S3 bucket storing model weights"
value = aws_s3_bucket.model_weights.arn
}
output "inference_endpoint" {
description = "Full HTTPS endpoint for the inference API"
value = "https://${aws_lb.inference.dns_name}/v1"
}
Environment-Specific Configuration
Use tfvars files to customize deployments per environment. Development can use smaller instances and a single availability zone, while production uses high-memory GPU instances across multiple zones.
# environments/dev.tfvars
aws_region = "us-east-1"
environment = "dev"
instance_type = "g5.xlarge"
min_instances = 1
max_instances = 2
model_bucket_name = "my-llm-models-dev"
# environments/prod.tfvars
aws_region = "us-east-1"
environment = "prod"
instance_type = "g5.12xlarge"
min_instances = 2
max_instances = 8
model_bucket_name = "my-llm-models-prod"
Deploying the Infrastructure
With all files in place, deploy the infrastructure using the standard Terraform workflow. Always review the plan before applying changes.
# Initialize the working directory
terraform init
# Validate the configuration
terraform validate
# Format the code
terraform fmt -recursive
# Review the execution plan for production
terraform plan -var-file=environments/prod.tfvars
# Apply the configuration
terraform apply -var-file=environments/prod.tfvars
After the apply completes, Terraform prints the outputs. You can test the inference endpoint with a curl request:
curl -X POST https://$(terraform output -raw load_balancer_dns)/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-2-7b-chat-hf",
"messages": [{"role": "user", "content": "Hello, what is infrastructure as code?"}]
}'
Best Practices
Use Modules for Reusability
As your LLM platform grows, extract repeated patterns into modules. For example, create a module for the inference server that accepts instance type, model name, and scaling parameters as inputs. This lets you deploy multiple model variants (a small fast model and a large high-quality model) from the same code.
Pin Provider Versions
Always pin provider versions in your required_providers block. Unpinned providers can introduce breaking changes that silently alter your infrastructure.
Secure Your State
Terraform state may contain sensitive values like database passwords or API keys. Use a remote backend with encryption enabled, restrict IAM access to the state bucket, and enable state locking with DynamoDB to prevent concurrent modifications.
Implement Cost Controls
GPU instances are expensive. Use terraform destroy for ephemeral dev environments, set up AWS Budgets alerts, and consider Spot Instances for non-critical workloads. You can add a lifecycle block with prevent_destroy = true on production resources to avoid accidental deletion.
Run Terraform in CI/CD
Integrate Terraform into your CI/CD pipeline using tools like GitHub Actions, GitLab CI, or Terraform Cloud. Every infrastructure change should go through a pull request with automated terraform fmt, terraform validate, and terraform plan checks before merge.
Tag Everything
Consistent tagging enables cost allocation, resource discovery, and automated cleanup. The default_tags block in the provider configuration ensures every resource inherits project and environment tags automatically.
Handle GPU Capacity Constraints
GPU instances frequently face capacity shortages. Configure your Auto Scaling Group with multiple instance types using mixed instances policies, and set up capacity reservations in production to guarantee availability during traffic spikes.
Conclusion
Infrastructure as Code with Terraform provides a reliable, reproducible, and auditable foundation for deploying LLM inference infrastructure. By codifying your VPC, storage, GPU compute, load balancing, and auto scaling configurations, you eliminate manual provisioning errors and enable rapid iteration across environments. The configuration presented in this tutorial is a starting point — as your platform matures, you can extend it with modules for multiple model variants, integrate it with monitoring tools like Prometheus and Grafana, and add sophisticated deployment strategies such as blue-green rollouts for model updates. The key takeaway is that treating your LLM infrastructure as version-controlled software gives you the same engineering rigor for operations that you already apply to application code, ultimately leading to more reliable and cost-effective AI deployments.