Scaling CloudFront: From Prototype to Production

Scaling CloudFront: From Prototype to Production

AWS CloudFront is Amazon's Content Delivery Network (CDN) that securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. While spinning up a CloudFront distribution for a prototype is straightforward, scaling it to production requires careful planning around caching, security, origin protection, observability, and cost optimization. This tutorial walks you through the journey from a basic prototype to a hardened, production-grade CloudFront setup.

Why Scaling CloudFront Matters

In a prototype, you might create a single distribution pointing to an S3 bucket and call it a day. However, as traffic grows and real users depend on your application, several concerns emerge:

• Latency variability: Poor cache hit ratios mean more origin fetches and slower responses.
• Origin overload: Without proper caching and shielding, your origin servers can be overwhelmed by traffic spikes.
• Security exposure: Publicly accessible origins and missing security headers create attack surfaces.
• Cost surprises: Misconfigured cache behaviors and unnecessary data transfer can inflate bills significantly.
• Operational blind spots: Without logging and metrics, diagnosing issues becomes guesswork.

Scaling CloudFront means addressing each of these concerns systematically so your CDN becomes a reliable, secure, and cost-efficient layer in your architecture.

Understanding the CloudFront Architecture

At its core, CloudFront consists of edge locations that cache content close to users, and origins that serve as the source of truth for your content. When a user requests a resource, CloudFront checks its edge cache. On a cache hit, it serves the content immediately. On a miss, it fetches from the origin, caches the response according to cache rules, and returns it to the user.

A production CloudFront setup typically involves multiple cache behaviors, origin access controls, Lambda@Edge or CloudFront Functions for request manipulation, WAF for security, and comprehensive logging. Let's build this up step by step.

Step 1: The Prototype Distribution

Here is a minimal CloudFront distribution created with Terraform that points to an S3 bucket. This is the kind of setup you might use for a prototype or demo.

# prototype.tf
resource "aws_cloudfront_distribution" "prototype" {
  enabled             = true
  is_ipv6_enabled     = true
  comment             = "Prototype distribution"
  default_root_object = "index.html"

  origin {
    domain_name = aws_s3_bucket.prototype.bucket_regional_domain_name
    origin_id   = "s3-prototype"

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.prototype.cloudfront_access_identity_path
    }
  }

  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "s3-prototype"

    forwarded_values {
      query_string = false
      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "allow-all"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    cloudfront_default_certificate = true
  }
}

This works, but it has several production problems: it allows HTTP, uses the default CloudFront certificate, has no logging, no WAF, and no fine-grained cache behaviors. Let's evolve it.

Step 2: Securing the Origin with Origin Access Control

In production, your S3 bucket should never be publicly accessible. CloudFront Origin Access Control (OAC) is the modern replacement for Origin Access Identity (OAI) and supports SSE-KMS encrypted buckets. Here is how to set it up:

# origin-access.tf
resource "aws_cloudfront_origin_access_control" "main" {
  name                              = "production-oac"
  description                       = "OAC for production S3 origin"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

resource "aws_s3_bucket_policy" "production" {
  bucket = aws_s3_bucket.production.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "AllowCloudFrontServicePrincipal"
        Effect    = "Allow"
        Principal = {
          Service = "cloudfront.amazonaws.com"
        }
        Action   = "s3:GetObject"
        Resource = "${aws_s3_bucket.production.arn}/*"
        Condition = {
          StringEquals = {
            "AWS:SourceArn" = aws_cloudfront_distribution.production.arn
          }
        }
      }
    ]
  })
}

This ensures that only your specific CloudFront distribution can read from the S3 bucket. Direct access to the bucket is denied, closing a common security gap.

Step 3: Multiple Cache Behaviors for Different Content Types

Production applications serve different types of content with different caching needs. Static assets like images and CSS can be cached aggressively, while API responses need shorter TTLs or no caching at all. You can define multiple cache behaviors using path patterns:

# cache-behaviors.tf
resource "aws_cloudfront_distribution" "production" {
  enabled             = true
  is_ipv6_enabled     = true
  comment             = "Production distribution"
  default_root_object = "index.html"

  # Static assets origin (S3)
  origin {
    domain_name = aws_s3_bucket.production.bucket_regional_domain_name
    origin_id   = "s3-static"
    origin_access_control_id = aws_cloudfront_origin_access_control.main.id
  }

  # API origin (ALB or API Gateway)
  origin {
    domain_name = aws_lb.api.dns_name
    origin_id   = "alb-api"

    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  # Default behavior: HTML files, short cache
  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD", "OPTIONS"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "s3-static"

    cache_policy_id = aws_cloudfront_cache_policy.html.id

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 300
    max_ttl                = 600
  }

  # Static assets: long cache, immutable
  ordered_cache_behavior {
    path_pattern     = "/assets/*"
    allowed_methods  = ["GET", "HEAD", "OPTIONS"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "s3-static"

    cache_policy_id          = aws_cloudfront_cache_policy.assets.id
    compress                 = true
    viewer_protocol_policy   = "redirect-to-https"
  }

  # API: no caching, pass everything through
  ordered_cache_behavior {
    path_pattern     = "/api/*"
    allowed_methods  = ["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "alb-api"

    cache_policy_id          = aws_cloudfront_cache_policy.api.id
    origin_request_policy_id = aws_cloudfront_origin_request_policy.api.id
    viewer_protocol_policy   = "https-only"
  }

  price_class = "PriceClass_100"
  retain_on_delete = false

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn      = aws_acm_certificate.production.arn
    ssl_support_method       = "sni"
    minimum_protocol_version = "TLSv1.2_2021"
  }
}

Defining Cache Policies

Cache policies control what CloudFront caches and for how long. Using managed policies is a good starting point, but custom policies give you precise control:

# policies.tf
resource "aws_cloudfront_cache_policy" "html" {
  name        = "html-cache-policy"
  comment     = "Short cache for HTML documents"
  default_ttl = 300
  max_ttl     = 600
  min_ttl     = 0

  parameters_in_cache_key_and_forwarded_to_origin {
    cookies_config {
      cookie_behavior = "none"
    }

    headers_config {
      header_behavior = "none"
    }

    query_strings_config {
      query_string_behavior = "none"
    }

    enable_accept_encoding_brotli = true
    enable_accept_encoding_gzip   = true
  }
}

resource "aws_cloudfront_cache_policy" "assets" {
  name        = "assets-cache-policy"
  comment     = "Long cache for immutable static assets"
  default_ttl = 86400
  max_ttl     = 31536000
  min_ttl     = 86400

  parameters_in_cache_key_and_forwarded_to_origin {
    cookies_config {
      cookie_behavior = "none"
    }

    headers_config {
      header_behavior = "none"
    }

    query_strings_config {
      query_string_behavior = "none"
    }

    enable_accept_encoding_brotli = true
    enable_accept_encoding_gzip   = true
  }
}

resource "aws_cloudfront_cache_policy" "api" {
  name        = "api-cache-policy"
  comment     = "Minimal caching for API responses"
  default_ttl = 0
  max_ttl     = 60
  min_ttl     = 0

  parameters_in_cache_key_and_forwarded_to_origin {
    cookies_config {
      cookie_behavior = "all"
    }

    headers_config {
      header_behavior = "whitelist"
      headers {
        items = ["Authorization", "Content-Type"]
      }
    }

    query_strings_config {
      query_string_behavior = "all"
    }
  }
}

resource "aws_cloudfront_origin_request_policy" "api" {
  name    = "api-origin-request-policy"
  comment = "Forward all request data to API origin"

  cookies_config {
    cookie_behavior = "all"
  }

  headers_config {
    header_behavior = "all"
  }

  query_strings_config {
    query_string_behavior = "all"
  }
}

Step 4: Adding Security with AWS WAF

For production, you should attach AWS WAF (Web Application Firewall) to your distribution. WAF protects against common web exploits like SQL injection, cross-site scripting, and rate-based attacks:

# waf.tf
resource "aws_wafv2_web_acl" "production" {
  name        = "production-waf"
  description = "WAF for production CloudFront"
  scope       = "CLOUDFRONT"

  default_action {
    allow {}
  }

  rule {
    name     = "rate-limit"
    priority = 1

    action {
      block {}
    }

    statement {
      rate_based_statement {
        limit              = 2000
        aggregate_key_type = "IP"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "rate-limit"
      sampled_requests_enabled   = true
    }
  }

  rule {
    name     = "aws-managed-common-rules"
    priority = 2

    override_action {
      none {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesCommonRuleSet"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "common-rules"
      sampled_requests_enabled   = true
    }
  }

  rule {
    name     = "aws-managed-sqli"
    priority = 3

    override_action {
      none {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesSQLiRuleSet"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "sqli-rules"
      sampled_requests_enabled   = true
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "production-waf"
    sampled_requests_enabled   = true
  }
}

resource "aws_cloudfront_distribution" "production" {
  # ... other configuration ...

  web_acl_id = aws_wafv2_web_acl.production.arn
}

Step 5: Adding Security Headers with CloudFront Functions

Security headers like Content-Security-Policy, Strict-Transport-Security, and X-Content-Type-Options should be present on every response. CloudFront Functions are lightweight JavaScript functions that run at the edge and are ideal for this task:

// security-headers.js
function handler(event) {
  var response = event.response;
  var headers = response.headers;

  headers["strict-transport-security"] = {
    value: "max-age=63072000; includeSubdomains; preload"
  };
  headers["x-content-type-options"] = {
    value: "nosniff"
  };
  headers["x-frame-options"] = {
    value: "DENY"
  };
  headers["x-xss-protection"] = {
    value: "1; mode=block"
  };
  headers["referrer-policy"] = {
    value: "strict-origin-when-cross-origin"
  };
  headers["content-security-policy"] = {
    value: "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self'; frame-ancestors 'none';"
  };

  return response;
}

Deploy this function and attach it as a response trigger to your cache behaviors:

# cloudfront-function.tf
resource "aws_cloudfront_function" "security_headers" {
  name    = "security-headers"
  runtime = "cloudfront-js-1.0"
  comment = "Add security headers to all responses"
  publish = true
  code    = file("security-headers.js")
}

# Reference in your distribution:
# ordered_cache_behavior {
#   ...
#   function_association {
#     event_type   = "viewer-response"
#     function_arn = aws_cloudfront_function.security_headers.arn
#   }
# }

Step 6: Enabling Logging and Monitoring

Observability is critical in production. CloudFront supports both standard logging (S3) and real-time logging (Kinesis Data Streams). For production, real-time logging gives you immediate visibility:

# logging.tf
resource "aws_kinesis_stream" "cloudfront_logs" {
  name             = "cloudfront-realtime-logs"
  shard_count      = 2
  retention_period = 48

  shard_level_metrics = [
    "IncomingBytes",
    "IncomingRecords",
  ]
}

resource "aws_cloudfront_realtime_log_config" "production" {
  name          = "production-realtime-logs"
  sampling_rate = 100
  fields        = [
    "timestamp",
    "c-ip",
    "time-to-first-byte",
    "sc-status",
    "sc-bytes",
    "cs-method",
    "cs-uri-stem",
    "cs-uri-query",
    "x-edge-location",
    "x-edge-request-id",
    "x-host-header",
    "cs-protocol",
    "cs-headers",
    "cache-hit-result",
    "origin-fbl",
  ]

  endpoint {
    stream_type = "Kinesis"
    kinesis_stream_config {
      role_arn   = aws_iam_role.cloudfront_logging.arn
      stream_arn = aws_kinesis_stream.cloudfront_logs.arn
    }
  }
}

# Attach to distribution:
# resource "aws_cloudfront_distribution" "production" {
#   ...
#   realtime_log_config_arn = aws_cloudfront_realtime_log_config.production.arn
# }

Additionally, enable standard access logs to S3 for long-term archival and analysis with Athena:

resource "aws_cloudfront_distribution" "production" {
  # ...
  
  logging_config {
    include_cookies = false
    bucket          = aws_s3_bucket.logs.bucket_regional_domain_name
    prefix          = "cloudfront/"
  }
}

Key Metrics to Monitor

• Cache Hit Ratio: The percentage of requests served from cache. Aim for 90%+ for static content.
• 4xx and 5xx Error Rates: Spikes indicate origin issues or misconfigured paths.
• Origin Latency: High values suggest your origin needs scaling or optimization.
• Request Count by Edge Location: Helps identify geographic traffic patterns.
• Data Transfer Out: Directly impacts cost; monitor for unexpected spikes.

Step 7: Cache Invalidation Strategies

When you deploy new content, you need to invalidate cached content. There are several strategies:

Versioned filenames: The best approach for static assets. Use content hashes in filenames (e.g., app.a3f5b2c1.js) so new deployments create new URLs that are automatically cached without invalidation.

Path invalidation: For HTML files that share the same name across deployments, use targeted invalidation:

# invalidate.sh
#!/bin/bash
DISTRIBUTION_ID="E123ABCDEF456"

aws cloudfront create-invalidation \
  --distribution-id $DISTRIBUTION_ID \
  --paths "/*" \
  --region us-east-1

For more targeted invalidation in a CI/CD pipeline:

# invalidate-specific.sh
#!/bin/bash
DISTRIBUTION_ID="E123ABCDEF456"

# Only invalidate changed paths
aws cloudfront create-invalidation \
  --distribution-id $DISTRIBUTION_ID \
  --paths "/index.html" "/sitemap.xml" "/feed.xml"

Wildcard invalidations like /assets/* are useful but count as a single invalidation path. Be aware that AWS allows 1,000 free invalidation paths per month; additional paths incur charges.

Step 8: Lambda@Edge for Advanced Use Cases

For more complex logic that CloudFront Functions cannot handle (such as making network requests or handling larger payloads), Lambda@Edge is the right tool. A common use case is rewriting URLs for single-page applications:

// lambda-edge-rewrite.js
'use strict';

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request;
  const uri = request.uri;

  // Check if the URI doesn't have a file extension
  if (!uri.includes('.')) {
    request.uri = '/index.html';
  }

  // Redirect old API paths to new ones
  if (uri.startsWith('/v1/')) {
    request.uri = uri.replace('/v1/', '/v2/');
  }

  callback(null, request);
};

Deploy this as a Lambda function in us-east-1 and attach it as an origin-request trigger:

# lambda-edge.tf
resource "aws_lambda_function" "url_rewrite" {
  filename         = "lambda-edge-rewrite.zip"
  source_code_hash = data.archive_file.url_rewrite.output_base64sha256
  function_name    = "url-rewrite-edge"
  role             = aws_iam_role.lambda_edge.arn
  handler          = "lambda-edge-rewrite.handler"
  runtime          = "nodejs18.x"
  publish          = true
}

# Reference in distribution:
# ordered_cache_behavior {
#   ...
#   lambda_function_association {
#     event_type   = "origin-request"
#     lambda_arn   = aws_lambda_function.url_rewrite.qualified_arn
#     include_body = false
#   }
# }

Best Practices for Production CloudFront

Optimize Cache Hit Ratios

High cache hit ratios are the single most impactful metric for both performance and cost. To maximize them:

• Only include in the cache key what actually affects the response. Avoid forwarding unnecessary headers, cookies, or query strings.
• Use separate cache behaviors for content with different caching characteristics.
• Enable compression (Brotli and Gzip) to reduce transfer sizes.
• Set appropriate Cache-Control headers on your origin responses. CloudFront respects these headers when your cache policy uses them.
• Use stale-while-revalidate and stale-if-error cache directives to serve stale content while refreshing in the background.

Choose the Right Price Class

CloudFront charges differently based on edge location regions. If your users are primarily in North America and Europe, use PriceClass_100 to avoid the higher costs of edge locations in South America, Australia, and Asia. If you need global coverage, use PriceClass_All:

resource "aws_cloudfront_distribution" "production" {
  # Use PriceClass_100 for NA + Europe only
  # Use PriceClass_200 for NA + Europe + Asia + Middle East + Africa
  # Use PriceClass_All for all edge locations
  price_class = "PriceClass_100"
}

Use Origin Shield for Better Origin Protection

Origin Shield is an additional caching layer that sits between CloudFront edge locations and your origin. It consolidates origin requests, reducing load on your origin and improving cache hit ratios:

resource "aws_cloudfront_origin_access_control" "main" {
  # ... existing config ...
}

resource "aws_cloudfront_distribution" "production" {
  # ...
  
  origin {
    domain_name = aws_s3_bucket.production.bucket_regional_domain_name
    origin_id   = "s3-static"
    origin_shield {
      enabled              = true
      origin_shield_region = "us-east-1"
    }
    origin_access_control_id = aws_cloudfront_origin_access_control.main.id
  }
}

Implement Proper Error Handling

Configure custom error responses so users see branded error pages instead of generic CloudFront errors:

resource "aws_cloudfront_distribution" "production" {
  # ...

  custom_error_response {
    error_code            = 404
    response_code         = 200
    response_page_path    = "/404.html"
    error_caching_min_ttl = 300
  }

  custom_error_response {
    error_code            = 500
    response_code         = 503
    response_page_path    = "/500.html"
    error_caching_min_ttl = 10
  }

  custom_error_response {
    error_code            = 502
    response_code         = 503
    response_page_path    = "/500.html"
    error_caching_min_ttl = 10
  }
}

Use Continuous Deployment Policies

For zero-downtime deployments, CloudFront supports continuous deployment policies that let you test changes on a staging distribution before promoting to production:

resource "aws_cloudfront_continuous_deployment_policy" "production" {
  enabled = true

  staging_distribution_dns_names {
    items    = [aws_cloudfront_distribution.staging.domain_name]
    quantity = 1
  }

  traffic_config {
    type = "SingleWeight"
    single_weight_config {
      weight = 0.05
    }
  }
}

This routes 5% of traffic to your staging distribution, allowing you to validate changes before a full rollout.

Secure with Field-Level Encryption

For sensitive data like credit card numbers or PII submitted through forms, use field-level encryption to encrypt specific fields at the edge before they reach your origin:

resource "aws_cloudfront_field_level_encryption_profile" "production" {
  name    = "production-fle-profile"
  comment = "Encrypt sensitive form fields"

  encryption_entities {
    items {
      public_key_id = aws_cloudfront_public_key.production.id
      provider_id   = "key-provider-1"
      field_patterns {
        items = ["credit_card_number", "ssn", "email"]
      }
    }
  }
}

Step 9: Cost Optimization Strategies

As you scale, CloudFront costs can grow significantly. Here are key strategies to keep costs under control:

• Maximize cache hit ratios: Every cache hit is significantly cheaper than an origin fetch. Even a 5% improvement in cache hit ratio can reduce costs substantially at scale.
• Use appropriate price classes: Don't pay for edge locations in regions where you have no users.
• Enable compression: Compressed responses transfer less data, reducing data transfer costs.
• Monitor invalidation usage: Stay within the 1,000 free paths per month. Use versioned filenames to minimize invalidations.
• Consider Reserved Capacity: If you have predictable, high traffic, CloudFront Security Edge Compute or data transfer commitments can provide significant discounts.
• Review Lambda@Edge invocation counts: These functions are invoked on every request and can become expensive. Use CloudFront Functions where possible, as they are significantly cheaper.

Step 10: Disaster Recovery and Failover

For high availability, configure origin failover so CloudFront automatically switches to a backup origin when the primary is unavailable:

# failover.tf
resource "aws_cloudfront_origin_group" "production" {
  origin_id = "origin-group-production"

  member {
    origin_id = "s3-primary"
  }

  member {
    origin_id = "s3-secondary"
  }

  failover_criteria {
    status_codes = [403, 404, 500, 502, 503, 504]
  }
}

resource "aws_cloudfront_distribution" "production" {
  # Primary origin
  origin {
    domain_name = aws_s3_bucket.primary.bucket_regional_domain_name
    origin_id   = "s3-primary"
    origin_access_control_id = aws_cloudfront_origin_access_control.main.id
  }

  # Secondary (failover) origin
  origin {
    domain_name = aws_s3_bucket.secondary.bucket_regional_domain_name
    origin_id   = "s3-secondary"
    origin_access_control_id = aws_cloudfront_origin_access_control.main.id
  }

  default_cache_behavior {
    # Point to the origin group instead of a single origin
    target_origin_id = aws_cloudfront_origin_group.production.origin_id
    # ... rest of config ...
  }
}

Putting It All Together

Here is a summary checklist for moving from prototype to production:

• Restrict S3 origin access using Origin Access Control (OAC)
• Redirect all HTTP traffic to HTTPS with a proper ACM certificate
• Define separate cache behaviors for HTML, static assets, and API routes
• Create custom cache and origin request policies tailored to each content type
• Attach AWS WAF with managed rule sets and rate limiting
• Add security response headers using CloudFront Functions
• Enable real-time logging to Kinesis and standard logs to S3
• Set up CloudWatch alarms for error rates, cache hit ratio, and origin latency
• Configure custom error pages for 4xx and 5xx responses
• Enable Origin Shield to reduce origin load and improve cache efficiency
• Use versioned filenames for static assets to minimize invalidation costs
• Configure origin failover for high availability
• Choose the appropriate price class for your user geography
• Set up a staging distribution with continuous deployment for safe rollouts

Scaling CloudFront from a prototype to a production-grade CDN involves much more than just creating a distribution. It requires thoughtful cache policy design, robust security layers, comprehensive observability, cost-conscious configuration, and resilient failover strategies. By following the steps and best practices outlined in this tutorial, you can build a CloudFront setup that delivers content fast, stays secure under attack, provides deep operational visibility, and scales economically with your business. The investment in proper configuration pays off quickly through better performance, lower costs, and fewer incidents as your traffic grows.