Designing a URL Shortener for 1 Million Users

Designing a URL Shortener for 1 Million Users

What Is a URL Shortener?

A URL shortener is a web service that transforms long, unwieldy URLs into compact, easily shareable links. Behind the scenes, it stores a mapping between the original URL and a generated short code. When a user clicks the shortened link, the service looks up the original URL and redirects them seamlessly. Think of services like Bit.ly or TinyURL — they take a 200-character tracking URL and condense it into something like https://short.ly/x7K9pQ.

Why Designing for Scale Matters

When you're building for one million users, every architectural decision ripples across performance, cost, and reliability. A million users might generate:

• 10–50 million URLs created per month
• 500+ million redirects per month (reads far exceed writes)
• Peak traffic of thousands of requests per second

Without proper design, you'll hit bottlenecks in database lookups, exhaust your primary keys, or suffer from hash collisions. A system built for 1,000 users will crumble under this load. This tutorial walks through building a production-ready URL shortener that handles massive scale while remaining fast, reliable, and cost-effective.

Core System Architecture

The high-level architecture consists of these components:

• API Layer — REST endpoints for creating and resolving short URLs
• Short Code Generator — produces unique, collision-free identifiers
• Database — stores URL mappings with high read throughput
• Cache Layer — absorbs the majority of redirect lookups
• Redirect Engine — performs fast 301 redirects with minimal latency

Step 1: Choosing the Short Code Generation Strategy

This is the heart of your system. You need unique, short, and URL-safe codes. There are three main approaches:

Approach A: Hash-Based (MD5 / SHA256 + Truncation)

Hash the URL, then take the first N characters. Simple but prone to collisions when truncated aggressively. For 1 million users generating millions of URLs, collision probability rises quickly.

// Node.js example — hash-based approach (NOT recommended for scale)
const crypto = require('crypto');

function generateShortCode(url) {
  const hash = crypto.createHash('sha256').update(url).digest('hex');
  // Take first 8 chars — collision risk grows with volume
  return hash.substring(0, 8);
}
// Problem: 8 hex chars = 16^8 ≈ 4.29 billion combinations
// But with millions of URLs, birthday paradox makes collisions likely

Approach B: Distributed Counter with Base62 Encoding

Maintain a distributed counter (e.g., using ZooKeeper or a dedicated counter service). Each server instance claims a range of IDs, then converts them to Base62. This guarantees uniqueness and gives you short, readable codes.

// Base62 encoding — converts integer IDs to short alphanumeric strings
const BASE62_ALPHABET = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';

function encodeBase62(num) {
  if (num === 0) return BASE62_ALPHABET[0];
  let result = '';
  while (num > 0) {
    result = BASE62_ALPHABET[num % 62] + result;
    num = Math.floor(num / 62);
  }
  return result;
}

function decodeBase62(str) {
  let num = 0;
  for (let i = 0; i < str.length; i++) {
    num = num * 62 + BASE62_ALPHABET.indexOf(str[i]);
  }
  return num;
}

// Examples:
// encodeBase62(1) → "1"
// encodeBase62(1000000) → "4c92"
// encodeBase62(1000000000) → "15FTgN"

For a counter-based system, each application instance claims a block of IDs:

// Distributed counter using Redis INCR
const redis = require('ioredis');
const client = new redis();

class IDGenerator {
  constructor(serviceId) {
    this.serviceId = serviceId;
    this.currentMax = 0;
    this.currentOffset = 0;
  }

  async claimIDBlock(blockSize = 10000) {
    // Atomically increment the global counter
    const newMax = await client.incrby('global:url_id_counter', blockSize);
    this.currentOffset = newMax - blockSize;
    this.currentMax = newMax;
    return this.currentOffset;
  }

  async getNextID() {
    if (this.currentOffset >= this.currentMax) {
      await this.claimIDBlock();
    }
    const id = this.currentOffset;
    this.currentOffset++;
    return id;
  }

  async generateShortCode() {
    const id = await this.getNextID();
    return encodeBase62(id);
  }
}

// Usage
const generator = new IDGenerator('instance-3');
const shortCode = await generator.generateShortCode(); // e.g., "L2xP9k"

Approach C: Pre-Generated Code Pool (Snowflake-Style)

Generate codes in advance and store them in a queue. When a URL needs shortening, pop a code from the pool. This decouples code generation from API response time but adds operational complexity.

Step 2: Database Schema Design

For 1 million users, you need a schema optimized for extremely fast reads and write throughput. Use a primary key on the short code and add proper indexing.

-- PostgreSQL schema for URL shortener (1M+ users)
CREATE TABLE url_mappings (
    short_code VARCHAR(10) PRIMARY KEY,   -- Base62 encoded ID
    original_url TEXT NOT NULL,
    user_id BIGINT,                        -- optional: track which user created it
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    expires_at TIMESTAMP WITH TIME ZONE,   -- optional: auto-expiring links
    click_count BIGINT DEFAULT 0,          -- denormalized counter for analytics
    last_accessed_at TIMESTAMP WITH TIME ZONE
);

-- Index for user-level queries
CREATE INDEX idx_url_mappings_user_id ON url_mappings(user_id);

-- Index for expiration cleanup
CREATE INDEX idx_url_mappings_expires_at ON url_mappings(expires_at)
    WHERE expires_at IS NOT NULL;

-- Partitioning by hash for horizontal scaling (optional)
-- Consider partitioning on hash of short_code for multi-tenancy

For even higher scale, consider a NoSQL approach with DynamoDB or Cassandra, where the partition key is the short code:

// DynamoDB table design (AWS SDK v3)
// Table: url_mappings
// Primary Key: short_code (String) — partition key
// Attributes: original_url, user_id, created_at, ttl, click_count

const createTableCommand = {
  TableName: 'url_mappings',
  KeySchema: [
    { AttributeName: 'short_code', KeyType: 'HASH' }  // partition key
  ],
  AttributeDefinitions: [
    { AttributeName: 'short_code', AttributeType: 'S' },
    { AttributeName: 'user_id', AttributeType: 'N' }
  ],
  GlobalSecondaryIndexes: [{
    IndexName: 'user-id-index',
    KeySchema: [{ AttributeName: 'user_id', KeyType: 'HASH' }],
    Projection: { ProjectionType: 'KEYS_ONLY' }
  }],
  BillingMode: 'PAY_PER_REQUEST'  // scales automatically with traffic
};

Step 3: Building the REST API

Your API needs two core endpoints: one for creating short URLs, another for redirecting. Here's a production-grade implementation using Node.js and Express:

const express = require('express');
const rateLimit = require('express-rate-limit');
const { Pool } = require('pg');
const redis = require('ioredis');

const app = express();
app.use(express.json());

// PostgreSQL connection pool — tuned for high concurrency
const dbPool = new Pool({
  max: 50,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
});

// Redis for caching and ID generation
const cache = new redis({
  maxRetriesPerRequest: 3,
  enableOfflineQueue: false,
});

// Rate limiter: 100 requests per minute per IP
const createLimiter = rateLimit({
  windowMs: 60 * 1000,
  max: 100,
  standardHeaders: true,
  legacyHeaders: false,
  message: { error: 'Too many requests, slow down.' }
});

// ---------- POST /api/shorten ----------
// Creates a new short URL
app.post('/api/shorten', createLimiter, async (req, res) => {
  try {
    const { url, customCode, ttlDays } = req.body;

    // Validate URL
    if (!url || !isValidUrl(url)) {
      return res.status(400).json({ error: 'Invalid URL provided' });
    }

    // Validate custom code if provided
    let shortCode;
    if (customCode) {
      if (customCode.length > 20 || !/^[a-zA-Z0-9_-]+$/.test(customCode)) {
        return res.status(400).json({ error: 'Custom code must be alphanumeric, max 20 chars' });
      }
      // Check availability
      const exists = await dbPool.query('SELECT short_code FROM url_mappings WHERE short_code = $1', [customCode]);
      if (exists.rows.length > 0) {
        return res.status(409).json({ error: 'Custom code already taken' });
      }
      shortCode = customCode;
    } else {
      // Generate unique code
      const generator = require('./idGenerator');
      shortCode = await generator.generateShortCode();
    }

    // Calculate TTL
    const expiresAt = ttlDays
      ? new Date(Date.now() + ttlDays * 24 * 60 * 60 * 1000).toISOString()
      : null;

    // Insert into database
    await dbPool.query(
      `INSERT INTO url_mappings (short_code, original_url, expires_at, created_at)
       VALUES ($1, $2, $3, NOW())`,
      [shortCode, url, expiresAt]
    );

    // Warm the cache immediately
    await cache.set(`url:${shortCode}`, url, 'EX', ttlDays ? ttlDays * 86400 : 86400 * 30);

    res.status(201).json({
      shortCode,
      shortUrl: `${process.env.BASE_URL}/${shortCode}`,
      originalUrl: url,
      expiresAt,
    });
  } catch (err) {
    console.error('Shorten error:', err);
    res.status(500).json({ error: 'Internal server error' });
  }
});

// ---------- GET /:shortCode ----------
// Redirects to the original URL
app.get('/:shortCode', async (req, res) => {
  try {
    const { shortCode } = req.params;

    // 1. Check cache first — this handles 95%+ of traffic
    const cachedUrl = await cache.get(`url:${shortCode}`);
    if (cachedUrl) {
      // Async update click count — don't block the redirect
      dbPool.query(
        `UPDATE url_mappings SET click_count = click_count + 1, last_accessed_at = NOW()
         WHERE short_code = $1`,
        [shortCode]
      ).catch(() => {}); // fire-and-forget

      return res.redirect(301, cachedUrl);
    }

    // 2. Cache miss — query database
    const result = await dbPool.query(
      `SELECT original_url, expires_at FROM url_mappings WHERE short_code = $1`,
      [shortCode]
    );

    if (result.rows.length === 0) {
      return res.status(404).json({ error: 'Short URL not found' });
    }

    const { original_url, expires_at } = result.rows[0];

    // Check expiration
    if (expires_at && new Date(expires_at) < new Date()) {
      return res.status(410).json({ error: 'This link has expired' });
    }

    // Populate cache with TTL
    const ttlSeconds = expires_at
      ? Math.max(1, Math.floor((new Date(expires_at) - Date.now()) / 1000))
      : 86400 * 30;

    await cache.set(`url:${shortCode}`, original_url, 'EX', ttlSeconds);

    // Async click count update
    dbPool.query(
      `UPDATE url_mappings SET click_count = click_count + 1, last_accessed_at = NOW()
       WHERE short_code = $1`,
      [shortCode]
    ).catch(() => {});

    return res.redirect(301, original_url);
  } catch (err) {
    console.error('Redirect error:', err);
    res.status(500).json({ error: 'Internal server error' });
  }
});

// URL validation helper
function isValidUrl(str) {
  try {
    const url = new URL(str);
    return url.protocol === 'http:' || url.protocol === 'https:';
  } catch {
    return false;
  }
}

app.listen(3000, () => console.log('URL shortener running on port 3000'));

Step 4: Caching Strategy for Massive Read Throughput

For 1 million users, reads (redirects) will outnumber writes 10:1 or more. A robust caching layer is essential. Here's the strategy:

• Cache all redirects in Redis with appropriate TTLs
• Use write-through caching — populate cache when creating URLs
• Set TTL based on expiration — if a URL expires in 3 days, cache it for exactly 3 days
• Implement cache-aside with fallback — on cache miss, fetch from DB and populate cache

// Advanced cache configuration with Redis Cluster (for 1M+ users)
const RedisCluster = require('ioredis').Cluster;

const cacheCluster = new RedisCluster([
  { host: 'redis-node-1', port: 6379 },
  { host: 'redis-node-2', port: 6379 },
  { host: 'redis-node-3', port: 6379 },
  { host: 'redis-node-4', port: 6379 },
  { host: 'redis-node-5', port: 6379 },
  { host: 'redis-node-6', port: 6379 },
], {
  scaleReads: 'slave',           // distribute reads across replicas
  retryDelayOnFailover: 100,
  maxRedirections: 16,
});

// Cache key design
// Pattern:  url:{shortCode}  →  original URL
// TTL sync: if DB row has expires_at, cache TTL = expires_at - now
// Fallback:  cache miss → DB query → populate cache → return

async function getUrlWithCache(shortCode) {
  const cacheKey = `url:${shortCode}`;

  // Try cache
  const cached = await cacheCluster.get(cacheKey);
  if (cached) return cached;

  // Cache miss — acquire distributed lock to prevent thundering herd
  const lockKey = `lock:url:${shortCode}`;
  const locked = await cacheCluster.set(lockKey, '1', 'NX', 'PX', 500);

  if (locked) {
    // This instance is responsible for fetching from DB
    const result = await dbPool.query(
      'SELECT original_url, expires_at FROM url_mappings WHERE short_code = $1',
      [shortCode]
    );
    if (result.rows.length > 0) {
      const { original_url, expires_at } = result.rows[0];
      const ttl = calculateTTL(expires_at);
      await cacheCluster.set(cacheKey, original_url, 'PX', ttl);
      await cacheCluster.del(lockKey);
      return original_url;
    }
    await cacheCluster.del(lockKey);
    return null;
  }

  // Wait and retry cache (another instance is fetching)
  await new Promise(resolve => setTimeout(resolve, 100));
  return await cacheCluster.get(cacheKey) || getUrlWithCache(shortCode);
}

function calculateTTL(expiresAt) {
  if (!expiresAt) return 30 * 24 * 3600 * 1000; // 30 days default
  const remaining = new Date(expiresAt).getTime() - Date.now();
  return Math.max(1000, remaining); // minimum 1 second
}

Step 5: Handling High-Concurrency Edge Cases

Custom Code Race Conditions

When users request custom short codes, two simultaneous requests for the same code could both pass the availability check before either inserts. Use database unique constraints as the ultimate guard:

// Safe custom code insertion with ON CONFLICT handling
async function createWithCustomCode(code, url) {
  try {
    const result = await dbPool.query(
      `INSERT INTO url_mappings (short_code, original_url, created_at)
       VALUES ($1, $2, NOW())
       ON CONFLICT (short_code) DO NOTHING
       RETURNING short_code`,
      [code, url]
    );
    if (result.rows.length === 0) {
      throw new Error('Code already taken — retry with different code');
    }
    return code;
  } catch (err) {
    // Let the user know to pick another custom code
    throw err;
  }
}

Preventing Malicious URL Submission

Users could submit URLs pointing to phishing sites, malware, or internal services. Implement URL validation and filtering:

// URL safety checks before shortening
const BLOCKED_DOMAINS = new Set(['localhost', '127.0.0.1', '0.0.0.0', 'internal-api']);
const BLOCKED_PROTOCOLS = new Set(['file:', 'ftp:', 'javascript:', 'data:']);

function validateAndSanitizeUrl(rawUrl) {
  let parsed;
  try {
    parsed = new URL(rawUrl);
  } catch {
    throw new Error('Malformed URL');
  }

  // Block dangerous protocols
  if (BLOCKED_PROTOCOLS.has(parsed.protocol)) {
    throw new Error(`Protocol ${parsed.protocol} is not allowed`);
  }

  // Block internal/private hosts
  const hostname = parsed.hostname.toLowerCase();
  if (BLOCKED_DOMAINS.has(hostname) || hostname.startsWith('192.168.') ||
      hostname.startsWith('10.') || hostname === '[::1]') {
    throw new Error('Internal URLs are not permitted');
  }

  // Enforce HTTPS for production (optional but recommended)
  if (process.env.NODE_ENV === 'production' && parsed.protocol !== 'https:') {
    throw new Error('Only HTTPS URLs are accepted in production');
  }

  return parsed.href; // normalized URL
}

Step 6: Analytics and Click Tracking

For 1 million users, you need analytics that don't slow down redirects. Use asynchronous, fire-and-forget updates:

// Asynchronous click tracking — never blocks the redirect
async function trackClickAsync(shortCode, metadata) {
  const clickData = {
    short_code: shortCode,
    timestamp: new Date().toISOString(),
    user_agent: metadata.userAgent?.substring(0, 200),
    ip_hash: hashIP(metadata.ip),  // privacy-conscious: hash, don't store raw IP
    referrer: metadata.referrer?.substring(0, 500),
    country: metadata.geoCountry,
  };

  // Push to a message queue for batch processing
  // This avoids slowing down the redirect response
  await messageQueue.push('click-events', clickData);

  // Also increment a Redis counter for real-time dashboard
  await cache.hincrby(`stats:${shortCode}`, 'clicks', 1);
  await cache.expire(`stats:${shortCode}`, 86400 * 30);
}

// Batch processor (runs separately, e.g., via worker process)
async function batchProcessClicks() {
  const batch = await messageQueue.popBatch('click-events', 100);
  if (batch.length === 0) return;

  const values = batch.map((e, i) =>
    `($${i * 6 + 1}, $${i * 6 + 2}, $${i * 6 + 3}, $${i * 6 + 4}, $${i * 6 + 5}, $${i * 6 + 6})`
  ).join(', ');

  const params = batch.flatMap(e => [
    e.short_code, e.timestamp, e.user_agent, e.ip_hash, e.referrer, e.country
  ]);

  await dbPool.query(
    `INSERT INTO click_events (short_code, timestamp, user_agent, ip_hash, referrer, country)
     VALUES ${values}`,
    params
  );
}

Step 7: Deployment and Scaling Considerations

When deploying for 1 million users, consider these infrastructure patterns:

• Load Balancer — distribute traffic across 4–8 API instances
• Database Read Replicas — redirect lookups hit replicas, writes go to primary
• Redis Cluster — shard cache across multiple nodes with replication
• CDN for Static Assets — offload landing pages and UI assets
• Monitoring — track p95/p99 latency, cache hit ratio, DB connection pool saturation

// Environment-based configuration for different scale tiers
const config = {
  // Small scale (< 10K users) — single server
  small: {
    dbPoolMax: 20,
    cacheInstance: 'single',
    apiInstances: 1,
    rateLimitPerMinute: 30,
  },
  // Medium scale (< 500K users) — multiple servers, read replicas
  medium: {
    dbPoolMax: 50,
    dbReadReplicas: ['replica1.internal', 'replica2.internal'],
    cacheCluster: true,
    apiInstances: 4,
    rateLimitPerMinute: 100,
  },
  // Large scale (1M+ users) — full distributed deployment
  large: {
    dbPoolMax: 100,
    dbReadReplicas: 8,
    cacheClusterNodes: 12,
    apiInstances: 16,
    rateLimitPerMinute: 200,
    cdnEnabled: true,
    geoDistributed: true,  // deploy to multiple regions
  },
};

function loadConfig() {
  const tier = process.env.SCALE_TIER || 'medium';
  return config[tier];
}

Best Practices Summary

• Use Base62 encoding with a distributed counter — avoids collisions, produces short codes, and scales horizontally
• Cache aggressively — aim for 95%+ cache hit rate on redirects to keep database load manageable
• Implement distributed locking on cache misses — prevents thundering herd when a popular link's cache entry expires
• Validate and sanitize URLs — block internal hosts, dangerous protocols, and excessively long URLs
• Use 301 (permanent) redirects — browsers cache them, reducing future requests to your service
• Rate limit creation endpoints — prevent abuse while keeping redirects fast and unrestricted
• Never block redirects for analytics — use fire-and-forget patterns and message queues for click tracking
• Partition your database early — plan for sharding by short code hash or user ID ranges
• Monitor cache hit ratio religiously — if it drops below 90%, investigate immediately
• Set sensible TTLs on both cache and database rows — auto-expire unused links to control storage growth

Conclusion

Designing a URL shortener for 1 million users is a classic system design challenge that touches on hashing strategies, distributed ID generation, caching architectures, and database scaling patterns. The key insight is that reads dominate writes by an order of magnitude — your architecture should optimize for sub-millisecond redirects while keeping URL creation reliable and collision-free. By combining Base62-encoded distributed counters, a Redis cluster for caching, PostgreSQL (or DynamoDB) with read replicas, and asynchronous analytics processing, you can build a system that handles millions of URLs and hundreds of millions of redirects per month with confidence. Start with the counter-based approach, implement write-through caching from day one, validate every URL rigorously, and always keep the redirect path as lean as possible. With these patterns in place, scaling to 1 million users becomes an exercise in incremental infrastructure growth rather than a fundamental redesign.