What is AI Agent Uptime?
AI agent uptime refers to the continuous operational availability of autonomous AI agents — software programs that use LLMs, tool calling, and reasoning loops to perform tasks without human intervention. Unlike traditional microservices that sit behind a load balancer and respond to HTTP requests, AI agents often run as long-lived processes, stateful workflows, or scheduled jobs that can span minutes, hours, or even days to complete a single task.
Uptime for an AI agent means the agent is:
- Running — its process hasn't crashed or been killed
- Reachable — if it exposes an API, it responds within a reasonable timeout
- Functional — the underlying LLM provider is accessible, API keys are valid, and tool integrations are working
- Not stalled — it isn't stuck in an infinite reasoning loop or waiting on a deadlocked external resource
In teams without dedicated DevOps or SRE personnel, maintaining agent uptime falls squarely on the developers who built the agent. This tutorial teaches you how to keep agents alive using lightweight, developer-friendly patterns that don't require Kubernetes, Prometheus, PagerDuty, or an on-call rotation.
Why Agent Uptime Matters
A down agent costs more than a down microservice for three reasons:
- State loss — Many agents maintain in-memory conversation context, tool results, or partial reasoning traces. A crash wipes all of that. Restarting means starting over, which wastes tokens and time.
- User trust — Agents often power chat interfaces, copilots, or automated workflows. A 5-minute outage during a user's session feels catastrophic because the agent disappears mid-thought rather than returning a clean 500 error.
- Cost amplification — Restarting an agent from scratch re-executes expensive LLM calls. If your agent runs a $0.50 reasoning chain and crashes 3 times a day, that's $1.50/day in wasted compute, or $45/month, just from instability.
Without a DevOps team, you can't rely on infrastructure-level resilience. You need application-level survivability baked into the agent itself.
Common Failure Modes for AI Agents
Before we build defenses, let's catalog what kills agents in production:
- LLM provider outages — OpenAI, Anthropic, or your chosen provider goes down or returns 5xx errors
- Rate limiting — You hit RPM/TPM limits and the agent throws an unhandled exception
- Token overflows — The conversation context grows beyond the model's context window, causing API errors
- Tool timeout — A tool call to an external API hangs and the agent waits forever
- Infinite reasoning loops — The agent keeps calling the same tool with the same arguments, never reaching a terminal state
- Invalid JSON parsing — The LLM returns malformed JSON, the parser crashes, and the agent exits
- Memory exhaustion — Long-running agents accumulate context until the process runs out of RAM
- Process death — The host machine restarts, the terminal session ends, or a container hits its memory limit
Each of these is recoverable if you plan for it. The key insight: your agent must treat itself as a system it is responsible for keeping alive.
Pattern 1: The Self-Healing Supervisor Loop
The most important pattern is wrapping your agent's core logic in a supervisor that catches failures, logs them, and decides whether to retry, reset, or escalate. Here's a minimal implementation in Python:
import time
import logging
import traceback
from typing import Callable, Optional
from dataclasses import dataclass, field
from enum import Enum
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("agent_supervisor")
class AgentState(Enum):
HEALTHY = "healthy"
DEGRADED = "degraded"
RECOVERING = "recovering"
FATAL = "fatal"
@dataclass
class SupervisorConfig:
max_retries: int = 3
base_delay_seconds: float = 1.0
max_delay_seconds: float = 30.0
reset_on_memory_error: bool = True
circuit_breaker_threshold: int = 5
@dataclass
class AgentHeartbeat:
state: AgentState = AgentState.HEALTHY
consecutive_failures: int = 0
last_success_timestamp: float = field(default_factory=time.time)
error_log: list = field(default_factory=list)
class AgentSupervisor:
"""Wraps an AI agent function with self-healing capabilities."""
def __init__(self, config: Optional[SupervisorConfig] = None):
self.config = config or SupervisorConfig()
self.heartbeat = AgentHeartbeat()
def run(self, agent_fn: Callable, *args, **kwargs):
"""Execute agent_fn with exponential backoff and state management."""
attempt = 0
delay = self.config.base_delay_seconds
while attempt <= self.config.max_retries:
try:
result = agent_fn(*args, **kwargs)
self.heartbeat.consecutive_failures = 0
self.heartbeat.last_success_timestamp = time.time()
self.heartbeat.state = AgentState.HEALTHY
return result
except Exception as e:
attempt += 1
self.heartbeat.consecutive_failures += 1
self.heartbeat.error_log.append({
"attempt": attempt,
"error": str(e),
"timestamp": time.time()
})
logger.warning(f"Agent failure (attempt {attempt}/{self.config.max_retries}): {e}")
if self.heartbeat.consecutive_failures >= self.config.circuit_breaker_threshold:
self.heartbeat.state = AgentState.FATAL
logger.critical("Circuit breaker triggered — manual intervention required")
raise RuntimeError("Agent entered FATAL state after repeated failures") from e
self.heartbeat.state = AgentState.RECOVERING
time.sleep(min(delay, self.config.max_delay_seconds))
delay *= 2 # exponential backoff
raise RuntimeError(f"Agent failed after {self.config.max_retries} retries")Use it by wrapping your agent's main function:
def my_agent(task: str) -> str:
# Your actual agent logic here
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "user", "content": task}]
)
return response.choices[0].message.content
supervisor = AgentSupervisor()
try:
result = supervisor.run(my_agent, "Analyze this dataset")
print(result)
except RuntimeError:
# Log, notify, or fall back to a simpler path
print("Agent unreachable — using cached response")
Pattern 2: LLM Provider Circuit Breaker
When your LLM provider has an outage, rapid retries make things worse. A circuit breaker stops calling the provider after N consecutive failures and falls back to a degraded mode. Here's a production-ready implementation:
import threading
import time
from enum import Enum
from functools import wraps
class CircuitState(Enum):
CLOSED = "closed" # normal operation
OPEN = "open" # failing, no calls allowed
HALF_OPEN = "half_open" # testing if provider recovered
class LLMCircuitBreaker:
def __init__(self, failure_threshold=3, recovery_timeout=60,
fallback_provider=None):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout # seconds before trying again
self.fallback_provider = fallback_provider # e.g., "anthropic" if primary is "openai"
self.state = CircuitState.CLOSED
self.failure_count = 0
self.last_failure_time = 0
self._lock = threading.Lock()
def call(self, primary_fn, fallback_fn=None, *args, **kwargs):
"""Execute primary_fn with circuit breaker protection."""
with self._lock:
if self.state == CircuitState.OPEN:
if time.time() - self.last_failure_time > self.recovery_timeout:
self.state = CircuitState.HALF_OPEN
print("Circuit breaker: attempting half-open test")
else:
if fallback_fn:
print("Circuit breaker OPEN — using fallback")
return fallback_fn(*args, **kwargs)
raise Exception("Circuit breaker is OPEN — no calls allowed")
try:
result = primary_fn(*args, **kwargs)
with self._lock:
if self.state == CircuitState.HALF_OPEN:
self.state = CircuitState.CLOSED
self.failure_count = 0
print("Circuit breaker: provider recovered, closing circuit")
return result
except Exception as e:
with self._lock:
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = CircuitState.OPEN
print(f"Circuit breaker OPEN after {self.failure_count} failures")
raise e
# Usage with multiple providers
breaker = LLMCircuitBreaker(failure_threshold=3, recovery_timeout=120)
def call_openai(prompt):
return openai.ChatCompletion.create(model="gpt-4", messages=[{"role":"user","content":prompt}])
def call_anthropic(prompt):
return anthropic.messages.create(model="claude-3-opus", messages=[{"role":"user","content":prompt}])
response = breaker.call(
primary_fn=call_openai,
fallback_fn=call_anthropic,
prompt="Summarize this document"
)
Pattern 3: Stuck-Agent Detection and Kill Switch
The silent killer of agent uptime is the infinite reasoning loop. The agent isn't crashed — it's running happily, burning tokens, calling tools, and going nowhere. You need a watchdog that detects repetitive behavior and forces a reset.
import hashlib
import json
from collections import deque
from datetime import datetime, timedelta
class LoopDetector:
"""Detects when an agent is stuck repeating the same actions."""
def __init__(self, window_size=20, similarity_threshold=0.8,
max_identical_actions=5, max_agent_runtime_minutes=30):
self.action_history = deque(maxlen=window_size)
self.similarity_threshold = similarity_threshold
self.max_identical_actions = max_identical_actions
self.max_agent_runtime = timedelta(minutes=max_agent_runtime_minutes)
self.agent_start_time = datetime.now()
self.identical_count = 0
self.last_action_hash = None
def record_action(self, action: dict) -> bool:
"""Record an agent action. Returns True if a loop is detected."""
current_time = datetime.now()
# Check total runtime
if current_time - self.agent_start_time > self.max_agent_runtime:
print(f"WARNING: Agent exceeded max runtime of {self.max_agent_runtime}")
return True
action_str = json.dumps(action, sort_keys=True)
action_hash = hashlib.md5(action_str.encode()).hexdigest()
if action_hash == self.last_action_hash:
self.identical_count += 1
if self.identical_count >= self.max_identical_actions:
print(f"CRITICAL: Agent repeated exact same action {self.identical_count} times — likely stuck")
return True
else:
self.identical_count = 0
self.last_action_hash = action_hash
self.action_history.append({
"hash": action_hash,
"timestamp": current_time.isoformat(),
"tool": action.get("tool_name", "unknown")
})
# Check for cyclic patterns in recent history
recent_hashes = [a["hash"] for a in self.action_history]
if len(recent_hashes) >= 6:
# Look for repeating patterns of length 2-4
for pattern_len in [2, 3, 4]:
if self._detect_cycle(recent_hashes, pattern_len):
print(f"WARNING: Detected repeating cycle of length {pattern_len}")
return True
return False
def _detect_cycle(self, hashes, pattern_len):
if len(hashes) < pattern_len * 2:
return False
recent_pattern = hashes[-pattern_len:]
previous_window = hashes[-pattern_len*2:-pattern_len]
return recent_pattern == previous_window
def reset(self):
self.agent_start_time = datetime.now()
self.action_history.clear()
self.identical_count = 0
self.last_action_hash = None
# Integrate into your agent loop
loop_detector = LoopDetector()
while not task_complete:
action = agent.decide_next_action(context)
if loop_detector.record_action({
"tool_name": action.tool,
"arguments": action.args
}):
print("Loop detected — injecting circuit breaker prompt")
context.append({
"role": "system",
"content": "You appear to be repeating yourself. Please reconsider your approach. If you cannot make progress, state your best answer so far and stop."
})
loop_detector.reset()
continue
result = agent.execute_action(action)
context.append(result)
Pattern 4: State Checkpointing for Crash Recovery
Long-running agents lose everything on crash. Checkpointing saves intermediate state so the agent can resume from its last good state rather than starting over. This is especially important for agents that cost $0.50-$2.00 per run in LLM tokens.
import pickle
import os
import hashlib
from datetime import datetime
class CheckpointManager:
"""Persists agent state to disk for crash recovery."""
def __init__(self, checkpoint_dir="./agent_checkpoints",
max_checkpoints=5, checkpoint_interval_seconds=60):
self.checkpoint_dir = checkpoint_dir
self.max_checkpoints = max_checkpoints
self.checkpoint_interval = checkpoint_interval_seconds
self.last_checkpoint_time = 0
os.makedirs(checkpoint_dir, exist_ok=True)
def should_checkpoint(self) -> bool:
return time.time() - self.last_checkpoint_time >= self.checkpoint_interval
def save(self, agent_state: dict) -> str:
"""Save agent state. Returns checkpoint ID."""
timestamp = datetime.now().isoformat()
state_hash = hashlib.md5(str(sorted(agent_state.keys())).encode()).hexdigest()[:8]
checkpoint_id = f"{timestamp}_{state_hash}"
filepath = os.path.join(self.checkpoint_dir, f"{checkpoint_id}.pkl")
checkpoint_data = {
"timestamp": timestamp,
"state": agent_state,
"checkpoint_id": checkpoint_id
}
with open(filepath, "wb") as f:
pickle.dump(checkpoint_data, f)
self.last_checkpoint_time = time.time()
self._prune_old_checkpoints()
return checkpoint_id
def load_latest(self) -> dict:
"""Load the most recent checkpoint."""
checkpoints = self._list_checkpoints()
if not checkpoints:
return None
latest = checkpoints[-1]
with open(latest, "rb") as f:
data = pickle.load(f)
print(f"Resuming from checkpoint {data['checkpoint_id']}")
return data["state"]
def _list_checkpoints(self):
files = glob.glob(os.path.join(self.checkpoint_dir, "*.pkl"))
return sorted(files)
def _prune_old_checkpoints(self):
files = self._list_checkpoints()
while len(files) > self.max_checkpoints:
os.remove(files.pop(0))
# Usage in agent loop
checkpoint_mgr = CheckpointManager()
agent_state = checkpoint_mgr.load_latest()
if agent_state:
context = agent_state["conversation_context"]
completed_steps = agent_state["completed_steps"]
else:
context = [{"role": "system", "content": "You are a helpful assistant."}]
completed_steps = []
try:
for step in range(max_steps):
# ... agent logic ...
completed_steps.append(step)
if checkpoint_mgr.should_checkpoint():
checkpoint_mgr.save({
"conversation_context": context,
"completed_steps": completed_steps,
"current_step": step
})
except Exception as e:
# Emergency checkpoint before crash
checkpoint_mgr.save({
"conversation_context": context,
"completed_steps": completed_steps,
"crash_error": str(e)
})
raise
Pattern 5: Process-Level Auto-Restart with systemd
Even the best in-process recovery can't help if the process itself dies. For Linux servers and VPS instances, systemd provides free auto-restart without needing Docker, Kubernetes, or a process manager like PM2. Here's a complete systemd service file for an AI agent:
# /etc/systemd/system/ai-agent.service
[Unit]
Description=AI Agent Runner - Autonomous Task Agent
After=network.target network-online.target
Wants=network-online.target
[Service]
Type=simple
User=agentuser
Group=agentuser
WorkingDirectory=/opt/ai-agent
EnvironmentFile=/opt/ai-agent/.env
ExecStart=/opt/ai-agent/venv/bin/python /opt/ai-agent/agent_runner.py
# Critical: auto-restart configuration
Restart=always
RestartSec=10
StartLimitInterval=300
StartLimitBurst=5
# Prevent infinite restart storms
# After 5 restarts in 5 minutes, systemd stops trying
# Resource limits
MemoryMax=2G
CPUQuota=200%
TasksMax=100
# Graceful shutdown handling
TimeoutStopSec=30
KillSignal=SIGTERM
SendSIGKILL=no
# Logging
StandardOutput=append:/var/log/ai-agent/agent.log
StandardError=append:/var/log/ai-agent/agent_error.log
[Install]
WantedBy=multi-user.target
Enable and start the service:
sudo systemctl daemon-reload
sudo systemctl enable ai-agent.service
sudo systemctl start ai-agent.service
# Check status and recent restarts
sudo systemctl status ai-agent.service
journalctl -u ai-agent.service --since "10 minutes ago" -f
For the agent code itself, handle SIGTERM gracefully so checkpointing happens before shutdown:
import signal
import sys
checkpoint_mgr = CheckpointManager()
shutdown_requested = False
def graceful_shutdown(signum, frame):
global shutdown_requested
print(f"Received signal {signum} — checkpointing and shutting down")
shutdown_requested = True
signal.signal(signal.SIGTERM, graceful_shutdown)
signal.signal(signal.SIGINT, graceful_shutdown)
# In your agent loop
while not task_complete and not shutdown_requested:
# ... agent step ...
if checkpoint_mgr.should_checkpoint():
checkpoint_mgr.save(agent_state)
if shutdown_requested:
checkpoint_mgr.save(agent_state)
print("Graceful shutdown complete")
sys.exit(0)
Health Check Endpoint for External Monitoring
Even without a DevOps team, you need to know if your agent is alive. A lightweight HTTP health check endpoint lets you use free uptime monitors like UptimeRobot, BetterStack, or Healthchecks.io to get alerts via email, Slack, or SMS when the agent is down.
from http.server import HTTPServer, BaseHTTPRequestHandler
import threading
import json
class AgentHealthHandler(BaseHTTPRequestHandler):
def __init__(self, *args, supervisor=None, heartbeat=None, **kwargs):
self.supervisor = supervisor
self.heartbeat = heartbeat
super().__init__(*args, **kwargs)
def do_GET(self):
if self.path == "/health":
status_code = 200
response = {
"status": self.heartbeat.state.value,
"uptime_seconds": time.time() - start_time,
"consecutive_failures": self.heartbeat.consecutive_failures,
"last_success": datetime.fromtimestamp(
self.heartbeat.last_success_timestamp
).isoformat()
}
if self.heartbeat.state == AgentState.FATAL:
status_code = 503
elif self.heartbeat.state == AgentState.RECOVERING:
status_code = 200 # still alive, just degraded
self.send_response(status_code)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(json.dumps(response).encode())
elif self.path == "/ping":
self.send_response(200)
self.send_header("Content-Type", "text/plain")
self.end_headers()
self.wfile.write(b"pong")
else:
self.send_response(404)
self.end_headers()
def start_health_server(supervisor, heartbeat, port=8080):
handler = lambda *args: AgentHealthHandler(*args,
supervisor=supervisor, heartbeat=heartbeat)
server = HTTPServer(("0.0.0.0", port), AgentHealthHandler)
server.supervisor = supervisor
server.heartbeat = heartbeat
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
print(f"Health check server on port {port}")
return server
# Start alongside your agent
start_time = time.time()
health_server = start_health_server(supervisor, supervisor.heartbeat, port=8080)
Configure UptimeRobot to hit http://your-server:8080/health every 60 seconds. If it gets a non-2xx response or a timeout, you'll get an alert. No DevOps required — this takes 3 minutes to set up.
Complete Integration: Putting It All Together
Here's how all the patterns combine into a single production-ready agent runner:
import time
import signal
import sys
from typing import Optional
class ProductionAgent:
"""A complete, self-healing AI agent that survives real-world failures."""
def __init__(self,
primary_llm_fn,
fallback_llm_fn=None,
checkpoint_dir="./checkpoints",
health_port=8080):
self.supervisor = AgentSupervisor(SupervisorConfig(
max_retries=3,
circuit_breaker_threshold=5
))
self.circuit_breaker = LLMCircuitBreaker(
failure_threshold=3,
recovery_timeout=120,
fallback_provider="claude"
)
self.loop_detector = LoopDetector()
self.checkpoint_mgr = CheckpointManager(
checkpoint_dir=checkpoint_dir,
checkpoint_interval_seconds=60
)
self.primary_llm_fn = primary_llm_fn
self.fallback_llm_fn = fallback_llm_fn
# Start health server
self.health_server = start_health_server(
self.supervisor, self.supervisor.heartbeat, port=health_port
)
# Signal handling
self.shutdown_requested = False
signal.signal(signal.SIGTERM, self._handle_shutdown)
signal.signal(signal.SIGINT, self._handle_shutdown)
def _handle_shutdown(self, signum, frame):
self.shutdown_requested = True
def run_task(self, task: str) -> str:
"""Execute a task with full survivability stack."""
# Try to resume from checkpoint
saved_state = self.checkpoint_mgr.load_latest()
if saved_state:
context = saved_state.get("conversation_context", [])
else:
context = [{"role": "system", "content": "You are a capable AI assistant."},
{"role": "user", "content": task}]
def agent_step(context):
"""Single step of the agent — wrapped by circuit breaker."""
def call_llm():
return self.primary_llm_fn(context)
response = self.circuit_breaker.call(
primary_fn=call_llm,
fallback_fn=lambda: self.fallback_llm_fn(context)
if self.fallback_llm_fn else None
)
return response
try:
return self.supervisor.run(self._agent_loop, context)
except RuntimeError:
# Final fallback — return whatever we have
if saved_state:
return saved_state.get("partial_result",
"Agent failed to complete task")
return "Agent unavailable — please try again later"
def _agent_loop(self, context):
"""The actual agent reasoning loop."""
max_steps = 50
for step in range(max_steps):
if self.shutdown_requested:
self.checkpoint_mgr.save({
"conversation_context": context,
"partial_result": "Shutdown mid-execution",
"completed_steps": step
})
raise SystemExit("Graceful shutdown")
response = self._call_llm_with_protection(context)
context.append({"role": "assistant", "content": response.content})
# Check for loops
if self.loop_detector.record_action({
"tool_name": getattr(response, 'tool_calls', None),
"content_hash": hash(response.content)
}):
context.append({
"role": "system",
"content": "You are repeating yourself. Please change approach or conclude."
})
self.loop_detector.reset()
# Checkpoint periodically
if self.checkpoint_mgr.should_checkpoint():
self.checkpoint_mgr.save({
"conversation_context": context,
"completed_steps": step,
"timestamp": time.time()
})
# Check for terminal condition
if response.tool_calls is None:
return response.content
return context[-1].get("content", "Agent reached step limit")
# Deploy with:
agent = ProductionAgent(
primary_llm_fn=lambda ctx: openai.ChatCompletion.create(
model="gpt-4", messages=ctx
).choices[0].message,
fallback_llm_fn=lambda ctx: anthropic.messages.create(
model="claude-3-haiku", messages=ctx
).content[0].text
)
Monitoring Without a DevOps Team
You need visibility into agent health without Grafana dashboards or Prometheus exporters. Here are three lightweight approaches:
Approach 1: Structured JSON Logging
import json
from datetime import datetime
class AgentLogger:
def __init__(self, log_path="/var/log/ai-agent/structured.log"):
self.log_path = log_path
def log(self, event_type: str, data: dict):
entry = {
"timestamp": datetime.now().isoformat(),
"event": event_type,
**data
}
with open(self.log_path, "a") as f:
f.write(json.dumps(entry) + "\n")
# Usage
logger = AgentLogger()
logger.log("agent_start", {"task_id": "abc123", "model": "gpt-4"})
logger.log("tool_call", {"tool": "search", "duration_ms": 234})
logger.log("checkpoint_saved", {"checkpoint_id": "xyz", "step": 12})
logger.log("circuit_breaker_open", {"provider": "openai", "downtime_seconds": 120})
# Analyze with: cat structured.log | jq '.event' | sort | uniq -c
Approach 2: Healthchecks.io Ping (Dead Man's Switch)
import requests
import threading
class DeadMansSwitch:
"""Pings Healthchecks.io on a schedule. If pings stop, you get alerted."""
def __init__(self, ping_url, interval_seconds=60):
self.ping_url = ping_url
self.interval = interval_seconds
self.running = True
self.thread = threading.Thread(target=self._ping_loop, daemon=True)
self.thread.start()
def _ping_loop(self):
while self.running:
try:
requests.get(self.ping_url, timeout=10)
except Exception as e:
print(f"Failed to ping healthcheck: {e}")
time.sleep(self.interval)
def stop(self):
self.running = False
# Create a "dead man's switch" on healthchecks.io (free)
# Use the URL they give you:
deadman = DeadMansSwitch("https://hc-ping.com/your-uuid-here", interval_seconds=30)
Approach 3: Telegram / Discord Alert on Critical Failure
def send_alert(message: str, webhook_url: str):
"""Send a notification to Discord or Slack webhook."""
payload = {
"content": f"🚨 **Agent Alert**\n{message}",
"username": "Agent Monitor"
}
try:
requests.post(webhook_url, json=payload, timeout=5)
except:
print(f"Failed to send alert: {message}")
# Wire into supervisor's FATAL state
if supervisor.heartbeat.state == AgentState.FATAL:
send_alert(
f"Agent entered FATAL state after {supervisor.heartbeat.consecutive_failures} failures. "
f"Last error: {supervisor.heartbeat.error_log[-1]['error']}",
"https://discord.com/api/webhooks/your-webhook-url"
)
Best Practices Summary
- Never trust a single LLM provider — Always have a fallback provider configured. The cost of maintaining an API key for a backup provider is zero; the cost of being down for 2 hours is much higher.
- Checkpoint before expensive operations — Save state before making a $0.10+ LLM call. If the call fails, you resume right before it, not from the beginning.
- Set a maximum agent runtime — Every agent loop should have a hard time limit. A stuck agent burning tokens at 2 AM is worse than a crashed agent.
- Use exponential backoff on transient errors — Rate limits and temporary provider outages resolve in seconds. Don't hammer the API; back off gracefully.
- Log everything in structured JSON — When something goes wrong at 3 AM, you need to answer "what was the agent doing?" from a single log file, not by reconstructing state from scattered print statements.
- Test failure modes deliberately — Kill the process mid-execution. Pull the API key. Exhaust the context window. Your agent should survive all of these gracefully, not just in theory.
- Keep health checks simple — A /ping endpoint that returns 200 is infinitely better than a complex health check you never set up. Start with the simplest thing that works.
- systemd over Docker for simple deployments — If you're on a single VPS, systemd gives you auto-restart, logging, and resource limits without the complexity of container orchestration.
Conclusion
Keeping AI agents alive without a DevOps team is entirely achievable — it just requires building survivability into the agent from day one, rather than relying on infrastructure that you don't have. The patterns in this tutorial — supervisor loops, circuit breakers, loop detection, checkpointing, systemd auto-restart, and lightweight health monitoring — form a complete survivability stack that any solo developer can implement in an afternoon.
The key mindset shift is treating your agent as a system that must manage its own availability. When your agent can detect that it's stuck, fall back to a different provider when the primary is down, resume from a checkpoint after a crash, and alert you when things go truly wrong, you've built something resilient enough for production — without needing a single page from an on-call engineer.
Start with the supervisor loop and health check endpoint — those two alone will catch 80% of failure scenarios. Add checkpointing next if your agent runs tasks longer than a few minutes. The circuit breaker and loop detector complete the picture for long-running, high-reliability agents. Each piece is independent, testable, and incrementally deployable, so you can adopt them at your own pace without a big upfront migration.