What is AI Agent Versioning?
AI Agent Versioning is the systematic practice of tracking, managing, and deploying changes to the three core components that define an AI agent's behavior: prompts (system instructions and templates), models (the underlying LLM or reasoning engine), and configs (hyperparameters, tool integrations, memory settings, and runtime constraints). Think of it as infrastructure-as-code meets prompt engineering — every change is intentional, auditable, and reversible.
Unlike traditional software versioning, which primarily tracks source code, AI agent versioning must account for the non-deterministic nature of LLM outputs. A subtle prompt tweak or a model upgrade can dramatically shift agent behavior in ways that unit tests cannot easily capture. Proper versioning gives you the control plane to experiment safely, roll back confidently, and understand exactly what configuration produced a given agent output.
The Three Pillars: Prompts, Models, and Configs
Every AI agent, regardless of framework, boils down to three versionable layers:
- Prompts — System messages, few-shot examples, tool descriptions, output format instructions, and chain-of-thought scaffolding. Even a single word change can alter reasoning paths.
- Models — The specific model identifier (e.g.,
gpt-4o-2024-08-06,claude-3-5-sonnet-20241022), including provider, version date, and quantization level for local models. - Configs — Temperature, top_p, max_tokens, tool schemas, memory backends, rate limits, retry logic, and environment-specific overrides (staging vs. production).
These three layers are tightly coupled. A prompt optimized for gpt-4-turbo may fail on gpt-4o-mini. A temperature of 0.7 that works beautifully in development may produce gibberish in production with a different model version. Versioning them together as an immutable snapshot is the only reliable way to deploy AI agents.
Why AI Agent Versioning Matters
Without versioning, AI agent development quickly becomes chaos. You've likely experienced this: an agent that worked perfectly last week suddenly produces wrong outputs. Was the prompt changed? Did someone swap the model? Was the temperature bumped? Without a version history, you're reduced to guesswork and git archaeology across multiple files.
Here are the concrete problems that agent versioning solves:
- Reproducibility — Recreate the exact agent state that produced a specific output, critical for debugging customer complaints or regulatory audits.
- Safe rollbacks — If a new prompt/model combination degrades performance, revert to the previous known-good version in seconds, not hours.
- A/B testing — Run multiple agent versions in parallel, collect metrics, and promote the winner with confidence.
- Compliance & governance — Prove exactly which model processed which data, when, and with what instructions — increasingly required in regulated industries.
- Team collaboration — Prompt engineers, ML engineers, and product managers can iterate independently on their layer without stepping on each other's toes.
- Cost tracking — Tie cost-per-call metrics to specific model versions and configs to understand the financial impact of each change.
Building a Versioning System: A Practical Implementation
Let's build a complete agent versioning system in Python. We'll create a registry that snapshots prompts, models, and configs together, supports semantic versioning, and integrates with any LLM provider.
Step 1: Define the Agent Version Schema
First, we need a structured way to represent an agent version. We'll use a combination of a YAML manifest and a Python dataclass for runtime validation.
# agent_version_schema.py
from dataclasses import dataclass, field
from typing import Optional, Dict, List
from datetime import datetime
import hashlib
import json
@dataclass
class PromptVersion:
"""Immutable snapshot of a prompt template and its metadata."""
system_template: str
user_template: str
few_shot_examples: List[Dict[str, str]] = field(default_factory=list)
tool_descriptions: List[Dict] = field(default_factory=list)
output_format_instructions: str = ""
def compute_hash(self) -> str:
"""Generate a content-based hash for integrity verification."""
content = json.dumps({
"system": self.system_template,
"user": self.user_template,
"few_shot": self.few_shot_examples,
"tools": self.tool_descriptions,
"output_format": self.output_format_instructions
}, sort_keys=True)
return hashlib.sha256(content.encode()).hexdigest()[:12]
@dataclass
class ModelVersion:
"""Pinned model identifier with provider metadata."""
provider: str # e.g., "openai", "anthropic", "local"
model_id: str # e.g., "gpt-4o-2024-08-06"
version_date: Optional[str] = None # Provider's version date if applicable
quantization: Optional[str] = None # e.g., "q4_k_m" for local models
def to_api_string(self) -> str:
"""Convert to the string expected by the provider's API."""
if self.provider == "openai":
return self.model_id
elif self.provider == "anthropic":
return self.model_id
elif self.provider == "local":
return f"{self.model_id}:{self.quantization}" if self.quantization else self.model_id
return self.model_id
@dataclass
class RuntimeConfig:
"""Hyperparameters and execution settings."""
temperature: float = 0.7
top_p: float = 1.0
max_tokens: int = 2048
presence_penalty: float = 0.0
frequency_penalty: float = 0.0
timeout_seconds: int = 60
max_retries: int = 3
memory_backend: str = "postgres" # "redis", "sqlite", "none"
memory_window_tokens: int = 8000
rate_limit_rpm: int = 100
@dataclass
class AgentVersion:
"""Complete immutable snapshot of an AI agent."""
version: str # Semantic version like "1.2.3"
prompt: PromptVersion
model: ModelVersion
config: RuntimeConfig
description: str
created_at: datetime = field(default_factory=datetime.utcnow)
parent_version: Optional[str] = None # For tracking lineage
def to_manifest(self) -> Dict:
"""Serialize to a dictionary for YAML/JSON storage."""
return {
"version": self.version,
"description": self.description,
"created_at": self.created_at.isoformat(),
"parent_version": self.parent_version,
"prompt": {
"system_template": self.prompt.system_template,
"user_template": self.prompt.user_template,
"few_shot_examples": self.prompt.few_shot_examples,
"tool_descriptions": self.prompt.tool_descriptions,
"output_format_instructions": self.prompt.output_format_instructions,
"content_hash": self.prompt.compute_hash()
},
"model": {
"provider": self.model.provider,
"model_id": self.model.model_id,
"version_date": self.model.version_date,
"quantization": self.model.quantization
},
"config": {
"temperature": self.config.temperature,
"top_p": self.config.top_p,
"max_tokens": self.config.max_tokens,
"presence_penalty": self.config.presence_penalty,
"frequency_penalty": self.config.frequency_penalty,
"timeout_seconds": self.config.timeout_seconds,
"max_retries": self.config.max_retries,
"memory_backend": self.config.memory_backend,
"memory_window_tokens": self.config.memory_window_tokens,
"rate_limit_rpm": self.config.rate_limit_rpm
}
}Step 2: Build the Version Registry
Next, we need a registry that stores versions, handles semantic versioning logic, and provides lookup capabilities. We'll implement a file-based registry (suitable for git tracking) with an in-memory cache for production use.
# version_registry.py
import yaml
import json
import os
import re
from pathlib import Path
from typing import Dict, List, Optional, Tuple
from agent_version_schema import AgentVersion, PromptVersion, ModelVersion, RuntimeConfig
from datetime import datetime
class AgentVersionRegistry:
"""Central registry for managing agent versions.
Stores versions as YAML files in a versions/ directory,
enabling git-based version control and easy inspection.
"""
def __init__(self, storage_path: str = "./agent_versions"):
self.storage_path = Path(storage_path)
self.storage_path.mkdir(parents=True, exist_ok=True)
self._cache: Dict[str, AgentVersion] = {}
self._active_version: Optional[str] = None
self._load_all_versions()
def _version_filename(self, version: str) -> Path:
"""Get the file path for a version manifest."""
safe_name = version.replace(".", "_")
return self.storage_path / f"agent_v{safe_name}.yaml"
def _parse_semver(self, version: str) -> Tuple[int, int, int]:
"""Parse a semantic version string into (major, minor, patch)."""
match = re.match(r"^(\d+)\.(\d+)\.(\d+)$", version)
if not match:
raise ValueError(f"Invalid semantic version: {version}. Must be MAJOR.MINOR.PATCH")
return int(match.group(1)), int(match.group(2)), int(match.group(3))
def register(self, agent_version: AgentVersion) -> AgentVersion:
"""Register a new agent version. Validates and persists to disk."""
# Validate semantic version format
self._parse_semver(agent_version.version)
# Check for duplicates
if agent_version.version in self._cache:
raise ValueError(f"Version {agent_version.version} already exists. Use a new version number.")
# Validate parent exists if specified
if agent_version.parent_version and agent_version.parent_version not in self._cache:
raise ValueError(f"Parent version {agent_version.parent_version} not found.")
# Persist to YAML file
manifest = agent_version.to_manifest()
filepath = self._version_filename(agent_version.version)
with open(filepath, "w") as f:
yaml.dump(manifest, f, default_flow_style=False, sort_keys=False, allow_unicode=True)
# Update cache
self._cache[agent_version.version] = agent_version
return agent_version
def get(self, version: str) -> AgentVersion:
"""Retrieve a specific version by its semantic version string."""
if version == "latest":
version = self._get_latest_version()
if version not in self._cache:
raise KeyError(f"Version {version} not found in registry.")
return self._cache[version]
def set_active(self, version: str) -> None:
"""Designate a version as the active production version."""
if version not in self._cache:
raise KeyError(f"Cannot activate unknown version: {version}")
self._active_version = version
# Write active pointer file
pointer_path = self.storage_path / "active_version.json"
with open(pointer_path, "w") as f:
json.dump({"active": version, "updated_at": datetime.utcnow().isoformat()}, f)
def get_active(self) -> AgentVersion:
"""Get the currently active production version."""
if self._active_version is None:
pointer_path = self.storage_path / "active_version.json"
if pointer_path.exists():
with open(pointer_path) as f:
data = json.load(f)
self._active_version = data["active"]
else:
self._active_version = self._get_latest_version()
return self.get(self._active_version)
def list_versions(self) -> List[str]:
"""Return all registered versions sorted by semver."""
versions = list(self._cache.keys())
return sorted(versions, key=lambda v: self._parse_semver(v))
def get_lineage(self, version: str) -> List[str]:
"""Trace the ancestry chain of a version."""
lineage = [version]
current = self._cache.get(version)
while current and current.parent_version:
lineage.append(current.parent_version)
current = self._cache.get(current.parent_version)
return lineage
def diff(self, version_a: str, version_b: str) -> Dict:
"""Compare two versions and return the differences."""
v_a = self.get(version_a)
v_b = self.get(version_b)
diffs = {}
# Compare prompts
if v_a.prompt.system_template != v_b.prompt.system_template:
diffs["prompt.system_template"] = "changed"
if v_a.prompt.user_template != v_b.prompt.user_template:
diffs["prompt.user_template"] = "changed"
if v_a.prompt.few_shot_examples != v_b.prompt.few_shot_examples:
diffs["prompt.few_shot_examples"] = "changed"
# Compare models
if v_a.model.model_id != v_b.model.model_id:
diffs["model.model_id"] = f"{v_a.model.model_id} → {v_b.model.model_id}"
if v_a.model.provider != v_b.model.provider:
diffs["model.provider"] = f"{v_a.model.provider} → {v_b.model.provider}"
# Compare configs
for field in ["temperature", "top_p", "max_tokens", "max_retries"]:
val_a = getattr(v_a.config, field)
val_b = getattr(v_b.config, field)
if val_a != val_b:
diffs[f"config.{field}"] = f"{val_a} → {val_b}"
return diffs
def _load_all_versions(self) -> None:
"""Load all version manifests from disk into cache."""
for filepath in self.storage_path.glob("agent_v*.yaml"):
with open(filepath) as f:
manifest = yaml.safe_load(f)
version = self._manifest_to_version(manifest)
self._cache[version.version] = version
def _get_latest_version(self) -> str:
"""Get the highest semantic version."""
versions = self.list_versions()
return versions[-1] if versions else "0.0.0"
def _manifest_to_version(self, manifest: Dict) -> AgentVersion:
"""Reconstruct an AgentVersion from a YAML manifest."""
return AgentVersion(
version=manifest["version"],
description=manifest.get("description", ""),
created_at=datetime.fromisoformat(manifest["created_at"]) if "created_at" in manifest else datetime.utcnow(),
parent_version=manifest.get("parent_version"),
prompt=PromptVersion(
system_template=manifest["prompt"]["system_template"],
user_template=manifest["prompt"]["user_template"],
few_shot_examples=manifest["prompt"].get("few_shot_examples", []),
tool_descriptions=manifest["prompt"].get("tool_descriptions", []),
output_format_instructions=manifest["prompt"].get("output_format_instructions", "")
),
model=ModelVersion(
provider=manifest["model"]["provider"],
model_id=manifest["model"]["model_id"],
version_date=manifest["model"].get("version_date"),
quantization=manifest["model"].get("quantization")
),
config=RuntimeConfig(
temperature=manifest["config"].get("temperature", 0.7),
top_p=manifest["config"].get("top_p", 1.0),
max_tokens=manifest["config"].get("max_tokens", 2048),
presence_penalty=manifest["config"].get("presence_penalty", 0.0),
frequency_penalty=manifest["config"].get("frequency_penalty", 0.0),
timeout_seconds=manifest["config"].get("timeout_seconds", 60),
max_retries=manifest["config"].get("max_retries", 3),
memory_backend=manifest["config"].get("memory_backend", "postgres"),
memory_window_tokens=manifest["config"].get("memory_window_tokens", 8000),
rate_limit_rpm=manifest["config"].get("rate_limit_rpm", 100)
)
)Step 3: Create a Versioned Agent Runner
Now we build the actual agent runner that uses the registry to execute with a specific version. This runner is version-aware — it logs which version processed each request and can switch versions without code changes.
# versioned_agent_runner.py
import openai
import anthropic
import time
import json
import logging
from typing import Dict, Any, Optional, List
from agent_version_schema import AgentVersion
from version_registry import AgentVersionRegistry
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("versioned_agent")
class VersionedAgentRunner:
"""An AI agent runner that is fully version-aware.
Every execution is tied to a specific AgentVersion snapshot,
ensuring reproducibility and auditability.
"""
def __init__(self, registry: AgentVersionRegistry, version: Optional[str] = None):
self.registry = registry
self.version = version # If None, uses active version
self._agent_version = None
def _resolve_version(self) -> AgentVersion:
"""Resolve which version to use for execution."""
if self.version:
return self.registry.get(self.version)
return self.registry.get_active()
def run(self, user_input: str, conversation_history: Optional[List[Dict]] = None) -> Dict[str, Any]:
"""Execute the agent with the resolved version.
Returns a dict containing the response and full execution metadata.
"""
agent_version = self._resolve_version()
# Build the full prompt from templates
system_prompt = agent_version.prompt.system_template
# Inject tool descriptions into system prompt if present
if agent_version.prompt.tool_descriptions:
tools_json = json.dumps(agent_version.prompt.tool_descriptions, indent=2)
system_prompt = system_prompt.replace("{{TOOLS}}", tools_json)
# Inject output format instructions
if agent_version.prompt.output_format_instructions:
system_prompt = system_prompt.replace(
"{{OUTPUT_FORMAT}}",
agent_version.prompt.output_format_instructions
)
# Build user message with few-shot examples
user_message = agent_version.prompt.user_template.replace("{{USER_INPUT}}", user_input)
if agent_version.prompt.few_shot_examples:
few_shot_block = "\n\n".join([
f"Example:\nInput: {ex['input']}\nOutput: {ex['output']}"
for ex in agent_version.prompt.few_shot_examples
])
user_message = f"{few_shot_block}\n\n{user_message}"
# Prepare messages array
messages = [{"role": "system", "content": system_prompt}]
if conversation_history:
messages.extend(conversation_history)
messages.append({"role": "user", "content": user_message})
start_time = time.time()
# Route to the appropriate provider based on model config
try:
if agent_version.model.provider == "openai":
response = self._call_openai(agent_version, messages)
elif agent_version.model.provider == "anthropic":
response = self._call_anthropic(agent_version, messages)
else:
raise ValueError(f"Unsupported provider: {agent_version.model.provider}")
except Exception as e:
logger.error(f"Agent execution failed for version {agent_version.version}: {e}")
raise
elapsed_ms = (time.time() - start_time) * 1000
# Return full metadata alongside the response
return {
"response": response,
"metadata": {
"agent_version": agent_version.version,
"prompt_hash": agent_version.prompt.compute_hash(),
"model_id": agent_version.model.model_id,
"model_provider": agent_version.model.provider,
"temperature": agent_version.config.temperature,
"max_tokens": agent_version.config.max_tokens,
"elapsed_ms": round(elapsed_ms, 2),
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"retry_count": 0 # Would be populated by retry logic
}
}
def _call_openai(self, agent_version: AgentVersion, messages: List[Dict]) -> str:
"""Execute via OpenAI API with version-specific config."""
client = openai.OpenAI()
response = client.chat.completions.create(
model=agent_version.model.to_api_string(),
messages=messages,
temperature=agent_version.config.temperature,
top_p=agent_version.config.top_p,
max_tokens=agent_version.config.max_tokens,
presence_penalty=agent_version.config.presence_penalty,
frequency_penalty=agent_version.config.frequency_penalty,
timeout=agent_version.config.timeout_seconds
)
return response.choices[0].message.content
def _call_anthropic(self, agent_version: AgentVersion, messages: List[Dict]) -> str:
"""Execute via Anthropic API with version-specific config."""
client = anthropic.Anthropic()
# Extract system message for Anthropic's API format
system_msg = next((m["content"] for m in messages if m["role"] == "system"), "")
user_messages = [m for m in messages if m["role"] != "system"]
response = client.messages.create(
model=agent_version.model.to_api_string(),
system=system_msg,
messages=user_messages,
max_tokens=agent_version.config.max_tokens,
temperature=agent_version.config.temperature,
timeout=agent_version.config.timeout_seconds
)
return response.content[0].textStep 4: Using the Versioning System End-to-End
Here's how to put it all together — creating versions, switching between them, and running agents with full traceability.
# example_usage.py
from agent_version_schema import AgentVersion, PromptVersion, ModelVersion, RuntimeConfig
from version_registry import AgentVersionRegistry
from versioned_agent_runner import VersionedAgentRunner
# Initialize the registry (creates ./agent_versions/ directory)
registry = AgentVersionRegistry("./agent_versions")
# ── Version 1.0.0: Initial customer support agent ──
v1 = AgentVersion(
version="1.0.0",
description="Initial customer support agent with GPT-4o",
prompt=PromptVersion(
system_template="""You are a helpful customer support agent for Acme Corp.
Your tone is friendly and professional.
{{TOOLS}}
{{OUTPUT_FORMAT}}""",
user_template="""{{USER_INPUT}}
Please help the customer resolve their issue step by step.""",
few_shot_examples=[
{"input": "I can't log into my account",
"output": "I understand the frustration. Let's reset your password. First, can you tell me..."},
{"input": "My order hasn't arrived",
"output": "I'm sorry about the delay. Let me check your order status. Could you provide..."}
],
tool_descriptions=[
{"name": "lookup_order", "description": "Find order by customer email or order ID"},
{"name": "reset_password", "description": "Trigger password reset email for a user"}
],
output_format_instructions="Always end with a clear next step or action item."
),
model=ModelVersion(
provider="openai",
model_id="gpt-4o-2024-08-06"
),
config=RuntimeConfig(
temperature=0.7,
max_tokens=1024,
max_retries=2
)
)
registry.register(v1)
# ── Version 1.1.0: Lower temperature for more consistency ──
v1_1 = AgentVersion(
version="1.1.0",
description="Reduced temperature to 0.3 for more consistent support responses",
parent_version="1.0.0",
prompt=v1.prompt, # Same prompt
model=v1.model, # Same model
config=RuntimeConfig(
temperature=0.3, # Changed from 0.7
max_tokens=1024,
max_retries=2
)
)
registry.register(v1_1)
# ── Version 2.0.0: Major upgrade - switch to Claude Sonnet with revised prompt ──
v2 = AgentVersion(
version="2.0.0",
description="Migrated to Claude 3.5 Sonnet with redesigned concise prompt",
parent_version="1.1.0",
prompt=PromptVersion(
system_template="""You are Acme Corp's support AI. Be concise and actionable.
Available tools: {{TOOLS}}
Format requirement: {{OUTPUT_FORMAT}}""",
user_template="{{USER_INPUT}}",
few_shot_examples=[], # Claude needs fewer examples
tool_descriptions=v1.prompt.tool_descriptions,
output_format_instructions="Respond in 3 sentences max. Include exact next steps."
),
model=ModelVersion(
provider="anthropic",
model_id="claude-3-5-sonnet-20241022"
),
config=RuntimeConfig(
temperature=0.2, # Claude works well with lower temperature
max_tokens=512, # Enforce conciseness
max_retries=1
)
)
registry.register(v2)
# Activate version 1.1.0 for production (conservative rollout)
registry.set_active("1.1.0")
# Create runner instances for different environments
prod_runner = VersionedAgentRunner(registry) # Uses active version (1.1.0)
experimental_runner = VersionedAgentRunner(registry, version="2.0.0")
# Run both versions on the same input for comparison
user_query = "I received a damaged product in my order #12345"
print("=== Production (v1.1.0 - GPT-4o, temp 0.3) ===")
result_prod = prod_runner.run(user_query)
print(f"Response: {result_prod['response']}")
print(f"Metadata: version={result_prod['metadata']['agent_version']}, "
f"model={result_prod['metadata']['model_id']}, "
f"elapsed={result_prod['metadata']['elapsed_ms']}ms")
print("\n=== Experimental (v2.0.0 - Claude Sonnet, temp 0.2) ===")
result_exp = experimental_runner.run(user_query)
print(f"Response: {result_exp['response']}")
print(f"Metadata: version={result_exp['metadata']['agent_version']}, "
f"model={result_exp['metadata']['model_id']}, "
f"elapsed={result_exp['metadata']['elapsed_ms']}ms")
# Inspect version lineage
print("\n=== Version Lineage for 2.0.0 ===")
lineage = registry.get_lineage("2.0.0")
print(" → ".join(lineage))
# Diff between versions
print("\n=== Diff: 1.1.0 vs 2.0.0 ===")
diffs = registry.diff("1.1.0", "2.0.0")
for key, value in diffs.items():
print(f" {key}: {value}")
# List all registered versions
print("\n=== All Registered Versions ===")
for v in registry.list_versions():
agent = registry.get(v)
print(f" {v} - {agent.description} (model: {agent.model.model_id})")Step 5: Adding Evaluation Tracking
Versioning becomes truly powerful when paired with evaluation. Here's a lightweight evaluation store that ties test results to specific agent versions.
# eval_tracker.py
import json
from pathlib import Path
from datetime import datetime
from typing import Dict, List, Any
class EvalTracker:
"""Tracks evaluation results tied to specific agent versions."""
def __init__(self, storage_path: str = "./evals"):
self.storage_path = Path(storage_path)
self.storage_path.mkdir(parents=True, exist_ok=True)
def record_eval(self, agent_version: str, eval_name: str,
metrics: Dict[str, float], samples: List[Dict]) -> str:
"""Record an evaluation run against a specific agent version."""
eval_id = f"{agent_version}_{eval_name}_{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}"
record = {
"eval_id": eval_id,
"agent_version": agent_version,
"eval_name": eval_name,
"timestamp": datetime.utcnow().isoformat(),
"metrics": metrics,
"sample_count": len(samples),
"samples": samples
}
filepath = self.storage_path / f"eval_{eval_id}.json"
with open(filepath, "w") as f:
json.dump(record, f, indent=2, default=str)
return eval_id
def compare_versions(self, version_a: str, version_b: str,
eval_name: str) -> Dict[str, Any]:
"""Compare the latest eval results for two versions."""
evals_a = self._find_evals(version_a, eval_name)
evals_b = self._find_evals(version_b, eval_name)
if not evals_a or not evals_b:
return {"error": "Missing eval data for one or both versions"}
latest_a = evals_a[-1]
latest_b = evals_b[-1]
comparison = {}
for metric in set(list(latest_a["metrics"].keys()) + list(latest_b["metrics"].keys())):
val_a = latest_a["metrics"].get(metric, 0)
val_b = latest_b["metrics"].get(metric, 0)
comparison[metric] = {
"version_a": val_a,
"version_b": val_b,
"delta": round(val_b - val_a, 4),
"winner": "b" if val_b > val_a else "a" if val_a > val_b else "tie"
}
return {
"version_a": version_a,
"version_b": version_b,
"eval_name": eval_name,
"comparison": comparison
}
def _find_evals(self, version: str, eval_name: str) -> List[Dict]:
"""Find all eval records for a specific version and eval name."""
results = []
for filepath in self.storage_path.glob("eval_*.json"):
with open(filepath) as f:
record = json.load(f)
if record["agent_version"] == version and record["eval_name"] == eval_name:
results.append(record)
return sorted(results, key=lambda r: r["timestamp"])Best Practices for AI Agent Versioning
1. Use Semantic Versioning with AI-Aware Semantics
Adapt semver to the realities of LLM behavior:
- MAJOR (X.0.0) — Model provider change, fundamentally different prompt architecture, or breaking changes to tool contracts.
- MINOR (0.X.0) — Prompt refinement, few-shot example updates, temperature adjustments, or adding new tools while keeping existing ones.
- PATCH (0.0.X) — Typo fixes in prompts, retry logic tweaks, timeout adjustments, or non-behavioral config changes.
Don't blindly follow this — if a "minor" prompt tweak causes 20% accuracy swing, treat it as major in your changelog communication.
2. Version the Entire Snapshot, Never Individual Files
The cardinal mistake is versioning prompts in one repo, models in an environment variable, and configs in a YAML file. An agent version must be an atomic unit — one version number captures the complete state. Our registry implementation enforces this by design.
3. Maintain a Staging-to-Production Promotion Pipeline
Never directly edit the active production version. The workflow should be:
# Ideal promotion workflow
registry.register(candidate_version) # 1. Register candidate
candidate_runner = VersionedAgentRunner(registry, version="2.1.0")
eval_tracker.record_eval("2.1.0", "accuracy", results) # 2. Evaluate
# After passing eval gate:
registry.set_active("2.1.0") # 3. Promote to production4. Log the Version in Every Response and Decision
Every agent output should carry version metadata. This is invaluable for debugging: when a user reports "the agent gave a weird answer on Tuesday at 2pm," you can look up the exact prompt, model, and temperature that produced it.
5. Implement Automated Regression Gates
Before promoting any new version, run a battery of eval suites against it. Compare against the current production version. Set thresholds — if helpfulness score drops by more than 5%, block the promotion.
# Automated promotion gate pseudocode
def promote_if_safe(candidate_version: str, threshold: float = 0.05):
current = registry.get_active()
comparison = eval_tracker.compare_versions(
current.version, candidate_version, "helpfulness"
)
helpfulness_delta = comparison["comparison"]["helpfulness"]["delta"]
if helpfulness_delta < -threshold:
raise Exception(f"Blocked: helpfulness dropped by {abs(helpfulness_delta):.2%}")
registry.set_active(candidate_version)6. Keep a Human-Readable Changelog
YAML manifests are machine-readable, but humans need context. Maintain a CHANGELOG.md that explains