Introduction to Model Alignment: SFT vs RLHF
Large Language Models (LLMs) are typically pre-trained on massive amounts of raw text data. While this gives them a vast amount of world knowledge and language capabilities, a raw pre-trained model is essentially an advanced autocomplete. It does not inherently know how to follow instructions, hold a conversation, or refuse harmful requests. To transform a base model into a helpful assistant, developers use alignment techniques. The two most prominent methods are Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF).
What is Supervised Fine-Tuning (SFT)?
Supervised Fine-Tuning (SFT) is the process of taking a pre-trained base model and training it further on a dataset of high-quality prompt-and-response pairs. In this phase, the model learns by example. You provide the input (the prompt) and the exact desired output (the response). The model updates its weights to minimize the difference between its generated text and the target response. SFT teaches the model the format, tone, and basic instruction-following behaviors required to act as a chatbot or task-specific agent.
What is Reinforcement Learning from Human Feedback (RLHF)?
Reinforcement Learning from Human Feedback (RLHF) is a more complex alignment technique that refines the model's behavior based on human preferences. Instead of providing a single "correct" answer, human annotators are shown multiple model outputs for a given prompt and rank them from best to worst. This ranking data is used to train a separate Reward Model. Finally, an RL algorithm (like Proximal Policy Optimization, or PPO) is used to optimize the LLM to generate responses that maximize the score given by the Reward Model. Recently, Direct Preference Optimization (DPO) has emerged as a simpler, more stable alternative to traditional RLHF, bypassing the need for a separate reward model while achieving similar results.
Why Does This Matter?
Understanding the distinction between SFT and RLHF is critical for AI developers. SFT is excellent for teaching a model new domains, formats (like JSON outputs), or specific tasks. However, SFT can be limited by the quality and diversity of the human-written dataset, and it often struggles with subjective concepts like "helpfulness" or "harmlessness." RLHF (or DPO) is necessary to push the model beyond simple mimicry, allowing it to generalize human preferences, avoid toxic outputs, and handle edge cases where multiple valid responses exist but some are clearly better than others. Together, they form the standard pipeline for creating state-of-the-art conversational AI.
How to Use SFT: A Practical Example
Implementing SFT is straightforward using the Hugging Face transformers and trl libraries. Below is an example of how to fine-tune a base model using the SFTTrainer.
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
# 1. Load the base model and tokenizer
model_id = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
# 2. Load a dataset of prompt-response pairs
# Dataset format expected: {"text": "User: Hello!\nAssistant: Hi there!"}
dataset = load_dataset("timdettmers/openassistant-guanaco", split="train")
# 3. Configure the SFT training arguments
sft_config = SFTConfig(
output_dir="./sft_model_output",
per_device_train_batch_size=4,
num_train_epochs=3,
learning_rate=2e-5,
save_steps=500,
logging_steps=10,
)
# 4. Initialize the SFT Trainer
trainer = SFTTrainer(
model=model,
args=sft_config,
train_dataset=dataset,
tokenizer=tokenizer,
)
# 5. Start training
trainer.train()
trainer.save_model("./sft_model_output")
How to Use RLHF (via DPO): A Practical Example
Traditional RLHF with PPO is notoriously unstable and resource-intensive. Today, many developers use Direct Preference Optimization (DPO) as their RLHF method. DPO requires a dataset containing a prompt, a chosen response, and a rejected response. Here is how to apply DPO to your previously fine-tuned SFT model.
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset
# 1. Load the SFT model (the model you just trained above)
model_id = "./sft_model_output"
model = AutoModelForCausalLM.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
# 2. Load a preference dataset
# Expected format: {"prompt": "...", "chosen": "...", "rejected": "..."}
dataset = load_dataset("Anthropic/hh-rlhf", split="train")
# 3. Configure DPO training arguments
dpo_config = DPOConfig(
output_dir="./dpo_model_output",
beta=0.1, # Controls deviation from the initial SFT model
per_device_train_batch_size=4,
num_train_epochs=1,
learning_rate=5e-6,
save_steps=500,
logging_steps=10,
)
# 4. Initialize the DPO Trainer
trainer = DPOTrainer(
model=model,
args=dpo_config,
train_dataset=dataset,
tokenizer=tokenizer,
)
# 5. Start preference optimization
trainer.train()
trainer.save_model("./dpo_model_output")
Best Practices for SFT and RLHF
- Quality over Quantity: For SFT, a few thousand highly curated, diverse, and flawlessly formatted examples will outperform a million noisy, scraped examples. The same applies to preference data for RLHF.
- Always do SFT first: You cannot apply RLHF/DPO directly to a base model effectively. The SFT phase initializes the model to a reasonable distribution where it can understand instructions, making the RLHF phase stable and effective.
- Watch for Reward Hacking: In RLHF, models can sometimes learn to exploit the reward model (e.g., generating overly verbose responses if length correlates with higher scores). Monitor outputs closely and update your preference dataset if you notice degenerate behaviors.
- Prevent Catastrophic Forgetting: Aggressive fine-tuning can cause a model to forget its pre-trained knowledge. Use lower learning rates and consider mixing in general instruction datasets during SFT to retain broad capabilities.
- Evaluate Rigorously: Do not rely solely on training loss. Use automated benchmarks (like MT-Bench or AlpacaEval) and human evaluation to ensure the model is actually improving in helpfulness and safety.
Conclusion
Supervised Fine-Tuning and Reinforcement Learning from Human Feedback are complementary pillars of modern LLM development. SFT acts as the foundation, teaching the model the structural and stylistic basics of instruction following through direct examples. RLHF, particularly through accessible methods like DPO, builds upon that foundation by aligning the model with nuanced human preferences, safety guidelines, and subjective quality standards. By mastering both techniques and applying them sequentially, developers can transform raw, unpredictable base models into highly capable, safe, and reliable AI assistants tailored to their specific use cases.