How to Format Chat Templates for Fine-Tuning

Introduction to Chat Templates

When fine-tuning Large Language Models (LLMs) for conversational AI, the way you format your input data is just as important as the data itself. A chat template is a standardized structure that defines how conversational turns—such as system prompts, user inputs, and assistant responses—are concatenated into a single string of text that the model can process.

Without a proper chat template, an LLM cannot distinguish where a user's prompt ends and where its own generated response should begin. By using a consistent format during fine-tuning, you teach the model to recognize conversational boundaries, leading to more coherent and predictable interactions during inference.

Understanding the Structure of Chat Templates

Most modern instruction-tuned models rely on a structured format that separates different participants in the conversation. This structure relies heavily on specific roles and special tokens.

Roles in a Conversation

A typical chat dataset consists of dictionaries containing a role and content. The standard roles include:

System: Sets the behavior, persona, or context for the assistant.
User: The human interacting with the model, providing prompts or questions.
Assistant: The AI model's response to the user's prompt.

Special Tokens

To demarcate these roles, models use special tokens. For example, ChatML (used by models like Zephyr and Qwen) uses <|im_start|> and <|im_end|>. Other models might use [INST] and [/INST] (like Llama 2). These tokens prevent the model from confusing conversational context with the actual text content.

How to Format Chat Templates for Fine-Tuning

The Hugging Face transformers library provides built-in support for chat templates, making it easy to convert structured message lists into model-ready text. Every tokenizer can have a chat_template attribute, usually written in Jinja2 syntax.

Applying an Existing Chat Template

If you are fine-tuning a model that already has a chat template defined in its tokenizer, you can easily format your data using the apply_chat_template method.

from transformers import AutoTokenizer

# Load a tokenizer that already has a chat template
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")

# Define a sample conversation
messages = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "Write a Python print statement."},
    {"role": "assistant", "content": "print('Hello, World!')"}
]

# Apply the chat template
# tokenize=False returns a raw string instead of token IDs
formatted_text = tokenizer.apply_chat_template(messages, tokenize=False)

print(formatted_text)

Creating a Custom Chat Template

If you are training a base model from scratch or want to enforce a specific formatting style, you can define your own custom Jinja2 template and assign it to the tokenizer.

# Define a custom Jinja2 template
custom_template = (
    "{% for message in messages %}"
    "{{ '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>\n' }}"
    "{% endfor %}"
)

# Assign the template to the tokenizer
tokenizer.chat_template = custom_template

# Test the custom template
custom_text = tokenizer.apply_chat_template(messages, tokenize=False)
print(custom_text)

Preparing Datasets for Fine-Tuning

When preparing your dataset for fine-tuning, you need to map your raw JSON data into the formatted text strings using the chat template. This is typically done using the map function provided by the Hugging Face datasets library.

from datasets import load_dataset

# Assume we have a dataset with a 'messages' column
dataset = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")

def format_chat_template(example):
    # Apply the chat template to the messages
    text = tokenizer.apply_chat_template(
        example["messages"], 
        tokenize=False
    )
    # Return the formatted text in a new column
    return {"text": text}

# Map the formatting function over the dataset
formatted_dataset = dataset.map(format_chat_template, batched=False)

print(formatted_dataset[0]["text"])

Best Practices for Chat Template Formatting

Maintain Consistency: The template used during fine-tuning must exactly match the template used during inference. If you train with <|im_start|> but prompt the model with [INST], the model's performance will degrade significantly.
Mask User Prompts During Training: To prevent the model from learning to generate user prompts, ensure that your loss function only calculates loss on the assistant's responses. Libraries like trl (Transformer Reinforcement Learning) handle this automatically when using DataCollatorForCompletionOnlyLM or similar utilities.
Always Include an End Token: Ensure your template appends an end-of-turn token (like <|im_end|>) after the assistant's response. Without this, the model will not know when to stop generating text during inference.
Save the Tokenizer: If you define a custom chat template, always save the tokenizer after training so the template is preserved in the model repository for future inference.
Test Before Training: Always print a few examples of your formatted dataset before starting a long fine-tuning run. This simple step catches formatting errors, missing newlines, or broken Jinja logic early.

Conclusion

Formatting chat templates correctly is a foundational step in the fine-tuning pipeline for conversational LLMs. By leveraging standardized roles, special tokens, and tools like Hugging Face's apply_chat_template, developers can ensure their models clearly understand conversational boundaries. Adhering to best practices—particularly maintaining consistency between training and inference and properly masking loss—will result in a robust, instruction-following model that behaves predictably in production environments.