Introduction to EleutherAI LM Evaluation Harness
The EleutherAI LM Evaluation Harness is an open-source framework designed to evaluate language models on a wide variety of standardized benchmarks. Originally developed by EleutherAI, the research collective behind models like GPT-Neo and GPT-J, the harness has become one of the most widely adopted tools in the open-source AI community for measuring model capabilities across reasoning, knowledge, coding, mathematics, and language understanding tasks.
Whether you are fine-tuning a model from Hugging Face, training a new architecture from scratch, or simply comparing different checkpoints, the harness provides a consistent, reproducible way to measure progress. It supports hundreds of tasks out of the box and integrates seamlessly with popular model formats, including Hugging Face Transformers, GGUF, and vLLM-served models.
Why Benchmarking Matters
Without rigorous evaluation, claims about model performance are anecdotal. Benchmarks provide a shared vocabulary for comparison. When you publish a model with scores on MMLU, GSM8K, and HellaSwag, other developers can immediately contextualize its strengths and weaknesses relative to established baselines. This is especially important in the open-source ecosystem, where dozens of new models are released weekly.
Benchmarking also catches regressions. A fine-tuning run that improves conversational tone might silently degrade factual accuracy. Running a battery of evaluations before and after changes ensures you are not trading one capability for another unknowingly.
Key Features and Architecture
The harness is built around a modular design. At its core are three components: the model wrapper, the task definitions, and the evaluation engine. The model wrapper abstracts away the differences between inference backends, so the same task can be run against a local Hugging Face model, a remote API, or a quantized GGUF file. Task definitions are declarative YAML or Python files that specify the dataset, prompt formatting, answer extraction, and scoring metric. The evaluation engine ties these together, handling batching, caching, and result aggregation.
- Multi-backend support: Hugging Face Transformers, vLLM, GGUF via llama.cpp, OpenAI-compatible APIs, and more.
- Extensive task library: Over 200 tasks including MMLU, GSM8K, ARC, HellaSwag, TruthfulQA, HumanEval, and WinoGrande.
- Few-shot evaluation: Configurable number of few-shot examples per task, matching published evaluation protocols.
- Reproducibility: Deterministic seeding and result logging ensure runs can be compared across time and hardware.
- Custom tasks: Define your own evaluation tasks with minimal boilerplate.
Installation and Setup
The harness is distributed as a Python package called lm-eval. It requires Python 3.9 or higher. The recommended approach is to create a fresh virtual environment to avoid dependency conflicts, especially with Hugging Face libraries that update frequently.
# Create and activate a virtual environment
python -m venv lm-eval-env
source lm-eval-env/bin/activate
# Install the base package
pip install lm-eval
# Or install with extras for specific backends
pip install "lm-eval[vllm]"
pip install "lm-eval[api]"
pip install "lm-eval[all]"
After installation, verify that the CLI is available and check the version:
lm_eval --version
If you plan to evaluate models hosted on Hugging Face, ensure you are authenticated. Some gated models require an access token:
huggingface-cli login
Running Your First Evaluation
The fastest way to get started is to evaluate a small model on a single task. Let us run the HellaSwag benchmark against gpt2, which is small enough to run on a laptop CPU in a few minutes.
lm_eval --model hf \
--model_args pretrained=gpt2 \
--tasks hellaswag \
--device cpu \
--batch_size 8
This command downloads GPT-2 from Hugging Face, loads the HellaSwag dataset, runs the evaluation, and prints a results table to the console. The output will look something like this:
|Tasks|Version|Filter|n-shot|Metric| |Value | |Stderr|
|-----|------:|------|-----:|------|---|-----:|---|-----:|
|hellaswag| 1|none | 0|acc |↑ |0.2962|± |0.0045|
| | |none | 0|acc_norm|↑ |0.3142|± |0.0046|
The acc metric is raw accuracy, while acc_norm normalizes by the length of the continuation, which is the standard reporting metric for HellaSwag.
Evaluating Multiple Tasks at Once
In practice, you will want to evaluate a model across several benchmarks simultaneously. You can pass a comma-separated list of task names:
lm_eval --model hf \
--model_args pretrained=meta-llama/Llama-3.2-1B-Instruct \
--tasks hellaswag,arc_easy,arc_challenge,winogrande \
--device cuda \
--batch_size 16 \
--output_path ./results/llama-1b/
The --output_path flag writes a JSON file with detailed results, including per-example predictions, which is invaluable for error analysis.
Understanding Task Groups and Standard Suites
Manually listing every task becomes tedious. The harness provides task groups, which bundle related tasks into a single named suite. The most commonly used group is mmlu, which encompasses all 57 MMLU subjects. Similarly, leaderboard groups replicate the evaluation suite used by the Hugging Face Open LLM Leaderboard.
lm_eval --model hf \
--model_args pretrained=meta-llama/Llama-3.2-1B-Instruct \
--tasks mmlu \
--num_fewshot 5 \
--device cuda \
--batch_size 32
Here, --num_fewshot 5 instructs the harness to prepend five example questions to each prompt, matching the standard MMLU evaluation protocol. Different tasks have different canonical few-shot settings; MMLU uses 5-shot, GSM8K uses 5-shot, and HellaSwag typically uses 0-shot. Always check the original paper or leaderboard documentation for the correct setting.
Working with Different Model Backends
Hugging Face Transformers
The hf backend is the default and most flexible. It supports any model on the Hugging Face Hub. You can pass additional arguments to control loading behavior:
lm_eval --model hf \
--model_args pretrained=microsoft/phi-2,dtype=float16 \
--tasks gsm8k,mmlu \
--num_fewshot 5 \
--device cuda \
--batch_size 8 \
--max_length 4096
The dtype parameter controls precision. Using float16 or bfloat16 reduces memory usage and speeds up inference on modern GPUs. The max_length parameter sets the context window, which is important for tasks with long prompts.
vLLM Backend
For large models, the vLLM backend offers dramatically faster inference through paged attention and continuous batching. This is the recommended backend for models with 7 billion parameters or larger:
lm_eval --model vllm \
--model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16,gpu_memory_utilization=0.9 \
--tasks gsm8k,mmlu,hellaswag,arc_challenge,winogrande,truthfulqa \
--num_fewshot 5 \
--batch_size auto
Setting --batch_size auto lets vLLM manage its own batching dynamically, which typically yields the best throughput.
API Backend
If you want to evaluate a model served behind an OpenAI-compatible API endpoint, use the local-completions or local-chat-completions backend:
lm_eval --model local-completions \
--model_args model=my-model,base_url=http://localhost:8000/v1/completions,num_concurrent=8 \
--tasks mmlu,gsm8k \
--num_fewshot 5
This is useful when evaluating models deployed with TGI, Ollama, or a custom inference server. The num_concurrent parameter controls parallelism to avoid overwhelming the server.
Creating a Custom Evaluation Task
One of the most powerful features of the harness is the ability to define custom tasks. A task is defined by a YAML file that specifies the dataset, prompt template, target extraction, and metric. Let us create a simple custom task for a hypothetical medical Q&A dataset.
Create a directory called custom_tasks and add a file named medical_qa.yaml:
task: medical_qa
dataset_path: json
dataset_kwargs:
data_files:
validation: ./data/medical_qa.json
output_type: multiple_choice
training_split: null
validation_split: validation
test_split: null
doc_to_text: "Question: {{question}}\nAnswer:"
doc_to_target: "{{answer}}"
doc_to_choice: ["A", "B", "C", "D"]
metric_list:
- metric: acc
aggregation: mean
higher_is_better: true
- metric: acc_norm
aggregation: mean
higher_is_better: true
The dataset file ./data/medical_qa.json should be a JSON array of objects, each with question and answer fields:
[
{
"question": "What is the first-line treatment for hypertension in a 55-year-old diabetic patient?",
"answer": "A"
},
{
"question": "Which antibody is most specific for celiac disease?",
"answer": "B"
}
]
Run the custom task by pointing the harness to your task directory:
lm_eval --model hf \
--model_args pretrained=gpt2 \
--tasks medical_qa \
--include_path ./custom_tasks \
--device cuda \
--batch_size 4
The --include_path flag tells the harness to scan that directory for additional task definitions. This makes it straightforward to build proprietary evaluation suites for domain-specific applications.
Using the Python API
While the CLI is convenient for quick evaluations, the Python API gives you full programmatic control. This is essential when you want to integrate benchmarking into a training pipeline, compare checkpoints programmatically, or build custom reporting dashboards.
from lm_eval import simple_evaluate
from lm_eval.models.huggingface import HFLM
# Load the model
model = HFLM(
pretrained="meta-llama/Llama-3.2-1B-Instruct",
dtype="float16",
device="cuda",
batch_size=16,
)
# Run evaluation
results = simple_evaluate(
model=model,
tasks=["mmlu", "gsm8k", "hellaswag"],
num_fewshot=5,
)
# Print a summary
print(results["results"])
# Access per-task metrics
for task_name, task_results in results["results"].items():
for metric, value in task_results.items():
if isinstance(value, (int, float)):
print(f"{task_name} | {metric}: {value:.4f}")
The simple_evaluate function returns a dictionary containing all results, configuration metadata, and per-example logs. You can serialize this to JSON for long-term storage:
import json
with open("eval_results.json", "w") as f:
json.dump(results, f, indent=2, default=str)
Best Practices for Reliable Benchmarks
Match the Canonical Evaluation Protocol
Every benchmark has a canonical evaluation setup defined by its original authors. MMLU is 5-shot multiple choice. GSM8K is 5-shot with chain-of-thought reasoning. HellaSwag is 0-shot. If you deviate from these settings, your numbers are not comparable to published results. Always document the exact configuration you used, including the number of few-shot examples, prompt format, and any chain-of-thought scaffolding.
Use Deterministic Decoding for Generative Tasks
For tasks that require the model to generate free-form text, such as GSM8K or HumanEval, set the temperature to zero and disable sampling. This ensures reproducible results across runs:
lm_eval --model vllm \
--model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16,temperature=0.0,seed=42 \
--tasks gsm8k \
--num_fewshot 5 \
--gen_kwargs "do_sample=False,temperature=0.0"
Be Aware of Contamination
Many popular benchmarks have leaked into training data, especially for models trained on web crawls. A model that has memorized MMLU questions will score artificially high. To detect contamination, look for unusually large gaps between similar tasks, or evaluate on held-out variants like MMLU-Pro or recent benchmarks with controlled release dates. The harness includes several newer benchmarks that are less likely to be contaminated.
Control for Prompt Sensitivity
Language models are sensitive to prompt formatting. A model might score 5 points higher on MMLU with one prompt template versus another. The harness uses standardized prompt templates for each task, but if you modify them, be transparent about it. When comparing two models, ensure both are evaluated with identical prompts and few-shot examples.
Log Everything
Always save the full results JSON, the exact command used, the model commit hash, and the harness version. This metadata is essential for reproducibility. A good practice is to create a results directory structure organized by model name and date:
mkdir -p ./results/$(date +%Y-%m-%d)/llama-3.1-8b
lm_eval --model vllm \
--model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,dtype=bfloat16 \
--tasks mmlu,gsm8k,hellaswag,arc_challenge,winogrande,truthfulqa \
--num_fewshot 5 \
--batch_size auto \
--output_path ./results/$(date +%Y-%m-%d)/llama-3.1-8b/ \
--log_samples
The --log_samples flag saves every individual prompt, prediction, and gold answer, enabling detailed error analysis after the run completes.
Common Pitfalls and Troubleshooting
Out of memory errors: Reduce batch size or use a smaller dtype. For Hugging Face models, try dtype=float16 or enable 8-bit loading with load_in_8bit=True. For vLLM, lower gpu_memory_utilization.
Incorrect few-shot settings: If your MMLU score seems too low, check that you are using 5-shot. The harness defaults to 0-shot unless you specify --num_fewshot. Some tasks have task-level defaults that override the global setting, so verify in the task YAML.
Slow evaluation on CPU: CPU inference is inherently slow for large models. If you do not have a GPU, consider using the API backend with a remote server, or use a smaller model for prototyping.
Task not found errors: If you see an error like Task not found: my_task, ensure the task name is spelled correctly and that you have included any custom task directories with --include_path. You can list all available tasks with lm_eval --tasks list.
Conclusion
The EleutherAI LM Evaluation Harness is an indispensable tool for anyone working with language models. It provides a standardized, reproducible, and extensible framework for measuring model capabilities across hundreds of benchmarks. By following canonical evaluation protocols, using deterministic decoding, logging results thoroughly, and being mindful of data contamination, you can produce benchmark numbers that are trustworthy and comparable to published results. Whether you are a researcher tracking training progress, an engineer selecting a model for production, or an open-source contributor releasing a new checkpoint, the harness gives you the rigorous evaluation infrastructure needed to make informed decisions about model quality.