How to Merge LoRA Adapters with Base Models

Low-Rank Adaptation (LoRA) has become one of the most popular techniques for fine-tuning large language models efficiently. Instead of updating all the parameters of a massive model, LoRA freezes the original weights and trains small, low-rank adapter matrices that capture task-specific behavior. This dramatically reduces memory usage and training time, making it feasible to fine-tune models like Llama, Mistral, or Qwen on consumer hardware.

However, LoRA adapters are not standalone models. They are deltas — small weight updates that must be applied on top of a base model at inference time. While you can keep the adapter separate and load it dynamically, there are many scenarios where you want to merge the adapter weights directly into the base model. This produces a single, self-contained model that no longer requires the adapter at runtime, simplifying deployment and often improving inference speed.

What Is LoRA Merging?

LoRA works by approximating the weight update of a linear layer as the product of two smaller matrices. For a weight matrix W of shape (d_out, d_in), LoRA introduces two matrices A of shape (r, d_in) and B of shape (d_out, r), where r is the rank (typically 8, 16, 32, or 64). The effective weight becomes:

W_effective = W + (B @ A) * scaling

Merging means permanently computing W + (B @ A) * scaling and storing the result as the new weight matrix. After merging, the adapter matrices are no longer needed, and the model behaves identically to a fully fine-tuned model with those combined weights.

Why Merging Matters

Simplified deployment: A merged model is a single artifact. You do not need to ship both the base model and adapter files separately.
Faster inference: Without the need to compute adapter forward passes, inference latency is reduced, especially in serving frameworks like vLLM, TGI, or TensorRT-LLM.
Framework compatibility: Some inference engines and quantization tools do not support LoRA adapters natively. Merging sidesteps this limitation entirely.
Composition of multiple adapters: Merging allows you to combine several adapters with different weights, enabling multi-task or style-blended models.
Distribution: Sharing a single merged model on the Hugging Face Hub is more convenient for downstream users who may not be familiar with adapter loading.

Prerequisites

To follow along with the code examples, you will need a Python environment with the following packages installed:

pip install transformers peft accelerate torch safetensors

You will also need a base model and a trained LoRA adapter. For this tutorial, we will use meta-llama/Llama-3.1-8B as the base model and assume you have a trained adapter stored locally in a directory called ./my-lora-adapter.

Merging a Single Adapter with PEFT

The peft library from Hugging Face provides a straightforward API for merging LoRA adapters. The core method is merge_and_unload(), which performs the weight arithmetic in place and returns a standard transformers model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "meta-llama/Llama-3.1-8B"
adapter_path = "./my-lora-adapter"
output_dir = "./merged-model"

# Load the base model in float16 to save memory
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# Attach the LoRA adapter to the base model
model = PeftModel.from_pretrained(base_model, adapter_path)

# Merge adapter weights into the base model and unload the adapter
merged_model = model.merge_and_unload()

# Save the merged model and tokenizer
merged_model.save_pretrained(output_dir, safe_serialization=True)
tokenizer.save_pretrained(output_dir)

print(f"Merged model saved to {output_dir}")

After running this script, the ./merged-model directory will contain a complete model with merged weights in the safetensors format. You can load it with any standard transformers code path, just like the original base model.

Handling Memory Constraints

Merging large models can require significant memory because the full model weights must be loaded. If you are working on a GPU with limited VRAM, consider loading the model on CPU first, merging, and then moving to GPU for inference. Alternatively, use 8-bit or 4-bit loading for the base model, but note that merging into quantized weights is not always supported and may require dequantization first.

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(load_in_4bit=True)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

# Note: merging into 4-bit weights is NOT directly supported.
# For merging, load in float16 or bfloat16 on CPU instead.
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="cpu",
)

model = PeftModel.from_pretrained(base_model, adapter_path)
merged_model = model.merge_and_unload()
merged_model.save_pretrained(output_dir, safe_serialization=True)

Merging Multiple Adapters

One of the powerful features of LoRA is the ability to combine multiple adapters. The peft library supports adding multiple adapters to a single base model and merging them with custom scaling weights. This is useful for blending capabilities, such as combining a coding adapter with a reasoning adapter.

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

# Load the first adapter as the default
model = PeftModel.from_pretrained(base_model, "./adapter-coding", adapter_name="coding")

# Load additional adapters
model.load_adapter("./adapter-reasoning", adapter_name="reasoning")
model.load_adapter("./adapter-chat", adapter_name="chat")

# Set active adapters with custom weights
# The weights determine the contribution of each adapter to the final merged weights
model.set_adapter(["coding", "reasoning", "chat"])

# Merge with weighted combination
# Use add_weighted_adapter for explicit control
from peft import add_weighted_adapter

add_weighted_adapter(
    model,
    adapters=["coding", "reasoning", "chat"],
    weights=[0.5, 0.3, 0.2],
    adapter_name="merged_combo",
    combination_type="linear",
)

model.set_adapter("merged_combo")
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-combo-model", safe_serialization=True)

The combination_type parameter supports several strategies, including "linear" (weighted sum) and "cat" (concatenation along the rank dimension). Linear combination is the most common and predictable approach.

Verifying the Merged Model

After merging, it is critical to verify that the merged model produces the same outputs as the base model with the adapter loaded separately. A simple sanity check is to run the same prompt through both configurations and compare the generated tokens.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the merged model
merged_model = AutoModelForCausalLM.from_pretrained(
    "./merged-model",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("./merged-model")

prompt = "Explain the concept of gradient descent in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(merged_model.device)

with torch.no_grad():
    output = merged_model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=False,
        temperature=1.0,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Compare this output with the result from running the base model plus adapter using PeftModel.from_pretrained() without merging. The outputs should be identical or nearly identical, confirming that the merge was performed correctly.

Best Practices

Always keep the original adapter: Merging is irreversible for the saved artifact. Store your adapter separately so you can re-merge with different parameters or combine it with other adapters later.
Use safetensors format: When saving the merged model, always use safe_serialization=True. Safetensors is faster, safer, and increasingly the default in the Hugging Face ecosystem.
Test before deploying: Run a small evaluation suite on the merged model to ensure no degradation occurred. Merging is mathematically exact, but saving and reloading can introduce subtle issues with precision or configuration files.
Copy config files carefully: When saving the merged model, ensure that the config.json, generation_config.json, and tokenizer files are all copied from the base model. The save_pretrained method handles this automatically, but manual copying can lead to missing files.
Be cautious with quantized merging: Merging into GPTQ, AWQ, or bitsandbytes quantized models is not straightforward. If you need a quantized merged model, first merge in full precision, then apply quantization as a separate step.
Document adapter provenance: When distributing a merged model, include a README or metadata file documenting which base model and adapters were used, along with their versions and merge weights. This aids reproducibility.
Consider not merging for multi-tenant serving: If you serve multiple LoRA adapters to different users, keep them unmerged and use a serving framework that supports dynamic adapter loading, such as vLLM with LoRA support. Merging is best for single-use deployment scenarios.

Advanced: Merging with Custom Scripts

For maximum control, you can merge LoRA weights manually without relying on the peft library. This is useful when you need to integrate merging into a custom pipeline or when working with frameworks outside the Hugging Face ecosystem. The core operation is straightforward matrix arithmetic.

import torch
from safetensors.torch import load_file, save_file

# Load base model weights (simplified example for a single layer)
base_weights = load_file("base_model/model.safetensors")

# Load LoRA adapter weights
adapter_weights = load_file("my-lora-adapter/adapter_model.safetensors")

# Example: merge a single attention layer
# LoRA stores A and B matrices for each adapted linear layer
# The naming convention is typically: base_model.model.layers.{i}.self_attn.q_proj.lora_A.weight
# and base_model.model.layers.{i}.self_attn.q_proj.lora_B.weight

scaling = 16.0  # This value is stored in adapter_config.json

for key in list(adapter_weights.keys()):
    if key.endswith("lora_A.weight"):
        # Find the corresponding B matrix and base weight
        base_key = key.replace(".lora_A.weight", ".weight")
        b_key = key.replace("lora_A", "lora_B")

        A = adapter_weights[key]   # shape: (r, d_in)
        B = adapter_weights[b_key] # shape: (d_out, r)

        # Compute the delta: B @ A * scaling
        delta = (B @ A) * scaling

        # Add delta to the base weight
        base_weights[base_key] = base_weights[base_key] + delta.to(base_weights[base_key].dtype)

# Save the merged weights
save_file(base_weights, "merged_model/model.safetensors")

This manual approach gives you full visibility into the merging process but requires careful handling of key naming conventions, data types, and scaling factors. Always cross-reference the adapter_config.json file to confirm the correct lora_alpha and r values used during training.

Conclusion

Merging LoRA adapters with base models is a practical and often necessary step in the model deployment lifecycle. It transforms a modular training artifact into a unified, deployable model that integrates seamlessly with standard inference pipelines. Whether you use the high-level peft API for convenience or implement the merge manually for fine-grained control, understanding the underlying matrix arithmetic ensures you can debug issues, combine multiple adapters, and produce reliable production models. By following the best practices of preserving original adapters, verifying outputs, and documenting provenance, you can confidently ship merged models that retain the full capability of your fine-tuning work while simplifying the operational complexity of your serving infrastructure.