How to Fine-Tune a Code LLM on Private Repositories

How to Fine-Tune a Code LLM on Private Repositories

General-purpose code LLMs like Code Llama, StarCoder, and DeepSeek-Coder are trained on vast amounts of public code. But they don't know your company's internal libraries, coding conventions, API patterns, or architectural decisions. Fine-tuning a code LLM on your private repositories lets you build an assistant that understands your codebase the way a senior engineer does — producing code that fits your style, uses your internal SDKs correctly, and follows your team's conventions.

In this tutorial, you'll learn the full pipeline: from extracting and cleaning data out of your repositories, through formatting it for instruction tuning, to running a parameter-efficient fine-tune with LoRA, and finally evaluating the result.

Why Fine-Tune on Private Code?

Off-the-shelf models struggle with internal code for several reasons:

• Unknown APIs: Internal services, RPC clients, and SDKs are not in the training data, so the model hallucinates method signatures.
• Convention drift: Your team may prefer specific naming, error handling, or logging patterns that differ from public code.
• Domain context: Business logic, data models, and domain terminology are unique to your organization.
• Security and compliance: You cannot send proprietary code to third-party APIs, so you need a model you can host internally.

Fine-tuning addresses these by exposing the model to your actual code patterns, teaching it the vocabulary and structure of your codebase.

Prerequisites

Before starting, make sure you have the following:

• A GPU machine with at least 24GB VRAM (an A10G, RTX 3090, or A100 works well). For larger base models, you'll want 40GB+.
• Python 3.10+ and a virtual environment.
• Access to your private repositories, checked out locally.
• Hugging Face account and a read token if you need to download gated models like Code Llama.

Install the required libraries:

pip install transformers datasets peft trl bitsandbytes accelerate torch
pip install tree-sitter tree-sitter-languages

Step 1: Extracting and Cleaning Code from Repositories

The first step is turning a pile of Git repositories into a clean dataset. Raw code files alone are not enough — you want high-quality, self-contained examples. The best source is often pairs of function signatures and their implementations, or docstrings paired with the code they describe.

Here's a script that walks a repository, parses Python files with tree-sitter, and extracts function definitions along with their docstrings:

import os
import json
from tree_sitter_languages import get_parser

parser = get_parser("python")

def extract_functions(repo_path):
    examples = []
    for root, _, files in os.walk(repo_path):
        # Skip common noise directories
        if any(part in {".git", "node_modules", "__pycache__", "venv", ".venv"} for part in root.split(os.sep)):
            continue
        for fname in files:
            if not fname.endswith(".py"):
                continue
            fpath = os.path.join(root, fname)
            try:
                with open(fpath, "r", encoding="utf-8") as f:
                    source = f.read()
            except UnicodeDecodeError:
                continue
            tree = parser.parse(source.encode("utf-8"))
            examples.extend(extract_from_tree(tree, source, fpath))
    return examples

def extract_from_tree(tree, source, fpath):
    results = []
    query = parser.language.query("""
        (function_definition
          name: (identifier) @func.name
          body: (block) @func.body)
    """)
    captures = query.captures(tree.root_node)
    funcs = {}
    for node, cap_name in captures:
        if cap_name == "func.name":
            funcs.setdefault(node.parent.id, {})["name"] = node.text.decode("utf-8")
        elif cap_name == "func.body":
            funcs.setdefault(node.parent.id, {})["body"] = node.text.decode("utf-8")
    for fid, data in funcs.items():
        if "name" not in data or "body" not in data:
            continue
        # Skip trivial functions
        if len(data["body"].strip()) < 30:
            continue
        results.append({
            "file": fpath,
            "name": data["name"],
            "code": data["body"],
        })
    return results

if __name__ == "__main__":
    all_examples = []
    for repo in ["./repos/service-a", "./repos/service-b", "./repos/shared-lib"]:
        all_examples.extend(extract_functions(repo))
    with open("raw_code.jsonl", "w") as f:
        for ex in all_examples:
            f.write(json.dumps(ex) + "\n")
    print(f"Extracted {len(all_examples)} functions")

This produces a JSONL file with one function per line. You can extend the same approach to other languages by swapping the tree-sitter parser and adjusting the query.

Step 2: Building Instruction-Tuning Examples

Raw function bodies are useful for continued pretraining, but for a coding assistant you usually want instruction-tuning data: a prompt and a completion. A practical approach is to synthesize instructions from docstrings, or to generate natural-language descriptions of functions using a smaller model. If your codebase has good docstrings, you can build pairs directly:

import json
import re

def docstring_and_body(code):
    """Split a function body into its docstring and the rest."""
    match = re.search(r'("""[\s\S]*?"""|\'\'\'[\s\S]*?\'\'\')', code)
    if not match:
        return None, None
    doc = match.group(0).strip('\"\'').strip()
    body = code[match.end():].strip()
    return doc, body

def build_instruction_dataset(input_path, output_path):
    with open(input_path) as fin, open(output_path, "w") as fout:
        count = 0
        for line in fin:
            ex = json.loads(line)
            doc, body = docstring_and_body(ex["code"])
            if not doc or not body or len(body) < 50:
                continue
            instruction = f"Implement the following function based on its docstring.\n\nDocstring:\n{doc}"
            record = {
                "instruction": instruction,
                "input": "",
                "output": body,
            }
            fout.write(json.dumps(record) + "\n")
            count += 1
        print(f"Wrote {count} instruction examples")

build_instruction_dataset("raw_code.jsonl", "instructions.jsonl")

For richer data, you can also mine commit history: pair a commit message with the diff to teach the model how to make targeted changes. Pull request descriptions paired with their merged diffs are another excellent source.

Step 3: Formatting and Tokenizing

Code LLMs are sensitive to formatting. Pick a prompt template and stick with it consistently across training and inference. Here's a dataset class that formats examples using a clear delimiter structure:

from datasets import load_dataset
from transformers import AutoTokenizer

MODEL_NAME = "bigcode/starcoderbase-3b"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

PROMPT_TEMPLATE = """<|user|>
{instruction}
<|assistant|>
{output}{eos}"""

def format_example(example):
    text = PROMPT_TEMPLATE.format(
        instruction=example["instruction"],
        output=example["output"],
        eos=tokenizer.eos_token,
    )
    return {"text": text}

raw_ds = load_dataset("json", data_files="instructions.jsonl", split="train")
ds = raw_ds.map(format_example, remove_columns=raw_ds.column_names)

def tokenize_fn(example):
    tokens = tokenizer(
        example["text"],
        truncation=True,
        max_length=1024,
        padding=False,
    )
    tokens["labels"] = tokens["input_ids"].copy()
    return tokens

tokenized = ds.map(tokenize_fn, batched=False, remove_columns=["text"])
print(f"Tokenized dataset size: {len(tokenized)}")

Setting labels equal to input_ids trains the model on the full sequence. If you want to compute loss only on the completion (a common best practice for instruction tuning), mask the prompt tokens with -100 in the labels.

Step 4: Loading the Model with Quantization

To fit a multi-billion-parameter model on a single GPU, load it in 4-bit precision using bitsandbytes:

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
model.config.use_cache = False

Step 5: Applying LoRA Adapters

Full fine-tuning of a 3B+ parameter model is expensive. LoRA (Low-Rank Adaptation) trains small adapter matrices on top of the frozen base weights, reducing trainable parameters by over 99% while preserving quality:

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

For code models, targeting the attention projections is usually sufficient. If you have VRAM headroom, add gate_proj, up_proj, and down_proj to the target modules for slightly better results.

Step 6: Training

Use the TRL library's SFTTrainer for supervised fine-tuning. It handles packing, padding, and the training loop for you:

from trl import SFTTrainer, SFTConfig

training_args = SFTConfig(
    output_dir="./code-lora",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
    optim="paged_adamw_8bit",
    max_seq_length=1024,
    dataset_text_field="text",
    report_to="none",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=ds,
    processing_class=tokenizer,
)

trainer.train()
trainer.save_model("./code-lora-final")
tokenizer.save_pretrained("./code-lora-final")

On a single A10G with a 3B base model, three epochs over a few thousand examples typically completes in one to three hours. Monitor the loss curve — if it plateaus after the first epoch, you can stop early.

Step 7: Merging and Saving the Adapter

For inference, you can either keep the adapter separate and load it on top of the base model, or merge the LoRA weights into the base model for a single deployable artifact:

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
merged = PeftModel.from_pretrained(base_model, "./code-lora-final")
merged = merged.merge_and_unload()
merged.save_pretrained("./code-merged")
tokenizer.save_pretrained("./code-merged")

Step 8: Evaluating the Fine-Tuned Model

Loss curves are not enough. Build a held-out evaluation set of realistic tasks drawn from your codebase — for example, functions whose implementations you removed, or bug-fix prompts based on real issues. Then compare the base model and the fine-tuned model side by side:

import torch

def generate(model, tokenizer, prompt, max_new_tokens=256):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            temperature=1.0,
            top_p=1.0,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

prompt = "<|user|>\nImplement a function that retries an HTTP request with exponential backoff.\n<|assistant|>\n"
print(generate(merged, tokenizer, prompt))

For quantitative evaluation, compute pass@k metrics if you have executable test cases, or use a stronger model as an LLM judge to score outputs on correctness, style adherence, and use of internal APIs. Track these metrics across versions of your dataset and training hyperparameters.

Best Practices

• Quality over quantity. A few thousand well-formatted, representative examples beat tens of thousands of noisy ones. Deduplicate near-identical functions and drop generated or boilerplate code.
• Respect licensing. Even for internal use, be aware of any third-party code in your repositories and whether your organization's policies permit including it in training data.
• Keep the prompt template stable. The exact delimiters and special tokens used during training must match what you use at inference. Version your template alongside the model.
• Mask the prompt in labels. For instruction tuning, compute loss only on the completion to avoid wasting capacity learning to reproduce your prompt format.
• Use a held-out evaluation set. Reserve a slice of repositories (not just random examples) for evaluation so the model is tested on code it never saw during training.
• Iterate on data, not just hyperparameters. Most quality gains come from better data: richer instructions, more diverse task types, and cleaner code. Spend more time curating than tuning learning rates.
• Watch for memorization. Code models can memorize and regurgitate training data verbatim, which may leak secrets. Scan training data for credentials and secrets before training, and audit model outputs.
• Version your datasets. Treat your training data as code: store it in version control, document its provenance, and record which dataset version produced each model checkpoint.

Conclusion

Fine-tuning a code LLM on your private repositories is a practical way to build an assistant that speaks your team's language. The pipeline is straightforward: extract clean examples from your code, format them as instruction-tuning pairs, run a LoRA fine-tune on a quantized base model, and evaluate rigorously on held-out tasks. The biggest lever for quality is the dataset itself — focused, representative, and well-formatted examples will outperform any hyperparameter tweak. Start small with a few thousand examples, measure against your real-world tasks, and iterate from there. With each cycle, the model gets closer to being a genuine collaborator on your codebase rather than a generic code generator.