Tuning a model locally means taking an open-weights model, continuing its training on your own examples, and doing all of it on hardware you control. It is realistic today: with LoRA or QLoRA, a 7B or 8B model can be

fine-tuned on a single consumer GPU. The harder question is whether you should. Most teams that ask us about it need better prompts or retrieval first. A few really do need tuning, and for them the work is mostly about data and testing, not GPUs.

This guide walks through both halves: the decision, then the workflow.

TL;DR

  • Try prompt engineering, few-shot examples and retrieval (RAG) first. Fine-tune only when the model must learn a skill, style or output format that prompting can’t deliver.
  • LoRA and QLoRA make local tuning practical: roughly 6 to 8 GB of VRAM for a 7B model, against about 60 GB for full fine-tuning.
  • Fifty carefully curated examples usually beat thousands of sloppy ones. Data quality is the project.
  • Hosted models such as Claude and GPT can’t be tuned on your own machines. Local tuning means open-weights models like Llama or Gemma.
  • Judge the result on a frozen evaluation set and behavior you care about, not on training loss.

Table of Contents

First, decide whether to tune at all

Fine-tuning has a reputation as the serious, grown-up way to customize AI. In practice it is often the most expensive way to fix a problem you could have fixed in an afternoon.

The Fine-Tuning Local LLMs guide at lllm.dev lists the same ladder we use with clients. Prompt engineering is often enough for simple tasks. Retrieval-augmented generation, where the model looks up your documents at question time, covers most gaps in domain knowledge. Fine-tuning is for when the model has to adopt a new fundamental skill, a distinct writing style or highly specialized behavior.

A quick way to sort your situation:

Your problem Start with Tune?
The model doesn’t know our product docs or policies RAG Rarely
Answers are right but the tone is off Better system prompt, few-shot examples Maybe, if prompts fail
Output must follow a strict JSON or template format Structured output prompts and validation Often yes, if failure rate stays high
Needs a narrow skill (classifying tickets in our taxonomy, say) Few-shot prompting, then a small tuned model Yes, a good candidate
Data can’t leave our network Local open model plus RAG Only if behavior still falls short

Notice the last row. Privacy is a reason to run a model locally. It is not, by itself, a reason to tune it. Plenty of private deployments run an untuned open model with retrieval over internal documents, and that is a simpler system to maintain.

The cost of skipping this step

Tuning adds data collection, labeling, training runs, evaluation and a model you now own and must re-tune when the base model moves on. If a better prompt gets you 90% of the way, that extra machinery is overhead. Anthropic’s guidance on choosing a model makes a related point for hosted models: adjusting settings and refining prompts is often a better lever for balancing intelligence, latency and cost than customization.

What “local” rules out: Claude, GPT and other hosted models

People search for this topic usually after asking whether they can tune a Claude or OpenAI model on their own server. You can’t. Anthropic’s glossary defines fine-tuning as further training a pretrained model on additional data, and hosted proprietary models don’t run on your hardware. OpenAI’s own model optimization docs go further and say the company is winding down its fine-tuning platform, so check the current status before you plan around it.

If data must stay on controlled hardware, the practical route is an open-weights model such as Llama or Gemma. Google’s developer forum makes the same point for Gemini: use Gemma models with a runner like Ollama, vLLM or llama.cpp, and expect to need a capable GPU.

What hardware you actually need

What hardware you actually need — overview diagram

The big change in the last couple of years is parameter-efficient tuning. Instead of updating every weight in the model, LoRA trains small adapter layers on top of a frozen base. QLoRA does the same on a quantized (4-bit) base to save more memory.

For a 7B model, lllm.dev’s comparison puts it like this:

Method VRAM (7B model) Rough training time
LoRA ~8 GB ~1 hour
QLoRA ~6 GB ~1.5 hours
Full fine-tuning ~60 GB ~8 hours

Those are ballpark figures for a small dataset, not promises. Your sequence length, batch size and dataset size move them a lot.

A recent write-up on Unsloth cites the library’s published QLoRA minimums: about 3.5 GB of VRAM for a 3B model, 5 GB for 7B, 6 GB for 8B and 41 GB for 70B. That is why an 8B tune fits on a mid-range gaming card, while a 70B model needs a workstation-class or multi-GPU setup. AMD publishes its own Unsloth fine-tuning playbook if you are working on AMD hardware.

The usual toolchain

  • Training: Unsloth for speed and low memory use, or Hugging Face Transformers and AutoTrain for broader control.
  • Inference after training: llama.cpp or Ollama for single machines. For team-scale serving, a server such as vLLM is the usual step up. We cover that path in 3 first moves to deploy LLMs on Kubernetes with vLLM.

The tool list comes from Google’s Gemma documentation, which names these frameworks for fine-tuning and running models on local machines.

The workflow, step by step

1. Write down the behavior you want

Not “make it better at legal.” Something testable: “extract these eight fields from a vendor contract and return valid JSON,” or “answer in our support voice and never promise refunds.” If you can’t state the target in a sentence, you can’t evaluate it, and you will not know when to stop.

2. Build a small, careful dataset

This is where projects are won or lost. The lllm.dev guide is blunt: 50 high-quality curated examples significantly outperform 5,000 mediocre or automated ones. Founders often spend weeks hunting for a massive dataset when a few dozen reviewed examples would do more.

Practical habits:

  • Have a person who knows the domain write or approve every example.
  • Cover the awkward cases, not just the easy ones: refusals, ambiguous input, messy formatting.
  • Keep a consistent format, because the model will copy your formatting quirks.
  • Strip personal data you don’t need. Local training protects data from third parties, but it doesn’t make sensitive data a good idea to scatter through training files.

3. Split the data without leaking

Before training, set aside an evaluation set and never touch it. The subtle part is how you split. The Local LLMs reproducibility guide by Liangchao Deng warns that you must split by the true unit of generalization, such as source document or user session. If near-duplicates of a test example sit in the training data, the model memorizes the answer and your numbers look great right up until real users arrive.

A simple example: you are tuning a ticket classifier on 2,000 support threads, and each thread has five messages. Splitting by message puts pieces of the same thread on both sides. Split by thread.

Message-level versus thread-level data splitting

4. Start small and run a baseline

Before tuning, run the untuned model on your evaluation set with your best prompt. That is your baseline. If the tuned model doesn’t clearly beat it, you have your answer, and it cost you an afternoon instead of a quarter.

Then pick a modest base model (7B or 8B), use QLoRA, and run a short job. Fast iteration beats a single heroic run.

5. Evaluate behavior, not loss

Training loss going down tells you the model is fitting the data. It doesn’t tell you the model is useful. The same reproducibility guide recommends a separate, unchanged evaluation set and behavioral metrics: structured output validity, retrieval faithfulness and blinded human preference tests, rather than leaning only on perplexity or loss.

For most business tasks that translates to three checks:

  1. Format: what percentage of outputs parse correctly?
  2. Accuracy: on held-out examples with known answers, how often is it right?
  3. Preference: show reviewers outputs from the baseline and the tuned model without labels. Which do they pick?

6. Export, serve and monitor

A tuned adapter is only useful once it is running somewhere your application can call it. Merge or load the adapter, convert to the format your runtime expects, and put it behind an API with logging. Then keep scoring real traffic against your evaluation set. Models drift in usefulness as your business changes, and a new base model release can leave your tune behind.

Security notes for sensitive environments

If you chose local specifically because data is sensitive, a few habits matter more than the tuning method.

  • Verify the weights. In air-gapped setups, download model files outside the secure environment and check their SHA-256 hashes before transferring them in. A Google AI developer forum thread on non-cloud pipelines describes this pattern.
  • Mind residual memory. The same thread suggests scripts that explicitly flush RAM and swap after processing sensitive batches. Treat that as one input to your own review, not a compliance guarantee. If you’re under HIPAA, CMMC or a similar regime, have your compliance lead sign off on the design.
  • Treat training data as a data store. It needs the same access controls as the source records.

Our perspective: tune last, and tune small

We see the same pattern with founders and CTOs. Someone has watched a few videos, concluded that “our own model” is the finish line, and budgeted for GPUs before writing a single evaluation case. That is the prototype trap in a new costume: a flashy demo that never gets across the finish line to something the team can maintain.

Our approach is deliberately unglamorous. We start with the cheapest lever that could work, a tighter prompt or retrieval over your documents, and we measure it. Only when that plateaus do we talk about tuning, and then we tune the smallest model that clears the bar, because smaller models mean lower serving costs and fewer moving parts to maintain after launch. Keeping token costs and complexity down is how we work generally.

Local models earn their place when privacy, compliance, offline operation or predictable high-volume costs are the real constraint. If you’re weighing volume, our self-hosted LLM decision framework lays out the questions we ask. And if the code around your model was generated by AI tools, review it before you build on it. A tuned model behind a fragile, unreviewed application is not production-ready.

— Chad

How we can help

Bowtie builds and deploys local AI models for teams that need data to stay in their own environment, and our engineers help you decide whether tuning is justified before you spend on it. We don’t publish a price for local AI model development, because the scope depends on your data, hardware and target behavior. You can see what we do on our services page.

If you already have an AI-built prototype and aren’t sure it will hold up, our Vibe Check is a 449 $ one-off review. It is a fast way to learn what needs fixing before you add a custom model to the stack.

FAQ

Can I fine-tune a Claude or OpenAI model locally?

No. Hosted proprietary models don’t run on your hardware, so you can’t tune them locally. OpenAI’s documentation also says it is winding down its fine-tuning platform. If you need local tuning, use an open-weights model such as Llama or Gemma.

How much GPU memory do I need to fine-tune a local LLM?

For a 7B model, QLoRA needs roughly 6 GB of VRAM and LoRA about 8 GB, according to lllm.dev. Full fine-tuning of the same model needs around 60 GB. Larger models, like 70B, need far more even with QLoRA.

How many training examples do I need?

Fewer than most people expect, if they are good. Fifty well-curated examples can outperform thousands of automated, low-quality ones. Start small, evaluate against a held-out set, and add examples where the model fails.

Should I use RAG or fine-tuning?

Use RAG when the model lacks knowledge, such as your documents or policies. Use fine-tuning when it lacks a behavior, such as a style, format or narrow skill. Many production systems use RAG alone, and some combine both.

Sources