Hands-On Introduction to LoRA/QLoRA Fine-Tuning

Want an open-source model to perform better on your task? Fine-tuning is the ultimate tool. But "fine-tuning" sounds as though it requires dozens of A100s and is prohibitively expensive—that is full-parameter fine-tuning. This article covers LoRA/QLoRA (parameter-efficient fine-tuning), which lowers the barrier to a single consumer-grade GPU. It is the practical path to fine-tuning for individuals and small teams.

1. First, a Reality Check: Make Sure Fine-Tuning Is the Best Option

Before discussing how to fine-tune, you must first ask yourself: Does this problem really require fine-tuning?

As discussed previously in “Fine-Tuning vs. RAG vs. Prompting,” use RAG for knowledge-related problems and consider fine-tuning only for behavior or style problems. Fine-tuning is for “consistent, specific behavior or style,” not for “making a model smarter” or “teaching it new knowledge.”

Many people who want to fine-tune are actually trying to solve a problem that RAG or prompting could handle. First confirm: Have you pushed prompting as far as possible? Is RAG sufficient? Do you have enough training data? Fine-tune only if the answer is still no after checking all three. Otherwise, a model you spend a week training may perform worse than someone else's well-crafted prompt.

Once you have confirmed that fine-tuning is necessary, keep reading.

2. How LoRA Works: Train a Small Subset, Not Everything

Full-parameter fine-tuning updates every parameter in the model. For a 70B model, that means training 70 billion parameters, which requires enormous amounts of VRAM. The key insight behind LoRA (Low-Rank Adaptation) is that the “change” in model weights is low-rank, so it can be approximated by the product of two small matrices.

原始权重 W（冻结,不训）
变化量 ΔW ≈ A × B   (A: d×r, B: r×d, r 很小,如 8)

During training, only A and B are trained, reducing the parameter count from d×d to 2×d×r—several orders of magnitude smaller—while W remains frozen. At inference time, W' = W + A×B.

Benefits:

• Dramatically lower VRAM usage: You do not need to store gradients and optimizer states for every parameter.
• Fast: Fewer trainable parameters mean faster iterations.
• Interchangeable: A LoRA adapter is small and can be saved, loaded, or switched, allowing one base model to use multiple LoRA adapters for different tasks.

The tradeoff is slightly less expressive capacity than full-parameter fine-tuning because low-rank approximation is lossy, but it is sufficient for most tasks.

3. QLoRA: Quantize the Base Model, Then Train LoRA

LoRA already saves resources, but the base model is still loaded in fp16. A 70B model requires 140GB of VRAM and cannot fit on a single card. QLoRA means loading the base model quantized to 4-bit (NF4) and backpropagating only through the LoRA parameters.

This allows a 70B model to be fine-tuned on a single GPU with 48GB of VRAM, while 7B and 13B models can be trained on a consumer-grade 24GB card such as the 4090. QLoRA is now the de facto standard for “single-GPU fine-tuning.”

In terms of accuracy, models trained with QLoRA approach the quality of full-parameter fine-tuning, as demonstrated by the paper and extensive practical use, making it exceptionally cost-effective.

4. Complete Workflow

1. Data preparation. Fine-tuning requires “instruction-response” pairs, usually in this format:

{"instruction": "把下面句子翻译成正式中文", "input": "hello world", "output": "你好，世界"}

Key points:

• Quality > quantity. One thousand high-quality samples are better than 100,000 noisy ones. The LIMA paper fine-tuned a strong model with only 1,000 samples.
• Cover the task distribution. The data should resemble the input distribution the model will encounter during real inference. Do not fine-tune a customer-service model on Wikipedia.
• Deduplicate and clean. Duplicate or contradictory samples will undermine training.

2. Training. Choose one tool:

• PEFT (Hugging Face): The official and most general-purpose library for LoRA/QLoRA.
• LLaMA-Factory: A beginner-friendly, low-code tool for configuration-based fine-tuning.
• Unsloth: Highly optimized, 2–5 times faster, and uses over 50% less VRAM. Recommended.

Get started with LLaMA-Factory using one command:

llamafactory-cli train examples/train_qlora/llama3_lora.yaml

In the configuration, specify the base model, dataset, LoRA rank—commonly 8 or 16—learning rate, and number of epochs, where 2–3 is usually sufficient.

3. Merge or mount. Training produces a LoRA adapter that is only a few hundred MB. There are two ways to use it:

• Merge it into the base model: base + LoRA → 完整模型, which simplifies deployment by producing one model file.
• Mount it at runtime: Inference frameworks such as vLLM support dynamically loading LoRA adapters, allowing one base model to serve multiple adapters and conserve resources.

4. Deployment + evaluation. Use vLLM for deployment; see the dedicated article. Evaluation is critical—compare the fine-tuned model with the original base model on an evaluation set for the target task to confirm that it has actually improved. Without evaluation, fine-tuning may produce a model that performs worse than the original.

5. Hyperparameters and Pitfalls

• rank (r): 8 or 16 is the default sweet spot. More complex tasks may benefit from a larger value such as 64, but returns diminish and the risk of overfitting increases.
• epochs: 2–3 is usually enough. Instruction-tuning datasets are small, and too many epochs can cause overfitting, with the model memorizing samples.
• Learning rate: LoRA uses a higher learning rate than full-parameter fine-tuning, typically around 1e-4. The tool's default is usually sufficient.
• Catastrophic forgetting: Excessive fine-tuning can cause the model to forget general capabilities, making it more specialized but less capable overall. Mitigations include mixing in general-purpose data, limiting the number of epochs, and using a lower rank.

6. When to Use LoRA Fine-Tuning

Returning to the original decision, LoRA is genuinely useful in these scenarios:

• Consistent output formats or styles: You need the model to follow a specific structure or tone reliably, but prompting is inconsistent.
• Domain-specific behavior: Customer service, legal, medical, and other domains have specialized task patterns.
• Making a small model approach a large one: On a narrow task, a fine-tuned 7B model can outperform an untuned 70B model while dramatically reducing costs.
• Local deployment or privacy: You must self-host and also need customization.

It is not suitable for knowledge updates, where you should use RAG; tasks requiring stronger reasoning, where you should switch to a larger model; or situations without enough data, where you should collect data first.

Summary

LoRA/QLoRA makes “fine-tuning” accessible on a single GPU instead of restricting it to large companies:

• LoRA: Trains low-rank adapters, reducing VRAM usage and enabling interchangeable adapters.
• QLoRA: Combines 4-bit base-model quantization with LoRA, enabling single-GPU training for models from 7B to 70B.
• Workflow: High-quality data → PEFT/Unsloth training → merging/mounting → comparative evaluation.

But remember the point emphasized throughout this article: fine-tuning is a powerful tool for specific scenarios, not the default choice. Use LoRA only after confirming that prompting and RAG cannot solve the problem. When it is truly appropriate—for consistent behavior or style, domain customization, or accelerating smaller models—it becomes a core way to differentiate products built with open-source models. That is precisely why it deserves closer study in the AI coding community.