Self-Hosting LLMs: Choosing Between Ollama and vLLM

For most AI applications, calling a commercial API directly is the right choice: it is convenient, requires no operations work, and provides access to the strongest models. However, some scenarios require you to self-host an open-source model:

• Data compliance: Healthcare, financial, and internal data cannot leave your infrastructure.
• Cost: Usage is high enough that the API bill is no longer economical.
• Offline operation/low latency: The application runs at the edge, on an intranet, or has strict latency requirements.
• Customization: You need fine-tuning, control over inference parameters, or support for a specialized model.

This article covers the two primary approaches to self-hosting: Ollama for development and personal use, and vLLM for production, along with the unavoidable questions of quantization and hardware.

1. First Decide Whether You Actually Need to Self-Host

Self-hosting carries significant hidden costs, including GPUs, operations, model upgrades, and availability. Do the math before deciding:

Rule of thumb: if usage is modest and there are no compliance requirements, use an API. Self-hosting becomes economical only when either "API usage is prohibitively expensive at your volume" or "the data cannot leave your environment" is true. Many people self-host to "save money," only to discover that operations cost far more than the API fees they avoided. Do not make that mistake.

2. Ollama: The Best Choice for Local Development

If you want to run a model on your own machine for development, experiments, or offline tools, Ollama is almost a no-brainer. It packages model downloads, quantization, loading, and inference into an experience similar to Docker:

ollama pull qwen3:8b
ollama run qwen3:8b
# Or start an OpenAI-compatible service
ollama serve   # Default port 11434, OpenAI-compatible API

Key characteristics:

• Minimal setup: One command downloads a model, and another starts the service. No configuration is required.
• Automatic quantization: It downloads GGUF-quantized models by default, allowing consumer Macs and PCs to run models from 7B to 32B.
• OpenAI compatibility: Point baseURL at Ollama, and code written for OpenAI can run directly.
• Rich ecosystem: Mainstream open-source models such as Qwen, Llama, Mistral, and DeepSeek are available with one-command downloads.

Ollama is suitable for local development, personal tools, prototypes, and offline scenarios. It is not suitable for high-concurrency production services because it is not optimized for throughput, and latency rises sharply as concurrency increases.

3. vLLM: Production-Grade, High-Throughput Inference

When you need a real service for a team or product with concurrent users, vLLM is currently the most widely used open-source inference framework.

Its core advantage is PagedAttention + continuous batching. It manages attention data in pages and dynamically batches multiple requests, producing far better GPU utilization and throughput than naive inference. On the same GPU, vLLM can deliver several times to an order of magnitude more throughput than Ollama.

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --tensor-parallel-size 1   # Single GPU

This starts an OpenAI-compatible inference service, so your application code requires almost no changes beyond using a different baseURL.

Use cases:

• Production API services: Multiple internal applications share one model service.
• High concurrency: vLLM's batching keeps concurrent request latency under control.
• Customization requirements: Control sampling parameters, load LoRA adapters, or run specific models.

The tradeoff is that deployment is more involved than with Ollama: you must manage the Python environment, GPU drivers, VRAM, and version compatibility, which requires operational ownership. vLLM is a "service"; Ollama is a "tool". Choose according to your scale.

4. Quantization Is Unavoidable

The original weights of open-source models are enormous. A 70B model in fp16 requires 140GB of VRAM, which is beyond the capacity of consumer hardware and most single GPUs. This makes quantization necessary: reducing weights from 16 bits to 4–8 bits cuts VRAM usage by at least half, usually with an acceptable loss in quality.

Mainstream quantization formats:

• GGUF: Used by llama.cpp and Ollama. It supports mixed CPU/GPU execution, is flexible, and works well locally.
• AWQ / GPTQ: Supported by vLLM and Transformers. These GPU-focused quantization formats are suitable for serving.
• fp8: Supported natively by newer hardware such as the H100 and 4090, with nearly lossless quality.

Practical guidance:

• A 7B model quantized to 4-bit requires about 4–5GB of VRAM and can run on consumer GPUs such as the 4060 or 4090.
• A 70B model at 4-bit still requires approximately 40GB of VRAM, so you need an A100, H100, or multiple GPUs.
• Prefer a model already quantized by the community, whether by the official project or publishers such as TheBloke. Quantizing models yourself is advanced territory.

5. Hardware Selection: VRAM Comes First

The primary constraint when running models is VRAM, not compute performance. A rough estimate is:

VRAM ≈ model weights + KV cache + activations
weights ≈ parameter count × bytes per parameter (4-bit is about 0.5 bytes per parameter)

GPU selection guide:

Watch the KV cache: with long contexts or high concurrency, the KV cache may consume more VRAM than the weights themselves. This is exactly what vLLM's PagedAttention optimizes. Do not estimate capacity based only on model size.

6. Operational Blind Spots in Self-Hosting

Self-hosting is not finished once the model runs. Several commonly overlooked concerns remain:

• Availability: If the model service goes down, your application goes down. You need health checks, automatic restarts, and monitoring.
• Model upgrades: Open-source models evolve quickly, and upgrades require testing because output quality may change.
• Cost monitoring: Cloud GPUs are billed by time and continue costing money while idle. Can you scale down when there is no traffic?
• Security: Does the model service require authentication? Do not leave it exposed even on an intranet; if someone scans and finds it, your infrastructure becomes free compute for them.

Summary

There are two clear paths for self-hosting large language models:

• Local/personal/development → Ollama: Minimal setup, automatic quantization, and OpenAI compatibility.
• Production/high concurrency → vLLM: High throughput through PagedAttention, extensive customization, and an operational burden.

For the decision itself, remember: self-host only after confirming that either "API usage is prohibitively expensive at your volume" or "the data must remain under your control" is true. Otherwise, self-hosting for its own sake will let operational costs erase the benefits. The value of open-source models is not simply that they are "cheaper," but that they provide "data ownership + customization + censorship resistance." When those properties matter enough, Ollama or vLLM, quantization, and the right GPU are the answer.