Selecting and Evaluating Embedding Models

Every step in the RAG retrieval pipeline matters, but the embedding model sets the ceiling. Even the best chunking and reranking cannot rescue an embedding model that encodes similar content as vectors far apart. Unfortunately, many people pick one at random, then blame reranking or chunking when performance disappoints, unaware that the embedding model is the root cause.

This article explains how to choose the right embedding model.

1. Four Selection Dimensions

When choosing an embedding model, consider four things:

1. Language support. What language is your content in?

• English only: there are many options (OpenAI, Cohere, BGE-en).
• Chinese only: choose a model optimized for Chinese (BGE-zh, m3e, or Chinese models from Tongyi/BAAI).
• Mixed Chinese and English: choose a multilingual model (BGE-m3, Cohere multilingual, Jina multilingual). Do not force Chinese through an English-only model; recall will collapse.

This is the most common pitfall: using an English embedding model for Chinese produces dramatically worse results. Language matching is the first principle.

2. Dimensionality. The embedding output dimension (768 / 1024 / 1536 / 3072) affects:

• Storage and retrieval costs: the higher the dimension, the larger the vectors and the more expensive the storage and computation.
• Representational capacity: higher dimensions generally capture finer distinctions, but with diminishing returns.
• 1536 (OpenAI small) is a common sweet spot; 3072 is unnecessary for most use cases.

3. Domain fit. General-purpose embeddings may underperform in specialized domains such as medicine, law, and code:

• Code: choose a code-friendly model, such as voyage-code or a Jina version optimized for code.
• Specialized domains: when a general-purpose model is insufficient, consider fine-tuning or a domain-specific embedding model.

4. Cost and deployment.

• Commercial APIs (OpenAI/Cohere/Voyage): easy to use and billed by token, but they involve cross-border data transfers and dependence on a third party.
• Self-hosted open-source models (BGE/Jina/m3e): free and keep your data under your control, but you must operate the service yourself and bear the GPU costs.

2. A Rough Map of Mainstream Models (2026)

Consult MTEB (Massive Text Embedding Benchmark), the ImageNet of the embedding world. But...

3. Most Important: Evaluate on Your Own Data

MTEB rankings do not equal performance in your use case. A high score on a public benchmark does not guarantee strong results for your domain, language, or query distribution. The final step in model selection must therefore be: test it on your own data.

A minimal evaluation process:

1. Build a retrieval evaluation set: 50–200 query → relevant document ID examples.
2. Run every candidate embedding: index the corpus with that embedding, then run the evaluation set against it.
3. Compute metrics: Recall@k (the fraction of relevant results found in the top k), MRR (mean reciprocal rank), and nDCG.
4. Choose the best overall performer, not the model at the top of a leaderboard.

# Pseudocode
for model in [bge_m3, openai_3small, cohere_v3]:
    embeddings = [model.embed(d) for d in docs]
    for q, relevant in eval_set:
        hits = vector_search(model.embed(q), embeddings, k=5)
        recall += len(set(hits) & relevant) / len(relevant)
    print(model, recall / len(eval_set))

You cannot skip this step. I have seen far too many teams choose the "top-scoring" model on MTEB, only to have its recall in their Chinese long-document use case crushed by a lower-ranked Chinese model. Your data distribution has the final say.

4. Several Underestimated Considerations

1. Maximum input length. Embedding models have input limits (512 / 8192 tokens). If your chunks exceed the limit, you must either truncate them and lose information or switch models. For long-document use cases, choose a model with a longer input limit (Jina and BGE-M3 support 8K+).

2. Instruction prefixes. Some models support instructions, such as BGE's "Represent this sentence for retrieving relevant documents:". Using different instructions for the query and document can significantly improve recall. Do not overlook this feature.

3. Normalization. Most models require normalized vectors when calculating cosine similarity. Normalize both indexed vectors and query vectors. If you normalize one side but not the other, the similarity scores will be wrong.

4. Quantization. When storage is constrained, vector quantization (float32 → int8) can reduce storage by a factor of 4, usually with an acceptable quality loss. Both pgvector and Qdrant support it.

5. Should You Fine-Tune an Embedding Model?

When a general-purpose embedding model is insufficient, you can fine-tune it with domain data. However, fine-tuning an embedding model has prerequisites:

• You need "positive and negative sample pairs" (relevant/irrelevant document pairs).
• You need training infrastructure.
• You need evaluation and validation.

Start with a domain-specific model (such as voyage-code for code), and fine-tune only if it is still insufficient. A domain-specific model is already adequate for most use cases, so custom training is unnecessary.

6. Pin the Version and Do Not Switch Casually

After selecting an embedding model, pin it for the long term. If you switch models and the embedding dimension changes, every historical vector must be recomputed, requiring a complete reindex. That is expensive. Spend more time evaluating during selection, then do not switch lightly.

Summary

The embedding model is the foundation of RAG. Model selection has four steps:

1. Match the language and domain: use a Chinese model for Chinese and a code model for code. Do not force a general-purpose English model into the wrong use case.
2. Balance dimensionality and cost: 1536 is a sweet spot; 3072 is unnecessary for most use cases.
3. Use MTEB as a reference, but test on your own data: leaderboards do not represent your use case. You must compare Recall/MRR on your own evaluation set.
4. Pin the version: switching embedding models means rebuilding the entire index, so proceed carefully.

Remember this: the embedding model determines RAG's ceiling. Spending an extra week evaluating embedding models is more valuable than spending a month later tuning reranking and prompts. Do not cut corners here.