The best embedding model is the one that retrieves the right passages from your documents, for your users’ real questions, within your latency and budget limits. A public leaderboard can help you build a shortlist. Only a test on your

own data can pick the winner. Below is the process we use: define requirements, cut the field to three to five candidates, run them against a small set of real queries, and check the cost and licensing before you commit.

TL;DR

  • Start with requirements (language, document length, domain, latency, hosting, budget), not with a leaderboard.
  • Use public benchmark scores on retrieval tasks to shortlist three to five models, then test them on 50 to 200 of your own real queries.
  • Hosted APIs and open-weight models trade quality, latency, privacy and maintenance. Neither wins everywhere.
  • Read the license before you fall for a model, and remember that changing models later means re-embedding everything.
  • Judge with retrieval metrics (Recall@k, MRR, nDCG@10) before any answer is generated.

Table of Contents

What an embedding model actually does in your app

An embedding is a numeric representation of text that captures its meaning. In a RAG system, the user’s query is embedded with the same model as the document chunks, the vector database finds the closest vectors, and the original text of those chunks goes to the language model as grounding data.

That makes the embedding model the gatekeeper. If the right chunk is not retrieved, the best language model in the world cannot answer from it. IBM’s RAG cookbook notes that the model you pick affects retrieval accuracy, latency and computational cost, and that larger models usually retrieve better but cost more in latency, storage and compute.

This is one decision inside a longer chain. The authors of a recent benchmarking framework make the same point: the path runs from embedding generation through indexing and search at scale, and chunking changes retrieval quality too. If you are also weighing the storage layer, our guide to choosing a vector database by workload covers that side. For how embeddings fit into the wider retrieval stack, vector search and retrieval pipelines are worth understanding as a whole.

What an embedding model actually does in your app

Write down your requirements first

Most bad choices start by opening a leaderboard before anyone has written down what the system must do. Weaviate Academy groups the requirements into four buckets. They make a good checklist:

  • Data characteristics: modality, language, domain, document length, and whether queries and documents differ in style (short question, long passage).
  • Performance needs: accuracy and recall, latency, throughput, volume, and the type of task.
  • Operational factors: hardware limits, API rate limits, and the expertise you have to deploy and maintain a model.
  • Business requirements: hosting options, license restrictions, long-term support, budget, and privacy or compliance needs.

Qdrant’s documentation boils the production side down to three numbers: throughput, latency and cost. Cost includes infrastructure plus any per-token or per-request fees for hosted models.

Language and domain

If your content is English business text, a general-purpose model is a sensible baseline. Teradata’s documentation says general models work well for standard business content and that multilingual models may give up some accuracy compared with a monolingual model for the target language. If you serve several languages, you accept that trade or run separate models per language.

Do not assume a specialist model beats a generalist in a specialist field. One academic study of electronic health records found that BGE, a small general-domain model, outperformed medical-specific models on three retrieval tasks, and cautioned that public benchmark performance may not transfer to a new domain. Test it.

Document length

Check the model’s maximum token input against your chunk size. Weaviate’s guide suggests that a 512-token limit is enough for most common RAG applications. If you embed long contracts or manuals without chunking, you need a model with a much longer window, and you should expect to pay for it in compute.

Build a shortlist of three to five models

Public leaderboards are a screening tool. The MTEB leaderboard on Hugging Face scores models across tasks such as retrieval, reranking, classification and clustering. Airbyte’s guidance is to use the retrieval-task scores to shortlist three to five candidates instead of picking by overall rank, since a model that tops the average may not top retrieval.

Two cautions that save real money:

  1. Match the training objective to the job. An August 2026 benchmarking report found that models trained for sentence similarity rather than retrieval substantially underperformed on retrieval tasks, even with competitive similarity scores.
  2. Include a general-purpose baseline. Weaviate recommends starting with a general-purpose model as a baseline, then refining by task, benchmark score, size, dimensions and max tokens.

A healthy shortlist usually has one strong hosted API model, one or two open-weight models you could run yourself, and one small, fast model as the cost floor.

Hosted API or open-weight model?

This is the decision that most often shapes the rest of the project. Here is how the two compare.

Factor Hosted API model Open-weight model you run
Retrieval quality Often highest on benchmarks Often close behind, sometimes equal
Latency Network round trip plus provider load Local inference, usually faster
Cost shape Per-token or per-request, grows with volume Fixed GPU or CPU infrastructure
Privacy Text leaves your environment Text stays in your environment
Maintenance Provider handles it, but models can be retired You handle serving, scaling and upgrades
Licensing Provider terms of service Model license, which may carry restrictions

The numbers from recent academic benchmarks show why it is a real trade, not a clear winner. In one August 2026 report, the top hosted model reached an average nDCG@10 of 0.638, but at roughly 7 to 14 times the latency of open-source alternatives, plus per-query API costs. The strongest open-source alternative came within 0.09 nDCG points at a fraction of the latency and with no per-query cost, and the authors suggested it as a default when requirements are unspecified.

A separate benchmark of a hosted embedding model against open-source models found the hosted one ranked first on every task, but at a 231.6 ms median latency it was about 14 times slower than the fastest local models. On Italian text, an open model landed within 0.003 nDCG of it at 31 ms, which makes it the better pick when you have a sub-100 ms target.

So ask two questions. Can your text leave your environment? And does your user-facing latency budget leave room for a network call? If the answer to either is no, open-weight models move to the front. Our local vs cloud AI decision guide walks through that choice for the business as a whole, and reducing LLM latency in production apps covers the speed side.

Check the license before you fall for a model

For open-weight models, the first legal question is usually the license itself. Law firm Cooley’s August 2026 note points out that open-weight licenses are increasingly customized and can include commercial-use limits, volume restrictions, copyleft provisions, and bans on using outputs to train competing models.

Before deploying, confirm that the license permits your intended use, and check what it says about redistribution, fine-tuning, attribution and downstream deployment. Also look at where the weights came from, and plan for the possibility that licensing or availability changes. If your counsel has not seen the license, that is the next step, not a code review.

Test your shortlist on your own documents

This is the step that settles it. Benchmark scores average across many datasets that are not yours, and the gap between a leaderboard and your corpus can be wide. The process is short:

  1. Collect real queries. Pull 50 to 200 from search logs, support tickets or chat history. MongoDB’s engineers suggest a test set of at least 50 samples with common questions, edge cases and adversarial inputs.
  2. Label the right answers. For each query, mark which documents or chunks should come back.
  3. Run every shortlisted model on the same queries, with the same chunking, the same index settings and the same top-k.
  4. Compare metrics, latency and cost side by side.
  5. A/B test the top two or three with live traffic before a full rollout, as Airbyte recommends.

Do not generate your test from your chunks

Vercel’s production guide warns that an evaluation set made by generating questions from the chunks you retrieve is misleading, because those questions sit unnaturally close to their source. Real logs and support tickets are harder, and that is the honest test.

Include the hard questions

A test of single-document lookups can pass a system that falls apart when someone asks a question that spans two policies. Santiago Company’s analysis argues the labeled set should include multi-hop questions and real contradictions from your highest-stakes documents, and that you should rerun the same set on every embedding model or reranker change. We cover turning this into an automated gate in our piece on RAG evaluation datasets and CI gates.

Which metrics to trust

Measure the retriever on its own, before any answer is generated. If you only score final answers, you cannot tell a retrieval miss from a generation error.

  • Recall@k: the share of relevant items found in the top k results.
  • MRR: how high the first relevant result appears.
  • nDCG@10: handles graded relevance and rewards putting the best results first.

Which to lead with depends on your context size. NVIDIA’s engineers recommend Recall as the primary metric when retrieved context is up to 4K tokens, since the language model can use a relevant chunk even if it is not ranked first. They point to nDCG when chunks are larger, contexts are big, or the app shows chunks as citations.

Whatever you choose, fix the corpus snapshot, chunking version and index settings so a rerun is a fair comparison.

Estimate the real cost before you commit

Per-token pricing is easy to compare and usually not the expensive part. Vercel’s guide notes that the token bill is typically the smallest line in the estimate: engineering time, index rebuild windows and cutover risk cost more.

Two things lock you in:

  • Dimension count. It is set at first ingest. Changing it means regenerating and reloading every vector. A model that supports Matryoshka-style truncation keeps your options open as the corpus grows.
  • Model switching. Embeddings from different models are not interchangeable, so a switch means re-embedding the entire corpus.

Build a simple estimate: total tokens in your corpus, times the price per token (or the GPU hours for a self-hosted model), plus query volume, plus storage for the vectors at your chosen dimension. Then add the cost of one full re-embed, because you will probably do one. If you run models yourself, quantization can shrink memory needs, and our self-hosted LLM decision framework helps you size the infrastructure.

Our perspective

Teams usually come to us after the prototype works and production does not. The demo used a default embedding model, ten tidy documents and a friendly question. Then real users arrive with vague, misspelled, multi-part questions, and retrieval quietly starts missing.

Our view is simple: treat the embedding model as a swappable component and treat your evaluation set as the asset. When the labeled queries live in your repository and run on every change, switching models, adding a reranker or moving from a hosted API to a local model becomes a measured decision instead of a gamble.

We also lean toward simplicity. A small model that you can run locally, keep private and serve fast is often the right answer, and it keeps token costs and complexity down. Pair that with a senior engineer who reviews the pipeline end to end, and you cross the last mile from prototype to something you can maintain.

Chad

Get a second pair of eyes on your retrieval stack

If you would rather not run this bake-off alone, we can help. Our team builds and reviews AI pipelines, including local AI model development, and we audit AI-generated and vibe-coded apps that stall on the way to production. A Senior Developer Review starts from $449, and you can see everything we offer on our services page.

Bowtie

Bring your corpus, your sample queries and your constraints, and we will help you pick, test and ship the model that fits.

FAQ

How many models should I compare?

Three to five is a practical shortlist, per Airbyte’s guidance. Test them on the same queries, then A/B test the top two or three with real traffic before rollout.

Is the top model on the leaderboard the best choice for me?

Not necessarily. Leaderboards average across many tasks and datasets. Use the retrieval scores to shortlist, then measure on your own documents, since public results may not transfer to a new domain.

Can I switch embedding models later?

Yes, but it is a project. Vectors from different models are not compatible, so you regenerate and reload every vector, and dimension count is fixed at first ingest. Keep your evaluation set so you can verify any switch.

Should I use an open-weight model to keep data private?

Running an open-weight model in your own environment keeps text from leaving it, and recent benchmarks show open models can come close to hosted quality at lower latency. Review the license first, since terms can restrict commercial use.

Sources