rag-retriever-bench

Three steps

1. rag-retriever-bench prepare Download the dataset, sample the corpus, embed it once (cached).
2. rag-retriever-bench run Benchmark all 9 configurations against the same queries.
3. results/*.md Compare the leaderboard: quality, latency, ingestion and self_check evidence per backend.

Who it’s for

• AI engineers picking a vector database with measured numbers instead of vendor benchmarks
• RAG & search developers verifying the index actually works before touching a single prompt
• AI platform teams standardizing one benchmark across the backends they operate
• LLM researchers who need reproducible retrieval baselines with complete ground truth

When to reach for it

• Before touching prompts: rule the retriever out first, so you stop debugging the wrong layer
• Before switching vector databases: decide with your own measured numbers instead of vendor benchmarks
• Before production deployment: verify with self_check that the index is not being silently bypassed
• When search feels off but nothing errors: the three silent-degradation cases below are exactly that

Install

git clone https://github.com/kenimo49/rag-retriever-bench
cd rag-retriever-bench
pip install -e ".[all]"

docker compose up -d
cp .env.example .env   # OPENAI_API_KEY (embeddings only)

Server backends (pgvector, ClickHouse, Qdrant, Weaviate, Milvus) run via docker compose; Chroma and LanceDB are embedded and need no services.

What it measures

• Retrieval quality: recall@k, hit@k, MRR@k, nDCG@k against human-annotated relevance judgments, no LLM judge
• Query latency: client-side p50 / p95 / p99 / mean per query, serialization included
• Ingestion: bulk-load seconds and index-build seconds per backend
• 7 backends, 9 configurations: pgvector, ClickHouse (HNSW ×2 + brute force), Qdrant, Weaviate, Milvus, Chroma, LanceDB
• A self_check per backend: an EXPLAIN or server-statistics probe that verifies the ANN index was actually used
• HNSW parameters aligned across backends (m=16, ef_construction=64, ef_search=100), so differences reflect the engine, not the tuning
• MIRACL-ja dataset: the sampled corpus always contains every positive passage, so recall is measured against complete ground truth

Measured results (100k passages)

MIRACL-ja, 100,000 passages · 860 queries · text-embedding-3-small · top_k=10 · measured 2026-07-11 on a single node.

Server backends (network hop included)

Embedded backends (in-process; latency not comparable to server backends)

Brute force sets the quality ceiling at recall 0.952; the closest ANN configuration is Qdrant at 0.947. At this scale the HNSW engines differ mainly in latency: Weaviate and Milvus answer under 2.5 ms at p95, while ClickHouse pays 13 to 52 ms depending on index granularity.

Full report (MRR, hit@k, ingestion, self_check evidence) →

How much one choice moves the numbers

Same corpus, same embeddings, same HNSW budget. Only the decision below changed.

Usage

# 10k-passage smoke run (MIRACL-ja downloads on first use)
rag-retriever-bench run -c configs/miracl-ja.yaml --corpus-size 10000

# full 100k run
rag-retriever-bench run -c configs/miracl-ja.yaml

# prepare only: download dataset, sample corpus, embed
rag-retriever-bench prepare -c configs/miracl-ja.yaml

What it caught

Three of the nine configurations shipped a way to silently degrade, with zero errors raised. The self_check probe caught all three during development:

• ClickHouse HNSW: skip the OPTIMIZE FINAL step and queries quietly fall back to brute force. The EXPLAIN probe caught it before it could surface as a "ClickHouse is slow" result.
• Qdrant: with indexing_threshold_kb above the corpus size the index is never built; the collection reports indexed_vectors_count = 0 and searches run on the raw segment.
• Milvus: after a flush it could load a stale snapshot missing sealed segments, returning partial results with no error.

Every one of those degradation paths produced plausible-looking numbers. Reports land in results/ as Markdown + JSONL, one row per backend with quality, latency, ingestion, and the index-verification evidence.

Numbers are MIRACL-ja + text-embedding-3-small on a single node. Embedded and server backends are reported in separate tables because their latency is not directly comparable. Measure your own data before deciding.

Why this tool exists

Most RAG teams optimize prompts before verifying retrieval quality. When an answer comes back wrong, the model takes the blame and the prompt gets rewritten, while the retriever that returned the wrong passages is never questioned. This harness exists because retrieval errors are routinely mistaken for LLM failures. Three of the nine configurations above could degrade with zero errors raised; nothing in the answers would have pointed at the index. Measure retrieval first.

Alongside RAGAS and DeepEval

Complementary layers: RAGAS and DeepEval score the contexts and answers your pipeline returns, while this harness benchmarks the retrieval backend underneath before you commit to one.

A dash means the official docs describe no such feature as of July 2026: both frameworks are designed to score what your pipeline returns, and that is the intended division of labor. Pick the backend here, then score your pipeline with them.

FAQ

About the author

Built by Ken Imoto: 300+ technical articles across Zenn, Qiita, Dev.to and this site, 40+ books in 4 languages, 400K+ pageviews on Zenn and Qiita, 4 research papers on Zenodo, and creator of the LLMO Framework.

More about Ken →

Related

The measurements from this harness feed rag-db-advisor, an MCP/CLI advisor that answers RAG-stack questions from the measured evidence.

rag-db-advisor →