OSS · Python CLI + MCP

rag-db-advisor

Name: rag-db-advisor
Author: Ken Imoto

Find the right vector database for your RAG workload, with the evidence attached.

"Which vector DB?" answers are usually made of opinions. This one is made of measurements: ask which backend fits your workload, what a latency number actually means or which trap you are about to step on, and it answers from 54 evidence chunks distilled from 9 benchmarked configurations. The advisor returns evidence; your LLM writes the answer.

Instead of choosing by popularity, choose by your workload.

Try it on GitHub How it decides what to say →

Free · Open Source · an evidence-backed answer in seconds

4.3 s measured answer time
~2 min setup to first answer
54 evidence chunks
Free Open Source · MIT

Answers you can trace 54 chunks, each traceable to a bench record or a reproduced trap
Comparison on demand 9 configurations × 10k / 100k passages, 860 annotated queries
Traps before you hit them 3 silent-degradation cases, reproduced before written down
Runs inside Claude Code 3 MCP tools, no generation key on the server

Terminal demo: rag-db-advisor answering a Japanese question about choosing a vector DB for 100k documents, returning measured evidence blocks with recall and latency numbers and their source records

Your question rag-db-advisor Evidence only. No generation inside. Measured evidence Your LLM answers

Three steps

Ask rag-db-advisor ask "…" One free-form question, over CLI or MCP. No questionnaire to fill.
Evidence comes back === note:qdrant#1 (knowledge/ja/qdrant.md) === Chunks with measured numbers, each headed by its chunk ID and source record.
You decide --llm / your LLM over MCP Synthesis happens outside: your LLM (or you) writes the verdict, sources attached.

Example: a real answer

rag-db-advisor ask "Building RAG search over 100k Japanese documents. Recall must hold, operations should stay light. Which vector DB?" --llm

✓ Recommended Qdrant (HNSW)

recall@10 0.947 at 100k · top ANN config, 0.5 pt under the 0.952 exact-search ceiling
search p50 3.3 ms · same corpus, same embeddings as every other candidate
LanceDB ruled out · default quantization drops recall to 0.811

Weighed against: LanceDB (recall@10 0.811 at 100k, built-in quantization)

4.3 s · 6 evidence chunks · gpt-4o synthesis

real run, 2026-07-12: rag-db-advisor ask --llm --model gpt-4o, condensed from the Japanese original

What the evidence can weigh

Every axis below is covered by measurements or reproduced traps, the only two things allowed into the store:

Dataset size measured at 10k and 100k passages
Retrieval quality recall@10, nDCG, MRR, hit@k vs human ground truth
Latency p50 / p95 / p99 per query
Write path bulk-load + index-build seconds
Operational traps 3 reproduced cases, machine-checked
Deployment shape embedded vs server, separate tables

Metadata filtering, hybrid search and cost estimates sit outside the v0.1 evidence, so the advisor errors instead of improvising on them. The measured scope is honest about its own edges.

Who it’s for

AI engineers who need a citable number for "why this database" in a design doc
RAG developers debugging "search feels slow" and wanting the known traps ruled out first
Claude Code users who want RAG-stack answers grounded in evidence instead of model priors
Tech leads reviewing a vector-DB proposal and checking what the numbers actually mean

When to reach for it

Before picking a database for a new RAG feature: start from same-conditions measurements instead of vendor material
When ClickHouse vector search "feels slow": the reproduced trap catalog exists exactly for that impression
When a review asks "why Qdrant?": answer with evidence blocks that carry their source record IDs
Mid-implementation in Claude Code: your LLM pulls the evidence over MCP without leaving the session

Install

pip install git+https://github.com/kenimo49/rag-db-advisor
export OPENAI_API_KEY=sk-...
rag-db-advisor ingest

The API key is used for embeddings only (text-embedding-3-small). Over MCP no generation key is needed server-side: the server returns evidence and your LLM writes the answer.

What it knows

18 bench records 36 note chunks 54-chunk store If it wasn’t measured or hit, it doesn’t ship. advise / compare / traps

Measurement records: 9 backend configurations × 2 corpus scales (10k / 100k MIRACL-ja passages, 860 human-annotated queries), each record carrying quality, latency, ingestion and index-verification evidence
Operational notes: 8 hand-written files, one per backend plus cross-cutting principles, covering only behavior reproduced during the bench work
A trap catalog: 3 ways a backend degrades with zero errors raised, each one reproduced, fixed and written down
Methodology caveats baked into answers: embedded vs server latency is not directly comparable, and the numbers are MIRACL-ja + text-embedding-3-small on a single node
3 MCP tools: advise (evidence retrieval), compare_backends (10k / 100k comparison table), list_traps (per-backend traps). Failures return {"error", "hint"} instead of raising

The trap catalog, sampled

Three of the seven backends shipped a way to degrade with zero errors raised. All three were hit during the bench work, reproduced, and only then written into the knowledge base:

Backend	What happens	What you see	Machine check
ClickHouse	Forget the allow_experimental_vector_similarity_index flag at query time and HNSW silently becomes brute force	"HNSW" at 27 ms vs brute force at 31 ms: almost the same	EXPLAIN indexes=1 shows whether the skip index ran
Qdrant	Segments under 20 MB are not HNSW-indexed by default (indexing_threshold)	Status green, indexed_vectors_count = 0, every query full-scans	Assert indexed_vectors_count == points_count
Milvus	The quick-setup path silently drops index_params and builds AUTOINDEX	You specified HNSW; describe_index says AUTOINDEX	Check index_type via describe_index after creation

source: knowledge/ja/clickhouse.md · qdrant.md · milvus.md (reproduced before written down)

Advice the evidence actually changes

Two examples of answers that flip once you look at measurements instead of datasheets:

Default quantization

recall@10 0.983 → 0.901

Same 10k corpus, same embeddings: the best plain-HNSW configuration vs LanceDB IVF_HNSW_SQ, whose built-in quantization costs 8 recall points (19 at 100k). Read "does quantization kick in", not "does it say HNSW".

source: knowledge/ja/lancedb.md, cross-cutting.md

The planner, not the index

2,048 rows: seq scan · 4,096 rows: index scan

pgvector builds your HNSW index, then the Postgres planner ignores it below a few thousand rows (measured at ef_search=100). Small-scale "HNSW feels slow" verdicts are often this.

source: knowledge/ja/pgvector.md

Usage

# evidence only
rag-db-advisor ask "pgvector or Qdrant for 100k Japanese docs?"

# + OpenAI synthesis (default model: gpt-4o-mini)
rag-db-advisor ask "ClickHouse vector search feels slow. What should I check?" --llm

# register as an MCP server in Claude Code
claude mcp add rag-db-advisor -- rag-db-advisor mcp

Terminal screenshot: rag-db-advisor answering a Japanese question, each evidence block headed by its chunk ID and source record, with measured recall, latency and index-verification lines — What an answer looks like: every block names its chunk ID and source, so each claim traces back to a record.

Refusing is a feature

The failure mode this tool is built against: the retrieval layer breaks, the calling LLM silently answers from prior knowledge, and nobody notices. So the advisor fails closed:

Retrieval failures return an explicit {"error", "hint"} across all three MCP tools; exceptions are never swallowed into prose
An empty result set is also an error, not a null answer: the calling model sees "no evidence" instead of "no problem, let me guess"
No verdicts are generated inside: the advisor retrieves, your LLM (or you) synthesizes, and the sources stay attached

Every chunk traces back to a bundled bench record (knowledge/results/*.jsonl) or a hand-written note about reproduced behavior (knowledge/ja/*.md). Speculative advice is out of scope by policy.

Dogfooding: the retrieval layer imports rag-retriever-bench’s own BaseRetriever abstraction and stores its knowledge in Chroma, because the bench data shows every plain-HNSW backend is quality-tied at this corpus size. The advisor follows its own advice.

Why this tool exists

While building rag-retriever-bench, the numbers turned out to be the durable part: they land in JSONL and reports and stay there. What kept evaporating was the operational knowledge around them: the flag that silently disables an index, the threshold that leaves a collection unindexed behind a green status. Opinions about databases do not reproduce; measurements and reproduced traps do. So this advisor closes the loop: measure, distill what was measured and hit into a knowledge base, and serve it to the next question. And when it has no evidence, it says so instead of guessing.

Alongside an LLM chat and ann-benchmarks

Different tools answer "which vector DB?" differently. The honest difference is where the answer comes from:

	rag-db-advisor	Asking an LLM directly	ann-benchmarks
Answer based on	Bundled measurements + reproduced traps	Model priors; sources not traceable	Public ANN benchmark runs
Scope	RAG retrieval backends on a Japanese text corpus (MIRACL-ja)	Anything	ANN algorithms and engines on standard feature datasets (glove, sift, etc.)
Operational traps	✓ 3 reproduced cases with machine checks	—	—
When evidence is missing	Explicit error (fail-closed)	Answers anyway	—
How you use it	MCP tools + CLI	Chat	Web charts

A dash means the option does not cover that axis, not that it is worse: an LLM chat is broader and ann-benchmarks is the reference for raw ANN performance. This advisor is deliberately narrow: one corpus family, one embedding model, and every claim traceable to a record.

FAQ

Do I need an OpenAI API key?

Yes, for embeddings only (text-embedding-3-small): queries and chunks are embedded through the OpenAI API. Over MCP the server needs no generation key: your LLM writes the answer. The optional --llm flag adds OpenAI synthesis (gpt-4o-mini by default).

Is my question sent anywhere?

The question text goes to the OpenAI embeddings API to be embedded; the knowledge base itself is local (Chroma, on disk). With --llm the question and evidence also go to the OpenAI chat API for synthesis. Over MCP, synthesis happens in whatever LLM you already use.

Can I ask in English?

Retrieval works across languages since it runs on embedding similarity, but the knowledge base itself is written in Japanese, both the notes and the rendered records. Over MCP your LLM translates the evidence when it writes the answer. Native English knowledge is not bundled in v0.1.

How much evidence is inside, exactly?

54 chunks in v0.1: 18 measurement records (9 backend configurations × 2 corpus scales from rag-retriever-bench) plus 36 chunks from 8 hand-written operational notes. If it wasn’t measured or actually hit, it isn’t in there.

Can I just follow its advice for my workload?

Treat it as a measured starting point, not a verdict: the numbers are MIRACL-ja + text-embedding-3-small on a single node, and the answers carry that caveat themselves. For your own corpus and infrastructure, run rag-retriever-bench: that is exactly what it is for.

What happens when there is no evidence for my question?

You get an explicit error, not a guess. All three MCP tools return {"error", "hint"} on failure, and an empty result set is treated as an error too, so the calling LLM cannot silently fall back to prior knowledge.

Can I add my own knowledge?

Yes: docs/adding-knowledge.md describes the format for notes and measurement records. The house policy still applies: write down only behavior you measured or reproduced.

Why does it not generate the answer itself?

Reproducibility. If the advisor ran its own LLM, the same question could get different verdicts between runs. By returning deterministic evidence and leaving synthesis to the calling LLM, the numbers only change when the bench is re-run, and every claim keeps its source attached.

About the author

Built by Ken Imoto: 300+ technical articles across Zenn, Qiita, Dev.to and this site, 40+ books in 4 languages, 400K+ pageviews on Zenn and Qiita, 4 research papers on Zenodo, and creator of the LLMO Framework.

All evidence comes from rag-retriever-bench, the benchmark harness that measures the same corpus and queries across 9 backend configurations.

Related developer tools

All products →

← All products