Web Search in 4 LLMs: Codex's Cache Went 24/24

The Kimi API usage in this post was paid for with credits Moonshot AI provided for evaluation. Moonshot AI had no part in designing, running, or concluding the test.

Four months ago I wrote that my own AI agents search Brave, not Google. Like most people, I thought about AI search in terms of which search engine sits underneath.

This time I reread the official docs to compare web search in Claude, Codex, Kimi K3 and Qwen. That framing no longer fits. None of the four vendors names its search engine. OpenAI makes its own maintained index the default in Codex. Kimi now sells search itself as a standalone API. The question has moved from “which engine?” to “where does search run, and who calls it?” I did not expect that much change in four months.

If the mechanics changed, the results should too. Before measuring, I assumed live search would beat a cache whenever the question needs fresh information.

Codex CLI’s web search is not live by default. It runs in “cached” mode against an index OpenAI maintains. The docs give the reason: it lowers prompt injection risk (Codex: Web search). I read that as trading some freshness for safety.

The measurements said the opposite. I asked the same 8 questions 3 times under 5 setups: Claude, Codex live, Codex cached, Kimi K3 and a local Qwen. Only cached search got every answer right. It even found the title of an article that had been online for about three hours.

The other surprise: the same search API produced 21 correct answers with one model and 9 with another. I gave Kimi K3 and a local Qwen identical access to Kimi’s search API. The results were the same. How each model read them was not.

Where web search runs differs by vendor

Before comparing anything, I checked the mechanics in public docs. The biggest difference is where the search actually runs.

None of the four names its search engine officially. For Claude, a March 2025 report said Brave Search had been added to Anthropic’s list of subprocessors (Simon Willison). Today’s docs only say “Anthropic’s web search backend.”

Claude and OpenAI run the search on their own servers and hand the results to the model. With Claude, results come back encrypted, and you must send them back unchanged to continue the conversation (Claude: Web search tool).

Kimi is mid-transition. The built-in $web_search tool retires on 2026-10-20. Its replacements are standalone REST APIs for search and page fetch, such as /v1/tools/search (Kimi API: Web Search). With the new APIs, you decide when and how often to search. They cost a third to a fifth of the others, billed only when a call succeeds and returns results (Kimi: pricing).

Qwen the model has no search of its own. Alibaba Cloud’s API offers search, but a Qwen running locally can only look things up if you wire in an external search tool.

Two details in the docs you will want to know

• Claude Code’s WebSearch returns only titles and URLs. It does not read the pages. Reading happens in WebFetch, which summarizes the page with a separate model call first. The docs call this “lossy by design” (Claude Code: Tools reference)
• Qwen’s search on Alibaba Cloud silently skips searching above 15 requests per second. There is no error; the search just does not happen. The FAQ also says English questions often fail to trigger search, and recommends asking in Chinese or forcing search. Both notes appear only in the Chinese docs (阿里云: 联网搜索)

I hit the “lossy” part myself while researching this post. I had Claude Code’s WebFetch read the Qwen docs, and it reported finding neither the 15 RPS note nor the English FAQ. Running grep over the raw HTML (about 460 KB) found both. Once a summary sits in the middle, “it doesn’t say that” may only mean “the summary dropped it.”

How I measured

Five setups, the same 8 questions, 3 runs each: 120 graded answers in total, collected on the evening of 2026-10-14.

For Codex, I called the API the CLI uses under the hood. In Codex’s source, cached mode maps to external_web_access=false and live to true (codex hosted_spec.rs). So the API calls reproduce the CLI’s two modes.

These were the 8 questions. I only used questions whose answers I could verify from primary sources or my own site.

q3 is my own book and q4 is my own article, chosen because I know the answers for certain. That choice carries my own bias, and I’d rather say so. Both are in Japanese, so they also test how well each setup reaches Japanese-language sources.

Results

Correct answers per question, out of 3 runs:

On the nonexistent Kimi K4 (q8), every setup said it hasn’t been announced. Qwen once added, without a source, that K4 had been “mentioned as part of plans for 2027 or later.”

Finding 1: the cache wasn’t stale

Cached search answered all 24 correctly. The q4 article went live a little after 5 p.m. that day, and I measured about three hours later. Cached search returned its title in all three runs. Honestly, that surprised me. At least for my site, “an index OpenAI maintains” was refreshed within hours.

Source quality also favored the cache at times. On q6 (when $web_search retires), cached search cited Kimi’s official docs. Live search got the same answer but, in one run, cited an unofficial roundup site. A wider reach also means picking up more unofficial pages.

Live search missed q3, the Japanese book, once, saying it couldn’t confirm a book by that title.

Finding 2: same results, different answers

Kimi K3 and the local Qwen got exactly the same search API in exactly the same way. They still scored 21 and 9. That’s more than double.

Qwen’s mistakes followed a few patterns:

• Its belief that it’s 2024 beat the search results. On q4 it said: “As of now (May 2024), an article dated October 14, 2026 does not exist. That is a future date.” On q1 it found the July 16 date, then concluded it was “not released as of 2025”
• Its sources and its answers disagreed. On q5, all three runs cited the release-candidate announcement dated May 21, then answered with the final release date, July 28
• It invented a source. On q3 it cited an Impress press release URL that doesn’t exist (404) and gave a wrong release date. Each run gave a different wrong date
• It barely searched. 1.7 searches on average, less than half of K3’s 4.0

Model size differs, so this isn’t a verdict on Qwen as a whole. What it does show is that web search accuracy is not decided by the search API alone.

Finding 3: Kimi’s search API has two gaps

All three of K3’s misses were q4. It rephrased the query 12 times and never found the same-day article. The page simply wasn’t in Kimi’s search index yet. The cause wasn’t the model. It was index freshness.

The second gap: queries containing the site: operator return HTTP 502. Both K3 and Qwen tried site:kenimoto.dev when hunting for a specific page. Each time, the search API returned a non-JSON error page, and on the first run it crashed my script. My tool now returns the failure to the model so it can rephrase and try again.

Hitting the search API directly also showed how much wording matters. Searching “Kimi K3 release date Moonshot AI” let “date” take over: I got a time-zone site and JavaScript Date docs. Searching just “Kimi K3” put the official site, the paper and GitHub on top. None of the models fell into this trap during the actual runs, though. Both K3 and Qwen rephrased their own queries around it.

By the way, the soon-to-retire $web_search tool didn’t work with K3. The second call, the one that returns search results, failed every time with 400 tokenization failed. Passing search through regular function calling, as in this test, works fine with K3 (more in Kimi K3 vs Claude: Video Review, Same Findings).

Claude struggled with a Japanese book

On q3, Claude said in 2 of 3 runs that it couldn’t find the book and that it may not exist. The book is listed, with its release date, on the publisher’s NextPublishing page and on Rakuten Books.

It was persistent elsewhere, though. In one q4 run, WebSearch didn’t surface the article, so Claude found it through the sitemap and opened the page to confirm the title and date. Because WebSearch returns only titles and URLs, fetching pages when needed is part of the design, and here it paid off. Claude also often noted that it had only checked secondary sources, and avoided firm claims.

The 22-second median includes Claude Code’s agent loop. Calling the search tool directly through the API should be faster.

How I’ll use them

Here’s what I’m taking from this:

• Leave cached search as the default for research. If you use Codex, there’s no need to pass --search every time. The safer setting didn’t lose on accuracy
• Don’t trust “not found” for non-English sources. Not found is not the same as not existing. When Claude says so, I check another way
• If you wire in search yourself, the model matters more. With the same search API, K3 and Qwen differed by more than 2x. Pick a model that can read search results before picking the cheapest search API
• Make your own search tool return failures. Kimi’s search API returns 502 on site:. Pass the failure back instead of crashing, and the model will rephrase and search again

Limits

• This is a small test: 8 questions, 3 runs. Topics lean toward Kimi K3 and MCP, and q3 and q4 are my own book and article
• Claude ran through Claude Code; Codex results come from calling the API the CLI uses
• The Qwen setup is a local 35B model wired to Kimi’s search, not Qwen’s official search
• Each setup works differently, so times aren’t directly comparable. I didn’t total the costs in this test

Next, I want to call Claude’s search tool directly through the API and try Qwen’s official search on Alibaba Cloud, to close the gaps between setups.

References

• Claude: Web search tool: pricing, citations, encrypted results, versions
• Claude Code: Tools reference: how WebSearch and WebFetch behave
• Codex: Web search: cached / live modes
• OpenAI: Web search tool: external_web_access
• openai/codex hosted_spec.rs: how the modes map to external_web_access
• Kimi API: Web Search / best practices / pricing
• 阿里云 百炼: 联网搜索 (Chinese; the 15 RPS note and the English-question FAQ appear only here)