Is grep better than RAG for AI agents?

On this benchmark, yes. A grep-over-filesystem agent answered 97.9% of 96 held-out questions correctly against 81.3% for naive top-k RAG, a gap of 16.7 points, with agentic RAG in between at 90.6%. The corpus was engineered to expose exactly the workloads embeddings handle badly, so treat it as an upper bound on the gap rather than a universal result.

Do AI agents still need a vector database?

Not for every workload. Where the agent can navigate and match text exactly, ls, find, cat, and grep retrieve better than a single-shot top-k lookup, because the agent can iterate. Vector search still earns its place on fuzzy similarity over very large record sets, and nothing stops you running both.

What does grep-based retrieval cost compared to RAG?

More tokens. The grep-agent read roughly 12.8x the input tokens of naive RAG, spread over more turns, because it explores before it answers. On corpora where top-k already answers correctly that is the worse trade; it pays for itself on counting, exact-identifier lookup, and multi-hop questions.

Can I reproduce the benchmark?

Partly. The 18,340-file Vela Ground Systems corpus and the 116 golden questions are public under CC BY 4.0, so the corpus and the questions can be checked independently. The grading harness and the held-out answer key stay private, because publishing the answers would make the test set worthless.

← All writing

Benchmark

Agent filesystem beats naive RAG on retrieval accuracy.

We gave an agent ls, find, cat, and grep over a read-only workspace instead of a vector database, then measured whether exploring retrieves better than embedding. It does.

The Grepticon teamJuly 23, 20265 min read

Retrieval for AI agents almost always works one way: embed your documents, then return the top-k chunks nearest each query. It is fast and it is everywhere, but it rests on an assumption worth questioning. Nearest in vector space is not the same as right, and a single lookup only ever sees a handful of chunks, so whole classes of questions stay out of reach: counting across a corpus, pinning one exact identifier, following a fact through several files. We wanted to test the opposite idea. Skip the embeddings, give the agent a read-only filesystem and the ordinary commands used to explore one, and measure whether exploring retrieves better than embedding.

The result

The grep-agent answered 97.9% of a 96-question held-out set correctly. That is +16.7 points over a naive top-k RAG baseline.

Retrieval accuracy

grep-agent97.9% (92.7–99.4)

agentic-RAG90.6% (83.1–95.0)

naive-RAG81.3% (72.3–87.8)

As graded, with 95% confidence intervals. Vela Ground Systems corpus, 96 questions.

grep-agent

+16.7points

naive-RAG

How we ran it

One fictional corpus, Vela Ground Systems: 18,340 files, 44.8 MB, roughly 29,229 chunks, generated from a fixed seed so the artifact is reproducible and hash-pinned. We engineered it so retrieval can fail: dense with near-duplicate records, cross-referenced incident chains, and answers scattered across many files instead of sitting in one tidy paragraph.

That corpus is public. All 18,340 files, all 116 questions, and the full answer key are on GitHub at Grepticon/vela-corpus, alongside the same hash-pinned artifact these runs were graded against. So you do not have to take any of this on trust: score your own system on the same 96 held-out questions and check our number against yours. The eval harness stays private, but nothing you need to reproduce the measurement does.

Three systems ran the same 96 held-out questions, on the same driver (gpt-5.6-luna), with the same answer-format instruction:

naive-RAG: one top-k vector lookup, k tuned on a separate dev split, no iteration.
agentic-RAG: the same vector search, exposed as a tool the agent can call as many times as it likes.
grep-agent: the four filesystem tools (ls, find, cat, grep) over the Grepticon VFS, through the same SDK we ship.

Grading is answer accuracy: did the answer match the reference. About a quarter of the questions go to a blind, rubric-based judge (claude-sonnet-5, a different model family from the driver and never told which system produced an answer); the rest score programmatically. We hand-audited a stratified sample of judge verdicts: 30/30 agreement, zero overrules. The harness runs all three systems from one command and writes a per-turn trace for every question, so every number is reconstructible.

Where the gap lives

The systems tie where retrieval is easy. Every one scores 100% on the control and verbatim-lookup questions. The gap concentrates in exactly the workloads top-k retrieval is structurally bad at:

Corpus-wide aggregationcounting every matching record across the workspace23.1% → 100%

Identifier haystacksfinding one exact ID among hundreds of near-identical records64.7% → 100%

Deep reference chainsfollowing a fact across four hops of linked documents60.0% → 100%

The pattern repeats. Counting every matching record means seeing all of them; top-k sees ten. Finding one exact identifier among hundreds of near-identical records is a needle embedding similarity blurs, and grep matches it exactly. Following a fact across four linked documents needs iteration a single-shot lookup cannot do. These are failure modes of the interface, not the model, so a better interface closes them.

What this means for your agents

Nothing here was a special rig. The grep-agent hit that number through the same SDK and hosted API you get on signup, reading a workspace pushed over the wire. That is the product: point Grepticon at your own documents and your agent gets the same read-only filesystem and the same four commands, with no embedding pipeline to build, no chunk size to tune, and no vector store to keep in sync as your data changes.

It is not free, and where it wins matters. The grep-agent reads more to answer better: roughly 12.8× the input tokens of naive RAG, over more turns. On corpora where top-k already answers correctly, that is the worse trade. Where it pays for itself is the retrieval embeddings quietly get wrong: counting across a whole workspace, pinning one exact identifier among near-identical records, following a fact through linked files. That is where the 16.7 points came from, and where a filesystem earns its cost.

If that is your agents' workload, the fastest way to know is to run it on your own files. Signup is free and there is no retrieval infrastructure to stand up first. The quickstart walks through pointing a workspace at your documents and handing your agent the four commands.

Common questions

Is grep better than RAG for AI agents?: On this benchmark, yes. A grep-over-filesystem agent answered 97.9% of 96 held-out questions correctly against 81.3% for naive top-k RAG, a gap of 16.7 points, with agentic RAG in between at 90.6%. The corpus was engineered to expose exactly the workloads embeddings handle badly, so treat it as an upper bound on the gap rather than a universal result.
Do AI agents still need a vector database?: Not for every workload. Where the agent can navigate and match text exactly, ls, find, cat, and grep retrieve better than a single-shot top-k lookup, because the agent can iterate. Vector search still earns its place on fuzzy similarity over very large record sets, and nothing stops you running both.
What does grep-based retrieval cost compared to RAG?: More tokens. The grep-agent read roughly 12.8× the input tokens of naive RAG, spread over more turns, because it explores before it answers. On corpora where top-k already answers correctly that is the worse trade; it pays for itself on counting, exact-identifier lookup, and multi-hop questions.
Can I reproduce the benchmark?: Partly. The 18,340-file Vela Ground Systems corpus and the 116 golden questions are public under CC BY 4.0, so the corpus and the questions can be checked independently. The grading harness and the held-out answer key stay private, because publishing the answers would make the test set worthless.