A filesystem beats naive RAG on retrieval accuracy.
ls, find, cat, and grep over a read-only workspace instead of a vector database, then measured whether exploring retrieves better than embedding. It does.Most retrieval for AI agents works one way: embed your documents, then return the top-k chunks nearest each query. Grepticon takes the other path — it hands the agent a read-only filesystem and the four unix verbs it already knows, and lets it look around. The obvious question is whether looking around actually retrieves better than embedding. We built an eval to answer it, and this is what the eval says.
The result
On our production stack — real credentials, over the network, the same hosted API your agents call — the grep-agent answered 97.9% of a 96-question held-out set correctly. That is +16.7 points over a naive top-k RAG baseline, and it clears the pre-registered ≥ +5-point bar we set before running by more than three times over.
Agentic RAG — the same vector search the agent can call repeatedly — lands in between at 90.6%. Iteration helps; the filesystem helps more.
In the lab, on the same corpus under formal grading, the margin is larger still: grep-agent 100.0% versus naive-RAG 78.1%, a +21.9-point gap. We lead with the production number on purpose — it is the conservative one, measured against everything that can go wrong between a laptop and a container.
| System | Production | In-lab (formal) |
|---|---|---|
| grep-agent | 97.9% | 100.0% |
| agentic-RAG | 90.6% | 91.7% |
| naive-RAG | 81.3% | 78.1% |
| Δ grep − naive | +16.7 | +21.9 |
How we measured
One fictional corpus, Vela Ground Systems (v2): 18,340 files, 44.8 MB, roughly 29,229 chunks, generated from a fixed seed so the artifact is reproducible and hash-pinned. It was engineered so that retrieval can fail — dense with near-duplicate records, cross-referenced incident chains, and answers scattered across many files rather than sitting in one tidy paragraph.
Three systems ran the same 96 held-out questions, on the same driver (gpt-5.6-luna), with the same answer-format instruction:
- naive-RAG — one top-k vector lookup, k tuned on a separate dev split, no iteration.
- agentic-RAG — the same vector search, exposed as a tool the agent can call as many times as it likes.
- grep-agent — the four filesystem tools (
ls,find,cat,grep) over the Grepticon VFS, through the same SDK we ship.
Grading is answer accuracy: did the answer match the reference. About a quarter of the questions are scored by a blind, rubric-based judge (claude-sonnet-5, a different model family from the driver and never told which system produced an answer); the rest score programmatically. We hand-audited a stratified sample of judge verdicts — 30/30 agreement, zero overrules. The harness runs all three systems from one command and writes a per-turn trace for every question, so each number is reconstructible.
What moved, and why
The systems tie where retrieval is easy — every one of them scores 100% on the control and verbatim-lookup questions. The gap is concentrated in exactly the workloads top-k retrieval is structurally bad at:
The pattern is the same each time. Counting every matching record means seeing all of them; top-k sees ten. Finding one exact identifier among hundreds of near-identical records is a needle embedding similarity blurs, and grep matches exactly. Following a fact across four linked documents needs iteration a single-shot lookup cannot do. These are failure modes of the interface, not the model — which is why a better interface closes them.
What we are not claiming
No latency numbers this round. grep over wide, multi-term patterns is currently slower in production than we want it, and that work is in flight. Quoting a speed we intend to change would be dishonest, so this benchmark reports accuracy only. When the latency work lands, it gets its own honest write-up.
It is one corpus. A hard, adversarial, deliberately-discriminating one — but fictional and singular. Read the absolute numbers as evidence for the mechanism, not a universal score. And the win is specific: two of the archetypes we built (version conflicts and verbatim lookups) did not discriminate at all, because their answers sit inside a single retrievable chunk — RAG handled them as well as grep.
Accuracy is not free. The grep-agent spends roughly 12.8× the input tokens of naive RAG and takes more turns to get there — it reads more to answer better. On corpora where top-k already works, that is a bad trade. The case for a filesystem is precisely the workloads where recall fails.