Does a call-graph index actually help coding agents?

Paired A/B studies on real repositories. Short answer: yes, on structural code questions. And the more surprising result: with the index, a small model does work that otherwise needs a frontier model.

codeindex is a call-graph index for Go, TS/JS, Python, and PHP. It answers who-calls-X, what-X-calls, and blast-radius questions with complete, always fresh path:line output. This page answers a narrower and more honest question than "is it good": when you give a coding agent this tool, does it spend fewer tokens and get more answers right, and on which kinds of task?

Method

Each task is run twice under identical conditions: arm A (control, the agent has grep and read only) and arm B (the agent additionally has codeindex). Same model, same prompts, paired and repeated. Tasks are mined from real open-source repositories (gin, flask, nest, laravel/framework, and peers), and ground truth comes from ripgrep and the call graph, never from the index under test. The control arm is hard-isolated from the index by an environment guard, so it cannot leak.

Result 1: the index wins big on structural tasks, and only on those

Not everything benefits. The value concentrates in tasks that are graph-shaped: caller attribution, occurrences, blast radius. For comprehension and locate tasks, plain grep is already the right tool and the index is roughly a wash, occasionally worse. That distinction matters: a good setup routes by task type instead of reaching for the index every time.

Bars show the median paired reduction in total task cost when the agent has the index. Figures from the Go OSS navigation and concept benchmarks (gin, prometheus, and peers).

Result 2: the effect is structural, it survives the Go to PHP boundary

The obvious worry: maybe these wins are memorized quirks of the Go repos the tool was tuned on. To test that, the full task set was reproduced on laravel/framework (PHP), a different language where static call resolution is genuinely harder (duck-typed dispatch produces ambiguous edges). If the signal is structural, its direction should hold across the boundary. It did.

All four task types point the same direction as Go. On the two hardest buckets the index does not just cut cost, it improves correctness: edit impact up 13 points, occurrences up 25 points of success rate. "Use the index for structural questions, skip it for comprehension" is a property of the task type, not the repository or the language.

Result 3: you do not even need a router, over-retrieval hits the ceiling

Result 1 says the setup should route by task type. Route with what? A trained classifier (local embeddings plus logistic regression) picking the right tool per task scored only 70% end to end, and its errors concentrated exactly in the caller-shaped tasks where the index matters most.

The measured answer is simpler: do not decide. Each retrieval costs milliseconds, so run all of them (callers, symbol search, word-boundary grep) and emit one union answer. That deterministic pipeline scores 100% success, F1 0.95 to 0.99, per-task identical to the best per-question tool, on all three languages tested. No agent in the loop, no model, no routing. It ships as a product verb, codeindex nav.

Every piece of the shipped pipeline sits behind its own measurement, including the final detail: nav's grep only switched to word-boundary matching after a re-run showed it held 100% success and raised F1 on every task set (short anchors like File stop matching superstrings).

Result 4: with the index, small models replace big ones on structural work

This is the strongest result and the clearest picture of where the value sits. We ran a five-model ladder, from a frontier-class cloud model down to a local 7B, on the 34 hardest caller and blast-radius tasks. Every model ran each task twice: once with only a shell (grep and read), once with the index attached as an MCP server, the same surface any MCP-capable agent gets. 340 runs total. Success means the model returned the complete, correct set of locations.

Read the chart top to bottom:

• The frontier model does not need the index to be right. glm-5.2 scores 94% with it and 88% without, a difference inside the noise at this sample size. What the index buys the big model is efficiency: half the tokens (median 19.5k vs 40k) for the same answers.
• An 8B with the index matches the frontier model without it. 85% vs 88%, at a third of the tokens. This is the compounding claim in one line: the index converts a cheap model into frontier-level performance on structural questions, so you do not have to buy scale to get them answered.
• Every model below the frontier is lifted 32 to 68 points. And the mechanism is checkable in the traces: within the index arms, runs that actually called the index tools succeeded at 56 to 94%, while runs that did not went 1 for 12 across the small models. The lift is the tool, not the prompt around it.
• The floor is a 7B coder finetune. qwen-coder 7B is the only model that fails to adopt the tools reliably (25 of 34 runs), and it converts only 11 of 25 even when it does. Notably it does far worse than the plain 7B. Tool use runs on instruction-following, not code knowledge.

Two honesty flags on this experiment. First, a harness confound found in the trace audit: the control arms could still see disabled index tools in their tool list (every call was refused and nothing executed, verified across all 340 runs), which wasted some control-arm turns. So treat the small-model lifts as upper bounds; the frontier-model ceiling result is unaffected. Second, 4 of 340 runs died on local infrastructure after two retries and are counted as control-arm failures, worth at most 3 points in two cells. Both are recorded in the benchmark findings, next to a pre-registered sister experiment where a 30B local model with the index beat the frontier model without it, 64.7% to 38.2%.

The trust problem the index also fixes

Success rates undersell one thing the traces show clearly. When a small model fails a callers question with only grep, it usually does not say "I could not find them all". It returns a partial list and claims it is complete. In the ladder experiment, cheap models without the index claimed complete coverage while being wrong in 50 to 80% of their failures. With the index the top models nearly stop doing this (glm: 8 false-confident runs down to 2, the 8B: 13 down to 1). A deterministic evidence source is the cheapest grounding mechanism available: the model stops guessing because looking is cheaper than guessing.

A note on honesty

The first PHP edit-impact run (small sample) showed the index making the agent faster but less correct, a tempting story about ambiguous PHP edges. A larger 30-per-arm re-run refuted it: the effect was noise from a handful of runs, and at scale edit impact is a clean win (60% cheaper, 13 points more correct). The lesson recorded in the findings: do not build a mechanism narrative on a 12-row cell. Negative results on this page stay published: comprehension tasks do not benefit, the classifier router lost to not routing, and the confounds above are flagged where they occur.

Why this matters

A large fraction of a coding agent's spend goes to navigation, locating the relevant code, not reasoning about it. This project set out expecting to train a small routing model for that layer. The measurements kept refusing: the classifier router lost to running everything, cheap query-formulation gates lost to deterministic retrieval, and at every gate where a model might have earned its way in, structure plus cheap glue matched or beat it. So the conclusion is stronger than "the index helps": for navigation-shaped questions you can keep the expensive model out of the loop entirely, or swap it for a model an order of magnitude smaller, and spend real reasoning capacity only where reasoning is required. What remains genuinely open for a model is narrow and named: query reformulation on the weakest semantic-search repos, and the trust-or-verify call on [ambiguous] edges.