The Context Gap

The primary failure point in most enterprise AI implementations is not the underlying model's reasoning capability, but the quality and scope of the information provided to it. In large-scale environments, an LLM—whether acting as a coding assistant or a CLI agent—is only as effective as its current context window allows.

When an agent lacks access to a structured representation of your repository, it defaults to one of two behaviors: it hallucinates a structure based on common patterns in its training data (which may not apply to your specific architecture), or it provides generic responses that ignore established internal conventions. This is the Context Gap.

The gap gets argued as a tooling question—"do we need a vector database?"—when it is a question about individual facts. Every fact has properties that decide how it should reach the window: stable or volatile, locatable by name or only by meaning. Classify one wrong and you either stand up infrastructure that retrieves the wrong thing, or maintain a file that quietly lies.

Declaration, Navigation, and Retrieval Aren't the Same Problem

The common framing is a binary—hand-maintained context files versus embedding-based retrieval. It omits the mode agents actually spend most of their turns in. The cleaner split is by who does the work of getting the fact into the window: you state it, the agent finds it, or infrastructure finds it.

1. Declaration: You State It Up Front, Every Turn

These are static files residing within the repository (e.g., CLAUDE.md, AGENTS.md, or .cursorrules), holding explicit instructions on project architecture, coding standards, security constraints, and recurring patterns. Nothing looks them up, because they are already there—loaded as part of the prompt on every turn.

In practice these are not one file but a stack: an enterprise or global policy layer, a user-level file, the project-root file, nested per-directory files scoped to a subtree, plus shared fragments pulled in by import. Multiple tools have converged on directory-scoped inclusion—the nearest matching file loads into the window rather than replacing the one above it—and several support glob-based scoping, so a rule only enters the window when a file it applies to is in play. That is where the real token savings live, and it is the mechanism most teams skip in favor of one monolithic file.

The mechanic that matters: layers concatenate, they do not merge. There is no precedence engine and no conflict resolution pass. Two layers that contradict each other both land in the window, and the model picks one non-deterministically. That is the operational reality behind the three-layer instruction hierarchy—global, repository, user—described in "The Proliferation Problem": the hierarchy is a convention your organization enforces, not a guarantee the tooling provides. An organization that ships a global policy layer without a contradiction check has shipped advisory text, because any repository can override the baseline silently by stating the opposite.

2. Navigation: The Agent Finds It Live

The second mode is what the binary framing leaves out: give the agent grep, glob, and file reads, and let it search the working tree at query time. This is the default operating mode of coding agents like Claude Code, and a lightweight structural map can guide it in place of a full index.

Aider's repo map is the reference implementation. It parses source files with tree-sitter to extract symbol definitions and references, builds a reference graph across the repository, ranks nodes with a PageRank-style algorithm weighted toward files relevant to the current request, and emits a token-budgeted skeleton—signatures only, on the order of a thousand tokens by default—into the prompt. The agent reads full file bodies on demand from there.

Two properties make this stronger than it looks. There is no semantic index to drift out of sync: the derived symbol cache is invalidated by file mtime, so the map is always generated against the tree as it currently exists, which makes it branch-correct by construction. And exact identifiers are what lexical search is good at and embeddings are bad at—for the definition of ERR_TOKEN_EXPIRED or every caller of a config key, grep answers definitively where a vector search approximates.

The cost is paid in tool turns rather than infrastructure: every search is a round trip, and latency scales with how many the agent needs. Navigation degrades in two specific conditions—conceptual queries against a badly-named codebase, where the vocabulary in the question never appears in the source, and logic spread thinly across many small files, where no single read gives the agent enough to work with.

3. Retrieval: Infrastructure Finds It, and You Maintain the Index

This is the RAG path: chunk the codebase and documentation, embed the chunks into a vector database, inject the top-scoring segments at query time. It scales to corpora where turn-by-turn navigation is too slow, and to documentation nobody will hand-summarize.

The part that gets glossed over is the word "relevant." Semantic similarity is not a solved relevance function, and on code it fails in four specific ways:

• Semantically right, temporally wrong. Deprecated code is frequently more topically dense with the query's vocabulary than the live implementation—it is surrounded by migration comments, "do not use" notes, and references to its replacement. Embeddings encode topic, not currency, so the dead version outscores the current one.
• Near-duplicate collapse. Vendored dependencies, generated clients, build output, and test fixtures are near-identical to source, so top-k collapses into k copies of one fact. Mitigations are MMR-style diversity re-ranking, content-hash dedup at index time, and hard exclusion globs—the cheapest of the three, and the one most teams never write.
• Prohibitions do not retrieve. A rule like "never call requests directly, use the internal HTTP client" is a few dozen tokens of low topical density competing against thousands of lines of actual HTTP code. Search for how the codebase does HTTP and you surface the forbidden pattern, not the rule banning it. Stated generally: a constraint that forbids something cannot be retrieved by searching for the thing it forbids.
• Exact identifiers are lexical, not semantic. Dense vectors are weak at exact-token match. Production retrieval for code is hybrid—dense and BM25 run in parallel, fused with reciprocal rank fusion, then re-scored by a cross-encoder rerank pass. A pure dense-vector setup is an incomplete build, not a baseline.

That third failure mode is the mechanism behind the architecture rule below, not a footnote to it. Prohibitions have to be declared, because no query finds them.

Architecture Rule: Route each fact by its properties, not by your tooling budget. Stable, universal, undiscoverable by reading the code—a prohibition, an architectural intent, a "we tried X and it failed because Y"—declare it. Locatable by name—symbol, error code, config key, path—navigate to it, and do not index what grep already answers. Locatable only by meaning, at a scale where turn count is the bottleneck—that case, and only that case, earns an index.

The Token Budget Behind the Split

"It scales to massive repositories" is the usual justification for retrieval, and repository size is the wrong axis. The real asymmetry is when you pay.

Declarations cost tokens × turns: a rule in a context file is billed on turn 1 and again on turn 40, whether or not that turn had anything to do with it. Retrieval costs tokens × queries—variable, paid per call, zero when not invoked. That asymmetry, not repo size, is why short universal constraints belong in declarations and long-tail specifics belong behind a query. A rule governing every edit is cheap at any session length; the same token count spent on one legacy module is not.

Prompt caching changes the arithmetic without removing it. A stable prefix—tool definitions, system prompt, context files—can be served from cache on repeat turns at a fraction of the cost of fresh input. But caching is a prefix match, valid only up to the first byte that differs. Put volatile retrieved content ahead of stable content and you invalidate the cache for everything after it, on every subsequent turn.

Order the window by volatility: stable content first, retrieved content last. A retrieval block injected above the system instructions is a cache miss disguised as a context improvement.

Larger windows do not remove the need for any of this; they change the failure mode from hard to soft. An overflow throws an error you can see and handle. A window stuffed with marginally-relevant retrieved text throws nothing and just degrades—the same context rot that makes long-running single-session loops unreliable, arriving through the retrieval layer instead of through conversation length. Hard failures get fixed; soft degradation gets absorbed into "the model isn't very good at this repo."

The Risk of Omission vs. The Danger of Stale Data

A common misconception is that a "smart enough" model will eventually figure out your conventions if they are just "somewhere" in the repo. This is false for exactly the facts that break things: an arbitrary, organization-specific decision—a naming convention, a deprecated flag, a security exception—has no prior in the model's training and cannot be derived by composing other facts. If it is not explicit in the context window, or reachable by navigation, at the moment of execution, there is nothing to reason from. The model fills the gap with a plausible guess instead of the actual fact, and a confident guess reads exactly like the truth until it is wrong.

The opposite failure is worse, and it applies to both artifacts—the file you wrote and the index you built.

Warning: A stale hand-maintained file is more dangerous than no file at all. An outdated instruction provides the agent with high-confidence, incorrect information. This leads to "Confident Failure"—where the agent generates code that looks correct but violates current infrastructure constraints or security protocols.

Indexes decay the same way, and the decay is harder to see:

• Deleted code lingers. Removing a file or function does not remove its chunks unless the pipeline can identify them. That needs deterministic chunk IDs—path plus symbol plus content hash—and a periodic reconcile pass diffing the index's ID set against the tree. Without it, the agent cites code that no longer exists.
• Branch divergence serves the wrong repository. An index built from main hands pre-refactor code to an agent working a feature branch, with full confidence and no warning, and it compounds across simultaneous worktrees. Options: index per branch, filter by commit SHA, or treat the index as background material only and always read the live tree for anything under edit.
• Embedding model drift is silent. Change the embedding model and old vectors sit in a different space than new ones. Similarity scores become meaningless and nothing throws. The index has to be versioned by embedding model ID, and a model change requires a full re-embed—incremental will not do it.

The asymmetry worth internalizing: a stale context file shows up in code review as a diff someone can object to. A stale index shows up as an agent that is subtly wrong, in a place no reviewer is looking.

Making Decay Mechanically Visible

Naming staleness as a risk is not a mitigation. The enforcement mechanisms, roughly cheapest to build first:

1. Path assertions in CI. A script extracts every backtick-quoted path and filename from the context files and fails the build if any no longer resolve. An afternoon of work, and it catches the most common decay: the file moved, the instruction did not.
2. Command assertions. Every documented build, test, or lint command must be the literal command CI runs, sourced from one place rather than copy-pasted into prose. If the pipeline changes and the documented command does not, the pipeline breaks—which is the point.
3. Promote enforceable rules out of prose. Anything a linter can express belongs in semgrep, eslint, or ruff, not in a paragraph an agent may or may not weight; the context file should point at the rule, not try to be it. Prose is for what a linter cannot express: architectural intent, and the record of what was tried and why it failed.
4. Path-coupled review gates. CI flags any pull request touching a governed path—auth/, migrations/, the deployment templates—without touching the matching context file. Mechanically identical to requiring a changelog entry.
5. Ownership and a last-verified date per rule block, with a CI warning past a threshold. Weakest of the five, because it verifies nothing—but it puts decay into a diff where a human sees it.

The gate has to live in CI or branch protection, not in a pre-commit hook. A hook is advisory—--no-verify exists, and the engineer under deadline pressure is precisely the one who will use it.

From Chaos to Coordinated Intelligence

The transition from "Shadow AI"—where individual engineers experiment with unmanaged prompts—to "Coordinated Intelligence" requires shifting the focus toward infrastructure. We have moved past the question of whether LLMs can write code; we are now solving for how the organization ensures those models are operating on a single, accurate version of the truth.

Closing the Context Gap is not a matter of picking the right retrieval product. It is the discipline of classifying facts by their properties, routing each to the mode that fits, ordering the window by volatility, and putting a mechanical gate under every claim that can rot. Keeping those files and indices current as the codebase moves is a pipeline problem, and the mechanisms above cover it.

The harder question lands immediately after. Once every agent call carries a context payload, something has to decide which model that payload is worth spending on—and a small model handed a bloated payload and a frontier model handed a thin one are the same misallocation pointed in opposite directions. Information flattened as it crosses a handoff between them is a Context Gap failure occurring at the routing layer rather than in the repository. The next problem is not what goes into the window, but who decides where the window gets sent.