Skip to content

fix(graph): survive un-indexable files, surface unresolved references, and stop storing references the resolver already turned into edges - #174

Merged
Yashasvi2229 merged 9 commits into
mainfrom
fix/graph-hard-aborts-and-unresolved-refs
Sep 7, 2026
Merged

fix(graph): survive un-indexable files, surface unresolved references, and stop storing references the resolver already turned into edges#174
Yashasvi2229 merged 9 commits into
mainfrom
fix/graph-hard-aborts-and-unresolved-refs

Conversation

@Yashasvi2229

Copy link
Copy Markdown
Collaborator

What

Five changes to the code graph, in the order they were built:

  1. A file the corpus policy will not read is skipped, not fatal. A single
    oversized file used to abort the entire build, and the loop breaks so
    remaining files were never attempted. It is now recorded with zero
    nodes/edges and a per-path warning, and the walk continues. Corpus-wide
    ceilings still fail the build, because there is no sensible partial answer to
    those.
  2. A config input outside the project root is declined, not fatal. For a
    source file, out-of-root was already a soft miss; for a tsconfig, the
    identical check threw and killed the build. configFileExists now returns
    false and readConfigFile returns undefined, with the declined
    specifier reported. The containment guard is unchanged — this only changes
    what happens when it declines.
  3. A repository can add its own corpus ignore globs, additive to the built-in
    defaults, which stay non-overridable.
  4. mex graph query who-calls falls back to recorded unresolved references
    when a name has call sites but no indexed declaration, emitted as a distinct
    record type, budgeted and capped.
  5. References the resolver bound are no longer stored. They became edges;
    keeping them stored a second copy.

Plus: evaluate/RESULTS.md is now linked from the READMEs.

Why

Closes #115. References #140

#115 raised three items. Two were already fixed in 0.7.2 (graph scope
stopword filtering) and by the benchmark rework (the README's grep comparison
claim). The third — who-calls returning TARGET_NOT_FOUND with no recovery
path, while the call sites sat in an internal table no command exposed — is
fixed here. The reporter's own blind eval is the argument for it: on
call-site enumeration mex answered 23/23 where a regex answered 15/23, and
reaching that answer required hand-written SQL against internal schema.

The two hard aborts were found while measuring seven real repositories for
this work. Two of them could not be indexed at all, each because of one
file: a 34.4 MB generated source in a 3,259-file TypeScript monorepo, and a
tsconfig extending a package hoisted to a parent node_modules in a
517-file repository. Both are ordinary — hoisting is normal in any pnpm/yarn
workspace, and generated blobs are normal anywhere.

Two further defects surfaced while fixing those:

  • The per-file limit name was selected by comparing byte values. The source and
    config per-file ceilings are numerically identical, so every oversized
    source file mex has ever rejected was reported as a config-limit breach.
  • Freshness inspection in status.ts had the same break-on-limit shape as
    indexing. Left alone, status and indexing would have disagreed about the
    corpus permanently, and a repository with one oversized file could never read
    fresh.

The storage change is the cheap, safe half of the #140 follow-up. Measured
across five repositories, unresolved_refs and its indexes were 32–42% of
every store, and rows with status = 'resolved' were 27–61% of the table with
100% of them already present as an edge on every store checked. edges is
a superset, so nothing is recoverable from those rows that is not already
there.

Type of change

  • Bug fix
  • New feature
  • Refactor
  • Docs
  • CI/Tooling

How to test

The two builds that used to be impossible

  1. Point mex graph at a repository containing a source file above
    maxSourceFileBytes. Before: the whole build fails with
    GRAPH_CORPUS_LIMIT_EXCEEDED. After: it completes, and the file is reported
    as skipped with its size and the limit.
  2. Point mex graph at a repository whose tsconfig.json extends a package
    resolved outside the project root. Before: the whole build fails with
    GRAPH_SOURCE_PATH_ESCAPE and filePath: ".". After: it completes and the
    declined config input is reported by its dependency specifier.

The fallback

  1. mex graph query who-calls <name> for a name with call sites but no indexed
    declaration. Before: {"type":"error","code":"TARGET_NOT_FOUND"}. After:
    labelled unresolved records with file, line and column, under budget.

No graph content changed

  1. Build the same repository on main and on this branch and compare. Node and
    edge counts, per-kind breakdown, and a content hash over
    (source, target, kind, line, col) are identical. Only the store shrinks.

Storage

  1. SELECT name, SUM(pgsize) FROM dbstat GROUP BY name before and after.

Checklist

  • Tests pass (npm test) — see note below
  • No breaking changes (or documented below)
  • Tested locally with a real project

Notes:

  • Verified on a mid-size repository: node and edge content hashes identical
    across main and this branch; whole store −16.7%; unresolved_refs plus its
    indexes −39.9%.
  • npm run eval could not be used as a gate. It fails identically on clean
    main with a RangeError reading lsh_buckets in
    evaluate/graph/lib/integrity.mjs — the v3 band hashes exceed
    Number.MAX_SAFE_INTEGER and node:sqlite refuses to return them as
    numbers. Confirmed independently: 29,619 of 29,664 rows on a real store are
    out of range. Pre-existing and unrelated to this PR; it needs its own issue.
    npm run eval:graph -- --validate passes.
  • Two test expectations changed because they encoded the defects being fixed,
    not invariants the fix broke. Both are called out in the diff with the
    reasoning, and both now assert the underlying property directly — in
    particular, that an external tsconfig is never read and never becomes graph
    provenance.

Code-graph changes

  • This PR targets main
  • A linked issue agrees on the bounded extractor/resolver scope
  • The change follows the frozen LanguageExtractor or FrameworkResolver interface
  • A focused fixture and assertions for the expected node/edge shape are included
  • Any new grammar WASM, extension mapping, extractor, or resolver is registered — n/a, none added
  • No graph identity, reconciliation, schema, or drift-semantics changes are included

…orting

A single oversized file could abort an entire graph build. The per-file
limit error was pushed as a staging failure, the discovery loop then broke
so no later file was attempted, and the accumulated failures were thrown.
On a 3,259-file TypeScript monorepo one 34 MB generated source made the
whole repository un-indexable, with no way to exclude it short of editing
the tree.

A per-file ceiling (maxSourceFileBytes, maxConfigFileBytes) now records the
file and continues; corpus-wide ceilings still abort, because they describe
the whole run and have no honest partial answer. The skip happens at the
single discovery seam every consumer shares, so the staged corpus,
publication verification, sync's corpus comparison and the freshness
inspector all agree about which files exist. Build results carry the
skipped files so a user learns why a symbol is missing from the graph.

Two supporting fixes:

  * The limit name in the error was inferred by comparing the byte ceiling
    against maxConfigFileBytes. The source and config per-file limits are
    numerically identical, so every oversized source file was reported as a
    config-limit breach. The name is now passed in, and the message states
    the observed size and the limit on its first line.

  * Freshness inspection had the same break-on-limit shape and would stop
    walking the corpus at the first oversized file. It now skips that file
    with a bounded per-path diagnostic and keeps the observation complete,
    matching what indexing does.

The containment and bounded-work guards are unchanged. This changes only
what happens when a guard declines, never whether it declines.
The eight built-in ignore globs were frozen with no user configuration
anywhere, so a repository containing one file the graph could not index had
no way to exclude it short of editing its own source tree.

Globs listed under "graph.ignore" in .mex/config.json are now appended to
the built-in list. Additive only: a configuration cannot un-ignore
node_modules or .mex, because the defaults are the floor rather than a
default value to replace. The list is read defensively and bounded — a
missing, oversized or malformed config contributes no extra globs rather
than trading one hard abort for another.

Configured globs enter the corpus policy hash, so changing them invalidates
the index and forces an explicit rebuild. A repository that configures
nothing hashes exactly as before, leaving existing indexes valid.

All four corpus walks — indexing, config discovery, freshness inspection and
the changed-file scan — now resolve the same per-repository list. Skipped
files are reported by mex graph, with the size, the limit, and the config
key that would exclude the path deliberately.
One `tsconfig.json` extending a package hoisted above the project root made
a 517-file repository completely un-indexable. `readFile` had always treated
an out-of-root source path as a soft miss; `readConfigFile` and
`configFileExists` threw on the identical condition. The two halves of one
class simply disagreed, and the thrown error surfaced as a staging failure
against filePath "." — naming no useful path at all.

`configFileExists` is a probe. TypeScript is asking whether a file is there,
and the honest answer for a path we will not read is `false`, not an
exception. `readConfigFile` returns undefined for the same reason.

The two sibling guards are treated the same way, because they are the same
defect: a tsconfig `include` above the root now matches no files, and a
project reference above the root is not traversed. A monorepo sub-package
routinely references its siblings and must not become un-indexable for it.

Every decline is recorded and reported, so a repository whose type graph is
less complete than its config asks for is told. Paths are reported by
dependency specifier rather than absolutely: TypeScript resolves a bare
`extends` by walking every ancestor directory, so one unresolvable specifier
otherwise produced a dozen near-identical entries carrying absolute paths
from outside the repository, including a user's home directory.

The containment guard is unchanged and still declines. Nothing outside the
project root is read, and nothing outside it enters graph provenance — now
asserted directly rather than implied by an abort.
Reported in #115. A dynamically generated method has real call sites and no
literal declaration, so no node resolves and who-calls answered
TARGET_NOT_FOUND with no next step. The call sites were not lost — they were
in unresolved_refs, which had no CLI or documentation surface anywhere.
Recovering them required hand-written SQL against internal schema.

who-calls now looks its target up among the recorded unresolved references
before giving up, and reports those call sites with file, line, column and
the node they were seen in.

They are emitted as a distinct `unresolved-reference` record, never as
`type: "result"`. An unresolved reference is not a resolved graph fact and
an agent must not be able to confuse the two. They are charged to the same
budget ledger as everything else and capped by maxNodes — common names
accumulate hundreds of references and an uncapped fallback on a hot name
would flood the caller — with the summary reporting the total that matched
alongside what was returned.

This path also emits proper meta and summary records, where it previously
emitted a single bare error with neither, inconsistent with the rest of the
protocol. status is "partial" and evidenceStrength is "weak", because that
is what this evidence is.

Scope is deliberately narrow. A name with neither a declaration nor a
recorded reference still abstains with TARGET_NOT_FOUND, and where-defined,
what-calls and impact are unchanged: they either resolve the requested
declaration exactly or abstain.

resolveSymbol still refuses fuzzy matching. This adds a second question
before giving up, not a looser answer to the first.
… edges

References #140. Once schema v3 shrank everything else, unresolved_refs
became the largest object in the store — 32-42% of it across every
repository measured, with its secondary indexes costing about as much as
the table.

Three changes, in increasing order of risk.

Drop the narrow (from_node_id) index. It is a strict prefix of the
(from_node_id, reference_name) composite, and EXPLAIN QUERY PLAN on a real
store confirms SQLite serves every lookup it used to from the composite —
the ON DELETE CASCADE probe included, still as a covering search rather
than a scan.

Make the status index partial.

Stop writing rows for references that resolved. Every such row duplicated
an edge: measured at 100% on fresh builds and existing stores alike, and
27-73% of the table. `edges` is a superset — it also holds structural
`contains` edges with no reference row — so nothing is recoverable from
here that is not already there. Resolution happens in memory during
staging, and publication clears and rewrites the whole derived graph, so no
read path consumed these rows.

Three store methods described a path that does not run: an incremental sync
that wipes reference edges and rebuilds them from unresolved_refs. None of
them had a caller anywhere in the tree. They are removed rather than left
as a comment contradicting what the code now does.

Measured on two repositories, per-object via dbstat:

  99-file TS/TSX/Python/JS   12.34 MB -> 10.29 MB   (-16.6%)
  366-file Python-heavy      65.07 MB -> 56.44 MB   (-13.3%)

with the table and its indexes down about 40% on both. No schema version
bump: the changes are index definitions and which rows are written, both
compatible with an existing store, which reclaims the space on its next
rebuild.
evaluate/RESULTS.md carries the blind-graded comparison against a
file-search baseline and was linked from nowhere — not the top-level
README, not evaluate/README.md. It is the evidence answering the benchmark
half of #115, so it should be reachable from both.
Both hard aborts fixed on this branch came from the same two mistakes:
treating a per-file ceiling as a corpus-wide one, and letting a containment
guard's refusal be an exception rather than an answer. Record them where the
next graph indexing change will read them.
The Explore further list in the Spanish, Portuguese and Chinese READMEs
mirrors the English one, so the benchmark link belongs in all four rather
than only the English entry point.

Also drops the who-calls fallback paragraph from the English README. The
behaviour stays documented in docs/code-graph-support.md, which is where
the detail belongs.
CI caught this on Linux: the filter used path.isAbsolute, so "C:/build/**"
was rejected on Windows and accepted on Linux.

That is not only a test discrepancy. .mex/config.json is tracked and travels
with the repository, and these globs feed the corpus policy hash — so one
repository would hash to two different manifests depending on the machine
that opened it, and the index would read as stale purely from moving between
platforms.

The check is now explicit and platform-free: reject a leading "/" (POSIX
absolute and "//server/share" UNC alike), a Windows drive prefix in either
its absolute or drive-relative form, and any ".." path segment. Upward
traversal is now caught anywhere in the glob rather than only at the front,
so "vendor/../../escape/**" no longer slips through. Globs that merely
contain dots, such as "a..b/**" and "**/*.min.js", are unaffected.
@Yashasvi2229
Yashasvi2229 merged commit faa6416 into main Sep 7, 2026
9 checks passed
@Yashasvi2229
Yashasvi2229 deleted the fix/graph-hard-aborts-and-unresolved-refs branch September 7, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

graph scope stopword pollution, undocumented unresolved_refs fallback, and benchmark vs. grep gap (with repro + eval data)

1 participant