fix(graph): survive un-indexable files, surface unresolved references, and stop storing references the resolver already turned into edges - #174
Merged
Conversation
…orting
A single oversized file could abort an entire graph build. The per-file
limit error was pushed as a staging failure, the discovery loop then broke
so no later file was attempted, and the accumulated failures were thrown.
On a 3,259-file TypeScript monorepo one 34 MB generated source made the
whole repository un-indexable, with no way to exclude it short of editing
the tree.
A per-file ceiling (maxSourceFileBytes, maxConfigFileBytes) now records the
file and continues; corpus-wide ceilings still abort, because they describe
the whole run and have no honest partial answer. The skip happens at the
single discovery seam every consumer shares, so the staged corpus,
publication verification, sync's corpus comparison and the freshness
inspector all agree about which files exist. Build results carry the
skipped files so a user learns why a symbol is missing from the graph.
Two supporting fixes:
* The limit name in the error was inferred by comparing the byte ceiling
against maxConfigFileBytes. The source and config per-file limits are
numerically identical, so every oversized source file was reported as a
config-limit breach. The name is now passed in, and the message states
the observed size and the limit on its first line.
* Freshness inspection had the same break-on-limit shape and would stop
walking the corpus at the first oversized file. It now skips that file
with a bounded per-path diagnostic and keeps the observation complete,
matching what indexing does.
The containment and bounded-work guards are unchanged. This changes only
what happens when a guard declines, never whether it declines.
The eight built-in ignore globs were frozen with no user configuration anywhere, so a repository containing one file the graph could not index had no way to exclude it short of editing its own source tree. Globs listed under "graph.ignore" in .mex/config.json are now appended to the built-in list. Additive only: a configuration cannot un-ignore node_modules or .mex, because the defaults are the floor rather than a default value to replace. The list is read defensively and bounded — a missing, oversized or malformed config contributes no extra globs rather than trading one hard abort for another. Configured globs enter the corpus policy hash, so changing them invalidates the index and forces an explicit rebuild. A repository that configures nothing hashes exactly as before, leaving existing indexes valid. All four corpus walks — indexing, config discovery, freshness inspection and the changed-file scan — now resolve the same per-repository list. Skipped files are reported by mex graph, with the size, the limit, and the config key that would exclude the path deliberately.
One `tsconfig.json` extending a package hoisted above the project root made a 517-file repository completely un-indexable. `readFile` had always treated an out-of-root source path as a soft miss; `readConfigFile` and `configFileExists` threw on the identical condition. The two halves of one class simply disagreed, and the thrown error surfaced as a staging failure against filePath "." — naming no useful path at all. `configFileExists` is a probe. TypeScript is asking whether a file is there, and the honest answer for a path we will not read is `false`, not an exception. `readConfigFile` returns undefined for the same reason. The two sibling guards are treated the same way, because they are the same defect: a tsconfig `include` above the root now matches no files, and a project reference above the root is not traversed. A monorepo sub-package routinely references its siblings and must not become un-indexable for it. Every decline is recorded and reported, so a repository whose type graph is less complete than its config asks for is told. Paths are reported by dependency specifier rather than absolutely: TypeScript resolves a bare `extends` by walking every ancestor directory, so one unresolvable specifier otherwise produced a dozen near-identical entries carrying absolute paths from outside the repository, including a user's home directory. The containment guard is unchanged and still declines. Nothing outside the project root is read, and nothing outside it enters graph provenance — now asserted directly rather than implied by an abort.
Reported in #115. A dynamically generated method has real call sites and no literal declaration, so no node resolves and who-calls answered TARGET_NOT_FOUND with no next step. The call sites were not lost — they were in unresolved_refs, which had no CLI or documentation surface anywhere. Recovering them required hand-written SQL against internal schema. who-calls now looks its target up among the recorded unresolved references before giving up, and reports those call sites with file, line, column and the node they were seen in. They are emitted as a distinct `unresolved-reference` record, never as `type: "result"`. An unresolved reference is not a resolved graph fact and an agent must not be able to confuse the two. They are charged to the same budget ledger as everything else and capped by maxNodes — common names accumulate hundreds of references and an uncapped fallback on a hot name would flood the caller — with the summary reporting the total that matched alongside what was returned. This path also emits proper meta and summary records, where it previously emitted a single bare error with neither, inconsistent with the rest of the protocol. status is "partial" and evidenceStrength is "weak", because that is what this evidence is. Scope is deliberately narrow. A name with neither a declaration nor a recorded reference still abstains with TARGET_NOT_FOUND, and where-defined, what-calls and impact are unchanged: they either resolve the requested declaration exactly or abstain. resolveSymbol still refuses fuzzy matching. This adds a second question before giving up, not a looser answer to the first.
… edges References #140. Once schema v3 shrank everything else, unresolved_refs became the largest object in the store — 32-42% of it across every repository measured, with its secondary indexes costing about as much as the table. Three changes, in increasing order of risk. Drop the narrow (from_node_id) index. It is a strict prefix of the (from_node_id, reference_name) composite, and EXPLAIN QUERY PLAN on a real store confirms SQLite serves every lookup it used to from the composite — the ON DELETE CASCADE probe included, still as a covering search rather than a scan. Make the status index partial. Stop writing rows for references that resolved. Every such row duplicated an edge: measured at 100% on fresh builds and existing stores alike, and 27-73% of the table. `edges` is a superset — it also holds structural `contains` edges with no reference row — so nothing is recoverable from here that is not already there. Resolution happens in memory during staging, and publication clears and rewrites the whole derived graph, so no read path consumed these rows. Three store methods described a path that does not run: an incremental sync that wipes reference edges and rebuilds them from unresolved_refs. None of them had a caller anywhere in the tree. They are removed rather than left as a comment contradicting what the code now does. Measured on two repositories, per-object via dbstat: 99-file TS/TSX/Python/JS 12.34 MB -> 10.29 MB (-16.6%) 366-file Python-heavy 65.07 MB -> 56.44 MB (-13.3%) with the table and its indexes down about 40% on both. No schema version bump: the changes are index definitions and which rows are written, both compatible with an existing store, which reclaims the space on its next rebuild.
evaluate/RESULTS.md carries the blind-graded comparison against a file-search baseline and was linked from nowhere — not the top-level README, not evaluate/README.md. It is the evidence answering the benchmark half of #115, so it should be reachable from both.
Both hard aborts fixed on this branch came from the same two mistakes: treating a per-file ceiling as a corpus-wide one, and letting a containment guard's refusal be an exception rather than an answer. Record them where the next graph indexing change will read them.
The Explore further list in the Spanish, Portuguese and Chinese READMEs mirrors the English one, so the benchmark link belongs in all four rather than only the English entry point. Also drops the who-calls fallback paragraph from the English README. The behaviour stays documented in docs/code-graph-support.md, which is where the detail belongs.
CI caught this on Linux: the filter used path.isAbsolute, so "C:/build/**" was rejected on Windows and accepted on Linux. That is not only a test discrepancy. .mex/config.json is tracked and travels with the repository, and these globs feed the corpus policy hash — so one repository would hash to two different manifests depending on the machine that opened it, and the index would read as stale purely from moving between platforms. The check is now explicit and platform-free: reject a leading "/" (POSIX absolute and "//server/share" UNC alike), a Windows drive prefix in either its absolute or drive-relative form, and any ".." path segment. Upward traversal is now caught anywhere in the glob rather than only at the front, so "vendor/../../escape/**" no longer slips through. Globs that merely contain dots, such as "a..b/**" and "**/*.min.js", are unaffected.
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Five changes to the code graph, in the order they were built:
oversized file used to abort the entire build, and the loop
breaks soremaining files were never attempted. It is now recorded with zero
nodes/edges and a per-path warning, and the walk continues. Corpus-wide
ceilings still fail the build, because there is no sensible partial answer to
those.
source file, out-of-root was already a soft miss; for a
tsconfig, theidentical check threw and killed the build.
configFileExistsnow returnsfalseandreadConfigFilereturnsundefined, with the declinedspecifier reported. The containment guard is unchanged — this only changes
what happens when it declines.
defaults, which stay non-overridable.
mex graph query who-callsfalls back to recorded unresolved referenceswhen a name has call sites but no indexed declaration, emitted as a distinct
record type, budgeted and capped.
keeping them stored a second copy.
Plus:
evaluate/RESULTS.mdis now linked from the READMEs.Why
Closes #115. References #140
#115 raised three items. Two were already fixed in 0.7.2 (
graph scopestopword filtering) and by the benchmark rework (the README's grep comparison
claim). The third —
who-callsreturningTARGET_NOT_FOUNDwith no recoverypath, while the call sites sat in an internal table no command exposed — is
fixed here. The reporter's own blind eval is the argument for it: on
call-site enumeration mex answered 23/23 where a regex answered 15/23, and
reaching that answer required hand-written SQL against internal schema.
The two hard aborts were found while measuring seven real repositories for
this work. Two of them could not be indexed at all, each because of one
file: a 34.4 MB generated source in a 3,259-file TypeScript monorepo, and a
tsconfigextending a package hoisted to a parentnode_modulesin a517-file repository. Both are ordinary — hoisting is normal in any pnpm/yarn
workspace, and generated blobs are normal anywhere.
Two further defects surfaced while fixing those:
config per-file ceilings are numerically identical, so every oversized
source file mex has ever rejected was reported as a config-limit breach.
status.tshad the same break-on-limit shape asindexing. Left alone, status and indexing would have disagreed about the
corpus permanently, and a repository with one oversized file could never read
fresh.
The storage change is the cheap, safe half of the #140 follow-up. Measured
across five repositories,
unresolved_refsand its indexes were 32–42% ofevery store, and rows with
status = 'resolved'were 27–61% of the table with100% of them already present as an edge on every store checked.
edgesisa superset, so nothing is recoverable from those rows that is not already
there.
Type of change
How to test
The two builds that used to be impossible
mex graphat a repository containing a source file abovemaxSourceFileBytes. Before: the whole build fails withGRAPH_CORPUS_LIMIT_EXCEEDED. After: it completes, and the file is reportedas skipped with its size and the limit.
mex graphat a repository whosetsconfig.jsonextends a packageresolved outside the project root. Before: the whole build fails with
GRAPH_SOURCE_PATH_ESCAPEandfilePath: ".". After: it completes and thedeclined config input is reported by its dependency specifier.
The fallback
mex graph query who-calls <name>for a name with call sites but no indexeddeclaration. Before:
{"type":"error","code":"TARGET_NOT_FOUND"}. After:labelled unresolved records with file, line and column, under budget.
No graph content changed
mainand on this branch and compare. Node andedge counts, per-kind breakdown, and a content hash over
(source, target, kind, line, col)are identical. Only the store shrinks.Storage
SELECT name, SUM(pgsize) FROM dbstat GROUP BY namebefore and after.Checklist
npm test) — see note belowNotes:
across
mainand this branch; whole store −16.7%;unresolved_refsplus itsindexes −39.9%.
npm run evalcould not be used as a gate. It fails identically on cleanmainwith aRangeErrorreadinglsh_bucketsinevaluate/graph/lib/integrity.mjs— the v3 band hashes exceedNumber.MAX_SAFE_INTEGERandnode:sqliterefuses to return them asnumbers. Confirmed independently: 29,619 of 29,664 rows on a real store are
out of range. Pre-existing and unrelated to this PR; it needs its own issue.
npm run eval:graph -- --validatepasses.not invariants the fix broke. Both are called out in the diff with the
reasoning, and both now assert the underlying property directly — in
particular, that an external
tsconfigis never read and never becomes graphprovenance.
Code-graph changes
mainLanguageExtractororFrameworkResolverinterface