Skip to content

[mache-b094fa] Lock Open's laziness, and close the ids-vs-nodes asymmetry - #628

Merged
jamestexas merged 3 commits into
mainfrom
feat/mache-b094fa-lazy-and-nodes
Aug 21, 2026
Merged

[mache-b094fa] Lock Open's laziness, and close the ids-vs-nodes asymmetry#628
jamestexas merged 3 commits into
mainfrom
feat/mache-b094fa-lazy-and-nodes

Conversation

@jamestexas

@jamestexas jamestexas commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Completes mache-b094fa. Follows #627 (RefRange + classified RefTarget).

1. graph.Open must stay lazy — now enforced

modmap assembles a ~100 GB corpus by ATTACH+INSERT across projections. That only works because Open is SQL-per-call. Its own doc already promised it — "Open sidesteps the whole class by not copying anything" — and nothing enforced it. A bulk load added later would break the assembled path silently, at a scale no unit test reaches.

Two assertions, because one is not enough:

  • scanOnce/dirChildren/recordIDs empty after Open. Populated exclusively by scanRoot, so emptiness directly observes that no scan ran. Paired with a LookupDef call so "lazy" cannot be satisfied by an Open that does nothing.
  • Open's heap cost does not grow with projection size. Catches what the above cannot: a DefsMap warm-up, an index build, a bulk read into the reader cache.

The first version of the second test was wrong

It compared large < small*8 and PASSED with an eager g.DefsMap() spliced into Open: at 400 defs the copy hid inside sql.Open's ~240 KB fixed cost. Ratios are the wrong instrument when a fixed cost dominates. Measured absolutely instead:

delta, 200 → 2000 defs
lazy 0 B
eager whole-index copy 611,136 B

The 100 KB ceiling sits far above the real measurement and ~6x below the cheapest regression it must catch. Falsified by mutation — which the earlier version survived.

2. LookupDefNodes closes the asymmetry

LookupDef(token)  -> []string          // ids
GetCallers(token) -> ([]*Node, error)  // nodes
GetCallees(id)    -> ([]*Node, error)  // nodes

Ids from one accessor and Nodes from the next is why modmap wrapped this surface in its own callerResolver; their read — "strings where types should be" — was correct.

LookupDef is deliberately unchanged: ids are the right answer when the caller only needs identity, and resolving nodes it will discard is waste. This adds the variant for when it wants the node, as graph.DefsNodeLookuper.

A definition whose node cannot load is skipped rather than failing the call — node_defs can outlive a node during an incremental reparse, and one stale row should not deny a caller the definitions that did resolve.

The test pins symmetry, not shape: both accessors must agree on which definitions exist and differ only in what they return, so a consumer switching between them cannot silently change its answer.

A correction to ADR-0027 (recorded on mache-40faff)

EagerScan is a no-op on the nodes-table path:

if g.useNodesTable { return nil }   // graph/sqlite_graph.go:113

useNodesTable is true for everything mache build produces, so mount's EagerScan call does nothing for modern projections and the scanRoot machinery serves only the legacy path. The short-circuit is correct; what is not is ADR-0027 listing EagerScan as a live capability axis worth ablating — on the default path it cannot vary, so an ablation matrix would measure a constant. Recommend dropping it from the axis list in the ADR's revision. Pinned by a test so the fact cannot drift.

3. [mache-93e84b] — the linter's node-id shape assumption, isolated

Rides along because it landed on this branch. Independent of the two above.

While preparing mache's answers to ley-line-open's integer-nid migration I claimed internal/linter/linter.go would silently match across files under the new keys. That was wrong, and I retracted it on ley-line-open-17c271. LintAST builds its own in-memory _ast from a single-file ASTPayload — no source_id column exists, so cross-file matching was never reachable. I read the SQL and inferred the table instead of reading the table.

The real fragility is narrower but genuine: parent/child containment was reconstructed from the node-id string via three substr/instr expressions spread across the rule SQL. That is a shape assumption in a place where a shape change makes rules silently stop matching — a linter reporting nothing rather than erroring.

Now the table carries an explicit parent_id, the SQL reads c.parent_id = v.node_id, and the sole remaining assumption lives in one named function:

func parentIDOf(nodeID string) string {
	i := strings.LastIndexByte(nodeID, '/')
	if i <= 0 { return "" }
	return nodeID[:i]
}

TestParentIDOf_IsTheOnlyNodeIDShapeAssumption pins that. Falsified by mutating LastIndexByteIndexByte, which the existing linter tests catch.

Verification

task ci passed via the pre-push gate; task smells returns 0. Every new assertion falsified by mutation. The smell ratchet additionally caught two structurally-identical test functions of mine and was right — grouped into one subtest-based test naming the shared contract.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TtGhz7QzUHZi52a3FeNtEs

…depends on it

modmap assembles a ~100 GB corpus by ATTACH+INSERT across projections. That
only works because graph.Open is SQL-per-call. Open's own doc promises it
("Open sidesteps the whole class by not copying anything") and nothing
enforced it — a bulk load added later would break the assembled path silently,
and only at a scale no unit test reaches.

Two assertions, because one is not enough:

  1. scanOnce/dirChildren/recordIDs are empty after Open. These are populated
     EXCLUSIVELY by scanRoot, so emptiness is a direct observation that no
     scan ran. Paired with a LookupDef call, so "lazy" cannot be satisfied by
     an Open that simply does nothing.

  2. Open's heap cost does not grow with projection size. This catches what
     (1) cannot: a DefsMap warm-up, an index build, a bulk read into the
     reader cache — none of which touch those maps.

The second assertion needed two attempts, and the first version was WRONG in
an instructive way. It compared large < small*8 and PASSED with an eager
g.DefsMap() spliced into Open: at 400 defs the copy hid inside sql.Open's
~240 KB fixed cost. Ratios are the wrong instrument when a fixed cost
dominates. Measured absolutely instead:

    lazy       200 -> 2000 defs :  delta =       0 B
    eager copy 200 -> 2000 defs :  delta = 611,136 B

so the 100 KB ceiling sits far above the real measurement and 6x below the
cheapest regression it must catch. Verified by mutation, which the earlier
version survived.

Also pins a fact that surprised the test into existence: EagerScan is a NO-OP
on the nodes-table path, which is what `mache build` produces. It short-
circuits on useNodesTable, so mount's EagerScan call does nothing for modern
projections and the scanRoot machinery serves only the legacy path. The no-op
is correct; what is not is ADR-0027 listing EagerScan as a live capability
axis worth ablating. Correction recorded on mache-40faff.
…symmetry

The ref surface returned two different things depending on which accessor you
reached for:

    LookupDef(token)  -> []string          // ids
    GetCallers(token) -> ([]*Node, error)  // nodes
    GetCallees(id)    -> ([]*Node, error)  // nodes

so every consumer that wanted nodes from a token lookup wrote the same
conversion loop. modmap wrapped the whole surface in its own callerResolver
for exactly this; their read was "strings where types should be", which was
correct.

LookupDef is deliberately unchanged. Ids ARE the right answer when the caller
only needs identity, and resolving nodes it will discard is waste. This adds
the variant for when it actually wants the node, exposed as
graph.DefsNodeLookuper alongside the other capability interfaces.

A definition whose node cannot be loaded is skipped rather than failing the
call: node_defs can outlive a node during an incremental reparse, and one
stale row should not deny a caller the definitions that did resolve.

The test pins SYMMETRY, not just shape — the two accessors must agree on which
definitions exist and differ only in what they hand back, so a consumer
switching between them cannot silently change its answer. Falsified by
dropping the append, which the symmetry assertion catches.
… to one function

The nil-slice rule expressed DIRECT-CHILD as three substr/instr expressions
over the node id, relying on the daemon's hierarchical path encoding (child id
= parent id + "/" + segment). It now reads an explicit parent_id column, and
the path assumption lives in one named Go function.

Why: ley-line-open-17c271 proposes replacing the path with an integer key.
Spread through SQL, that change would have made these rules silently stop
matching — the worst failure mode for a linter, which reports nothing rather
than erroring. Isolated in parentIDOf, it fails in one obvious place with a
test that names it as the sole assumption.

The table is mache's own (LintAST builds an in-memory _ast from a single
emit_ast payload), so adding the column costs nothing and needs nothing from
upstream.

Also corrects a claim I made TO ley-line-open and have now retracted there: I
said this query lacked a source_id predicate and would start matching across
files under integer keys. False — the in-memory table has no source_id column
and holds exactly ONE file's parse, so cross-file matching was never
reachable. I read the SQL and inferred the table instead of reading the table.

The real dependency is narrower and more useful to them: depth-1 containment
is not derivable from byte spans, because a parent sharing its only child's
span is indistinguishable by >=/<= alone. That is an independent argument for
materialising parent_nid, and it is the strongest one available.

Falsified by mutating LastIndexByte to IndexByte, which the existing linter
tests catch.
@jamestexas
jamestexas merged commit a5ebdb8 into main Aug 21, 2026
19 checks passed
@jamestexas
jamestexas deleted the feat/mache-b094fa-lazy-and-nodes branch August 21, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant