daemon: deferred passes lost when a cold warmup dies after the bulk flush — next start sees repos unchanged and never re-arms them

[pbednarcik]

On the first-ever cold start over an empty store, my daemon was killed between the coordinated bulk flush and RunDeferredPassesAll (routine restart, ~2 minutes into warmup, mid-resolve). The next start's mtime reconcile then saw every cold-tracked repo as unchanged — and the whole deferred-pass family (semantic enrichment, DI/contract synthesis, the is_test/partial stamp sweeps) never ran for those repos, silently and durably.

What I observed

Workspace of 4 tracked repos, fresh empty store, all repos in config:

• Start 1 (cold): parallel_parse fully tracks everything (largest repo 55.8s, index_seal ... nodes=288227 edges=2281004), changed_repos=4. The bulk load flushes nodes, edges, and the file-mtime sidecar to sqlite at the end of the parse phase. The process is killed ~3s into the resolve phase (resolver at 2048/16384 pending). No deferred pass has run; no enrichment completion marker was written.
• Start 2 (over the same store, ~100s later): reconcile routes the two untouched repos through census_noop (changed=0), two others pick up 1 stale file each → changed_repos=2. deferred_passes_all drains in 0.33s — only the 2 stale files' work. For the census_noop repos, nothing re-arms: csharp-types enrichment, the DI/contract synthesis pass, and the is_test/partial stamp sweeps are simply gone.

My counted C# fixture battery quantified it: 16 FAILs, all in exactly that pass family. An explicit index_repository re-parse re-armed everything and dropped it to 3 (those under separate watch). So the workaround exists, but nothing in the daemon detects the hole.

Root cause

The durable/volatile split in the arming state doesn't survive a kill at this point in warmup:

1. The parse phase's coordinated bulk load persists nodes, edges, and file mtimes before the deferred passes run. From the store's perspective the repo now looks fully indexed and current.
2. The arming for the deferred passes is in-memory only on this path: pendingEnrich (atomic) and pendingContractReg die with the process. The durable per-file ledger (graph.MetaReparsePendingEnrichment) is stamped only by the incremental/watch paths (incremental_batch.go) — a batch-deferred cold TrackRepoCtx never stamps it.
3. On restart, SeedPendingEnrichAll → MaybeSeedPendingEnrich is the only resume net, and both of its signals come up empty:
   • a tracked directory that is not a git repository: "no git head" → falls to the ledger → empty → decline (my fixture repo's case);
   • a git repo whose tree is dirty at restart: completion marker absent (the killed run never wrote one) but untrustworthy on a dirty tree → falls to the ledger → empty → decline (my production repo's case — a working tree is realistically always dirty). The decline is Debug-level, so it's invisible at default log level.
4. The DI/contract synthesis pass and the graph-wide derivation sweeps have no durable arming at all — they only run when parse work happened in the current process — so even a repo that would pass the enrichment gates loses those outright.

The dirty-tree decline itself is correct (pinned by TestMaybeSeedPendingEnrich_DirtyTreeNotSeeded — resuming every restart on a dirty tree would defeat the warm fast path). The hole is that a cold track leaves no durable trace for the ledger branch to resume from.

Repro (red on main @ 5f1e2df, Windows)

Two tests, using the existing deferred_enrich_*_test.go harness — a warmup-shaped cold index (SetDeferGlobalPasses(true) + SetDeferResolve(true), what BeginParallelBatch applies), dropped without running deferred passes, then a fresh Indexer over the same sqlite store:

// coldTrackThenDie runs the warmup-shaped cold index of repo into store —
// deferGlobalPasses + deferResolve, exactly what BeginParallelBatch applies to
// every per-repo Indexer — and returns without running any deferred pass,
// modelling a daemon killed between the coordinated bulk flush and
// RunDeferredPassesAll.
func coldTrackThenDie(t *testing.T, store graph.Store, repo, prefix string, spy *spyEnrichProvider) {
	t.Helper()
	idx := New(store, newTestRegistry(), config.Default().Index, zap.NewNop())
	idx.SetRepoPrefix(prefix)
	idx.SetSemanticManager(newSpyManager(spy))
	idx.SetDeferGlobalPasses(true)
	idx.SetDeferResolve(true)
	_, err := idx.Index(repo)
	require.NoError(t, err)
	require.Empty(t, spy.invoked(),
		"the batch-deferred cold track must not run enrichment inline")
	require.True(t, idx.pendingEnrich.Load(),
		"the cold track arms the in-memory gate — which dies with the process")
	// Process killed here: idx is dropped, no deferred pass ever runs.
}

func TestMaybeSeedPendingEnrich_ResumesInterruptedColdTrack_NonGit(t *testing.T) {
	repo := t.TempDir() // deliberately NOT a git repository
	writeFile(t, filepath.Join(repo, "main.go"),
		"package main\n\nfunc main() { helper() }\n\nfunc helper() {}\n")
	store := openTestSqlite(t)

	coldTrackThenDie(t, store, repo, "r", &spyEnrichProvider{})

	spy2 := &spyEnrichProvider{}
	idx2 := New(store, newTestRegistry(), config.Default().Index, zap.NewNop())
	idx2.SetRepoPrefix("r")
	idx2.SetRootPath(repo)
	idx2.SetSemanticManager(newSpyManager(spy2))

	assert.True(t, idx2.MaybeSeedPendingEnrich(),
		"an interrupted cold track left enrichment un-run for every file — the restart must resume it")
}

func TestMaybeSeedPendingEnrich_ResumesInterruptedColdTrack_DirtyGit(t *testing.T) {
	repo := commitGitRepo(t)
	store := openTestSqlite(t)

	coldTrackThenDie(t, store, repo, "r", &spyEnrichProvider{})

	// The tree goes dirty before the next start (any uncommitted edit).
	writeFile(t, filepath.Join(repo, "extra.go"), "package main\n\nvar Extra = 1\n")

	spy2 := &spyEnrichProvider{}
	idx2 := New(store, newTestRegistry(), config.Default().Index, zap.NewNop())
	idx2.SetRepoPrefix("r")
	idx2.SetRootPath(repo)
	idx2.SetSemanticManager(newSpyManager(spy2))

	assert.True(t, idx2.MaybeSeedPendingEnrich(),
		"a dirty tree must not hide an interrupted cold track — the deferred passes never ran at this store's content")
}

Both fail on main today:

--- FAIL: TestMaybeSeedPendingEnrich_ResumesInterruptedColdTrack_NonGit (0.07s)
--- FAIL: TestMaybeSeedPendingEnrich_ResumesInterruptedColdTrack_DirtyGit (0.19s)

Possible fix directions

Two shapes, smallest first — happy to PR whichever you prefer:

1. Stamp the durable per-file ledger during a batch-deferred cold track (mirroring what the incremental paths already do via reparsePendingEnrichmentBatch). The restart then resumes file-scoped through the existing ledger branch, which is deliberately git-state-independent, so both the non-git and dirty-tree cases heal. Covers enrichment only.
2. A repo-level durable "deferred passes outstanding" marker, written with the bulk flush and cleared only after the deferred + derivation tail completes for that repo. Covers the whole family — enrichment and contracts/DI and the stamp sweeps — and turns the invariant into "a store never looks current while any pass family is outstanding".

(1) is a targeted patch to the exact machinery that already exists; (2) closes the contracts/derivation half too, which (1) leaves open.