Failed graph-ingest mutation receipt fail-closes change.detect repo-wide with no reconcile path; reindex does not clear it

[tiendungdev]

Summary

A file mutation whose disk write succeeded but whose graph ingest failed (context deadline exceeded) leaves a graph_status: failed head in the mutation ledger. That head fail-closes change(operation="detect") for the entire repo, from every MCP session, and there is currently no operation that clears or reconciles it:

• full reindex and scoped workspace_admin.reindex(paths=[...]) complete but do not clear the failed head;
• daemon restart does not clear it (the ledger persists in the store);
• a superseding mutation on the same path replaces the head, but its own ingest can fail again under load, re-locking detect;
• a sweep of the MCP facade (change, edit, workspace_admin, session capabilities) and the CLI (gortex --help, gortex edit --help) found no clear/reconcile verb.

Related: #691 (deadline/cancellation asks overlap; this issue is about recovery, not latency).

Environment

• Gortex v0.63.8+2a33f3d, Windows 11, native MCP.
• Single-repo workspace at the time of the detect refusals: 3,058 files, 148,117 nodes, 1,398,981 edges, health_score=100, clean path liveness.
• Daemon under load during the re-failure: 15 connected MCP sessions, FTS doc count fluctuating (post-shrink store churn).

Sequence observed

1. edit_file mutation on a 15.7 KB bash script: disk_status: committed, graph_status: failed, receipt commit-3, caller mutation_id preserved. The bytes on disk are correct (verified by blob SHA).

2. Any change.detect (any scope, any session) now refuses:

   change detection refused a stale graph: graph freshness unavailable:
   failed receipt=mutation-3 repo="..." path="...\scripts\dev\....sh"
   generation=6 error="context deadline exceeded"
   

   Reproduced from a second, freshly oriented MCP session — the state is daemon-owned, not session-owned.

3. Full reindex (from the original session) and scoped workspace_admin.reindex(paths=[<file>]) both completed; change.receipt for the path still reports graph_status: failed.

4. Superseding byte-identical round trip: two edit_file mutations (with base_sha drift guard and physical_evidence), final content bit-identical to the pre-existing state. The detect refusal switched from failed generation=6 to pending generation=9/12 — so superseding works at the ledger level — but after ~10 minutes under load both new generations flipped to failed with context deadline exceeded again, re-locking detect.

Receipt surface observations

• change.receipt by source.path is honest and useful (committed/pending/failed, byte counts, SHAs).
• The key printed in the detect refusal (mutation-3) is an internal ledger key: querying it via options.mutation_id returns no mutation recorded for mutation_id mutation-3. The error text and the receipt API use different id spaces, which sent our tooling down a wrong path during triage.

Requested improvements

1. A reconcile path for a failed ingest head: an explicit clear/reconcile operation (e.g. on change.receipt or workspace_admin), or have a successful re-parse of the file during reindex clear the failed head — today reindex re-parses the file yet the head stays failed.
2. Automatic ingest retry with backoff for failed heads once load drains, instead of requiring a caller-initiated superseding mutation.
3. Degrade instead of hard-refusing in detect: report the affected path(s) as a disclosed coverage gap (or accept an explicit allow_stale option) rather than refusing the whole repo — one 15.7 KB file currently blocks change detection for a 3,000-file repo.
4. Name a queryable id in the refusal: print the receipt id (commit-N) or caller mutation_id rather than the internal ledger key.

Current workaround

None that fully recovers under load. Superseding the head works only if its ingest succeeds; until then agents fall back to targeted impact/tests/guards with known symbol ids and record the detect coverage gap. No raw pprof/source is attached.

[tiendungdev]

Correction after reading the source (upstream main; behavior re-confirmed live on v0.63.8+2a33f3d). Two claims in my report above are wrong, and the real mechanism is narrower than I described:

1. "Daemon restart does not clear it (the ledger persists in the store)" — wrong. The freshness gate scans Server.mutationReceipts, an in-memory sync.Map (internal/mcp/edit_serialization.go), and receipts are not rehydrated at startup. We never actually restarted the daemon after the failed mutation; what had persisted across an earlier restart was workspace membership, which I conflated with the receipt ledger. A restart would clear the gate (at warmup cost).
2. "A superseding mutation on the same path replaces the head" — misattributed. Completed receipts are deleted after mutationReceiptRetention = 10 * time.Minute; the failed generation disappeared because its retention expired, not because of the byte-identical round trip. Consistent with this, change.detect on the same repo now completes normally with no intervention — every failed receipt simply aged out.

What remains, and still seems worth fixing:

• A terminally failed receipt (ingest returned an error) refuses change.detect/change.impact for the whole repo for up to 10 minutes, even though waiting cannot heal a terminal failure — and the refusal text implies waiting will help.
• A subsequent successful re-parse of the same path — a later mutation generation or a scoped workspace_admin.reindex — does not resolve earlier failed receipts for that path, although the graph then reflects newer bytes than the failed generation ever wrote. Under continuous editing with intermittent ingest failures this produces a rolling block that looks permanent to agents.
• The id-space mismatch stands: the refusal prints the internal receipt id (mutation-N), which change.receipt options.mutation_id does not accept.

I'm preparing a small PR that resolves failed receipts for a path when a later generation for the same path completes successfully (and when a direct reindex of that path succeeds), with tests alongside the existing ones in internal/mcp/mutation_freshness_test.go. Happy to take a different approach if you'd prefer another design.

[tiendungdev]

A third locking layer, distinct from the receipt ledger: the per-path mutation lock can wedge and pin a file for every writer. Same build (v0.63.8+2a33f3d), Windows 11. Reporting here because it produces the same operator experience as the original issue — a lockout nothing seems to clear — but through a different mechanism, and PR #693 does not (and was not meant to) address it.

Observed sequence

An agent needed to roll back a deliberate test mutation (byte restore of one file). Every attempted write to that path — edit.file, edit.symbol, and the batch path — aborted with:

... cancelled while waiting for exclusive file access: context deadline exceeded

which traces to acquireMutationPath(s) (internal/mcp/edit_serialization.go — mutationPathLocks). Targeted workspace_admin.reindex of the path did not help (expected — the lock is unrelated to the index). A daemon restart did not visibly help either: writes to the path kept failing the same way afterwards. Since mutationPathLocks is process-local and cannot survive a restart, our reading is that the first post-restart mutation of the path wedged inside the write path during warmup (store busy) and then pinned the token for every subsequent writer — i.e., what persists across restart is not the lock but the wedge-inducing condition (heavy warmup + immediate retries queueing behind the stuck holder).

Additionally, after the restart change.receipt for the path reports graph_status: "stale" permanently — the in-memory receipt died with the process and nothing refreshes the mutation-commit record's graph half. That is a live instance of the concern from PR #693's review (a record left refreshing against a missing receipt id), just reached via restart instead of deletion.

Caveats

Agent-reported sequence, reconstructed; no goroutine dump was captured while wedged, and the root hold is not isolated — it could be store contention during warmup or an external OS file handle (Windows; we have separately seen unrelated processes pin directories). The bare context deadline exceeded gives the operator no way to distinguish those.

Requests

1. Holder visibility and a bounded hold on the path lock. When an acquire times out, name the current holder (operation, mutation id, elapsed hold time) instead of a bare deadline error; consider a maximum hold after which the holder is failed terminally and the token released.
2. Don't stack waiters behind a wedged holder — same shape as the detect retry-stacking in the issue body: each retry joins the queue and amplifies contention.
3. Post-restart reconciliation for mutation-commit records: a record whose receipt id no longer exists (process restarted) should resolve to a terminal state — e.g., verify the on-disk SHA and mark fresh or failed — rather than reporting stale/pending forever. This generalizes PR fix(mcp): resolve superseded failed mutation receipts on later success #693's resolve-in-place to the restart path.
4. If the daemon can detect that its atomic rename failed due to an external handle (sharing violation), saying so explicitly in the error would immediately separate "gortex wedged" from "something else holds your file".

Workaround that unblocked the agent, for anyone else who lands here: perform the rollback with plain file tools outside the daemon (the path lock only governs the daemon's own writers), verify bytes via git, and stop retrying through the MCP surface until the daemon quiets down.