[claude] Snapshot rollback resumes from a stale snapshot and loses intermediate state

[myieye]

[Claude, autonomous]

When an out-of-order commit arrives, DataModel.UpdateSnapshots deletes the snapshots after it and replays only the commits after it, assuming each entity's newest surviving snapshot is its state at that point. That's false once GenerateSnapshotForEntity has pruned intermediates in an earlier batch (it keeps only every second one): edits between the surviving snapshot and the rollback point are silently lost, and a cascade delete, which only ever existed as a snapshot, is undone.

Hit in production FwHeadless: a sense whose entry had been deleted came back alive when a late commit landed between the delete and later edits to that sense, and every sync since fails with FOREIGN KEY constraint failed in CrdtRepository.AddSnapshots. The replay probably needs to start from the oldest surviving snapshot of the affected entities rather than the oldest new commit. Reproducible in DataModelTestBase with WriteChangeBefore and two synced clients.

[myieye]

[Claude, autonomous]

Concrete sequence, taken from the production case (entry E with sense S; clients A, B, C):

1. A creates E and S. Everyone syncs.
2. B deletes E. MarkDeleted cascades to S, which only produces a snapshot (S-deleted@t1); no commit mentions S.
3. C, who never received the delete, edits S three times (t3, t4, t5). C's commits sit on the server.
4. A works offline for weeks, then syncs. Its backlog starts before t1. The server hands C's edits and A's backlog to the model in one payload, so UpdateSnapshots rolls back to before t1 and replays everything in one SnapshotWorker batch. In that replay S-deleted@t1 is recreated as a pending snapshot, then overwritten by the t3 snapshot. GenerateSnapshotForEntity keeps the previous pending snapshot only when commitIndex % 2 == 0; here it was odd, so S-deleted@t1 is never written. Same for t3 and t4. S ends up with two snapshots: its root (alive) and t5 (deleted).
5. Later another client's backlog arrives with an oldest commit at t2, where t1 < t2 < t3. DeleteStaleSnapshots(t2) removes the t5 snapshot, S's newest surviving snapshot is its root, and the replay set is everything after t2, which does not include the delete at t1. The t3..t5 edits are re-applied to an alive S. Projecting S fails on the EntryId foreign key because E's row is gone, the transaction rolls back, and every later sync repeats it.

Step 4 is ordinary: any payload that delivers several edits to one entity in a batch prunes intermediates. Step 5 only needs one out-of-order commit landing in the gap. Without the cascade the same mechanism silently drops the edits between the surviving snapshot and the rollback point instead of throwing.

[myieye]

[Claude, autonomous]

Test recipes, using DataModelTestBase (ForkDatabase, WriteNextChange, WriteChangeBefore, MockTimeProvider) and the sample model's Word/Definition (Definition references Word, and RemoveReference deletes it when the word goes, like Sense/Entry).

Plain edit loss (no cascade)

1. Client A creates word W. B = A.ForkDatabase().
2. A writes three edits to W's text, e1 < e2 < e3, as separate commits. A's own snapshots stay dense because each write is its own batch.
3. B.SyncWith(A): B receives e1..e3 in one batch. In GenerateSnapshotForEntity the previous pending snapshot survives only when commitIndex % 2 == 0, so exactly one of e1's or e2's snapshot is dropped on B. Which one depends on the commit indexes in that batch, so assert it via GetLatestSnapshotByObjectId / the Snapshots table, or cover both gaps in two tests.
4. On A, WriteChangeBefore(e2, <change to an unrelated entity>) if e1's snapshot was dropped, or WriteChangeBefore(e3, ...) if e2's was. Sync to B.
5. Expected: B's W equals A's W. Actual: the dropped edit is missing from B, no error.

Cascade case (the production failure)

1. A creates word W and definition D. Fork B. Sync.
2. A deletes W, sync to B. B now holds D-deleted as a snapshot only.
3. On A, write three SetDefinitionPartOfSpeechChange (or any Definition edit) on D, dated after the delete, plus one unrelated commit dated before the delete (WriteChangeBefore). Sync to B in one go: B rolls back past the delete and replays; the recreated cascade snapshot is pruned when the first D edit lands on an odd index (again, assert or cover both parities).
4. On A, WriteChangeBefore(<first D edit>, <unrelated change>), dated after the delete. Sync to B.
5. Expected: D deleted on B. Actual: DbUpdateException (FK on Definition.WordId) from CrdtRepository.AddSnapshots with projected tables on; an alive D pointing at a deleted W with them off.

Both need the edits to reach the affected model in a single sync; writing them directly on that model would keep its snapshots dense and hide the bug.