fix(ingest): clamp updated_at_ms on clock-skew writes, closes #167 - #168
Closed
pgcath wants to merge 1 commit into
Closed
fix(ingest): clamp updated_at_ms on clock-skew writes, closes #167#168pgcath wants to merge 1 commit into
pgcath wants to merge 1 commit into
Conversation
#167) Google Messages frames sometimes carry provider timestamps ahead of the local wall clock. When such a message is the first in a conversation, ensureMessageConversation stores created_at_ms = future_ms. A later conversation or identity write then sets updated_at_ms = nowMS where nowMS < created_at_ms, violating CHECK (updated_at_ms >= created_at_ms) and quarantining the frame instead of projecting it. Fix: in refreshConversation and resolveIdentity, after setting UpdatedAtMS = nowMS, clamp it up to the existing row's CreatedAtMS when the clock is behind. A new updated_at_clamped counter and Warn log make the skew visible in /api/status without losing any data. Regression test: TestWorkerPathsUpdatedAtClampedOnSkewedConversationCreate feeds a message with a future provider timestamp (to produce a conversation with created_at_ms = future), then feeds a conversation snapshot at nowMS < future. Without the fix the second frame quarantines; with it the frame stores, Quarantined stays 0, and UpdatedAtClamped > 0. Closes #167 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
internal/storage/sqlite/migrations/0002_identity_graph.sqlenforcesCHECK (updated_at_ms >= created_at_ms)on all identity-graph tables. Google Messages frames sometimes arrive with a provider timestamp ahead of the local wall clock. When such a message is the first in a conversation,ensureMessageConversationstorescreated_at_ms = provider_future_ms. Any subsequent write that setsupdated_at_ms = nowMS(wherenowMS < provider_future_ms) then violates the check and causes the frame to be quarantined rather than projected. The journal recorded 5 suchQuarantined ingest frameevents since 2026-08-13 (post v2 cutover).Fix
At the two write-sites that perform monotone
updated_at_msadvances on existing identity-graph rows:refreshConversation(internal/ingest/worker.go) — after settingconversation.UpdatedAtMS = nowMS, clamp it up toconversation.CreatedAtMSwhen the clock is behind the row's own creation timestamp.resolveIdentity(internal/ingest/worker.go) — same clamp for the identity table (defensive; guards the case where the system clock goes backward after identity creation).Both sites emit a
Warn-level log line withcreated_at_msandupdated_at_msfields so the skew is visible, and increment the newupdated_at_clampedcounter that surfaces in/api/statusunderper_account[*].updated_at_clamped.The schema
CHECKconstraint is left intact — this fix normalises the data before writing rather than relaxing the invariant, following the precedent of #160 (participant deduplication instead of quarantine).Test
TestWorkerPathsUpdatedAtClampedOnSkewedConversationCreateininternal/ingest/worker_paths_test.go:OccurredAt = nowMS + 50_000(50 s ahead of the worker clock) —ensureMessageConversationcreates the conversation withcreated_at_ms = nowMS + 50_000.nowMS < created_at_ms— this is the exact frame shape that previously quarantined.GetConversationByRemotesucceeds),Quarantined == 0,UpdatedAtClamped > 0.Test run
Full suite green locally:
No live binary rebuilt, no container restarted, no live data dir touched.
Closes #167