Skip to content

fix(ingest): clamp updated_at_ms on clock-skew writes, closes #167 - #168

Closed
pgcath wants to merge 1 commit into
MaxGhenis:mainfrom
pgcath:fix/ingest-updated-at-clamp
Closed

fix(ingest): clamp updated_at_ms on clock-skew writes, closes #167#168
pgcath wants to merge 1 commit into
MaxGhenis:mainfrom
pgcath:fix/ingest-updated-at-clamp

Conversation

@pgcath

@pgcath pgcath commented Aug 16, 2026

Copy link
Copy Markdown

Problem

internal/storage/sqlite/migrations/0002_identity_graph.sql enforces CHECK (updated_at_ms >= created_at_ms) on all identity-graph tables. Google Messages frames sometimes arrive with a provider timestamp ahead of the local wall clock. When such a message is the first in a conversation, ensureMessageConversation stores created_at_ms = provider_future_ms. Any subsequent write that sets updated_at_ms = nowMS (where nowMS < provider_future_ms) then violates the check and causes the frame to be quarantined rather than projected. The journal recorded 5 such Quarantined ingest frame events since 2026-08-13 (post v2 cutover).

Fix

At the two write-sites that perform monotone updated_at_ms advances on existing identity-graph rows:

  • refreshConversation (internal/ingest/worker.go) — after setting conversation.UpdatedAtMS = nowMS, clamp it up to conversation.CreatedAtMS when the clock is behind the row's own creation timestamp.
  • resolveIdentity (internal/ingest/worker.go) — same clamp for the identity table (defensive; guards the case where the system clock goes backward after identity creation).

Both sites emit a Warn-level log line with created_at_ms and updated_at_ms fields so the skew is visible, and increment the new updated_at_clamped counter that surfaces in /api/status under per_account[*].updated_at_clamped.

The schema CHECK constraint is left intact — this fix normalises the data before writing rather than relaxing the invariant, following the precedent of #160 (participant deduplication instead of quarantine).

Test

TestWorkerPathsUpdatedAtClampedOnSkewedConversationCreate in internal/ingest/worker_paths_test.go:

  1. Feed a message event with OccurredAt = nowMS + 50_000 (50 s ahead of the worker clock) — ensureMessageConversation creates the conversation with created_at_ms = nowMS + 50_000.
  2. Feed a conversation snapshot at nowMS < created_at_ms — this is the exact frame shape that previously quarantined.
  3. Assert: conversation is stored (GetConversationByRemote succeeds), Quarantined == 0, UpdatedAtClamped > 0.

Test run

Full suite green locally:

ok  github.com/maxghenis/openmessage/internal/ingest      2.232s
ok  github.com/maxghenis/openmessage/internal/storage/sqlite  1.895s
... (all 33 packages pass)

No live binary rebuilt, no container restarted, no live data dir touched.

Closes #167

#167)

Google Messages frames sometimes carry provider timestamps ahead of the
local wall clock. When such a message is the first in a conversation,
ensureMessageConversation stores created_at_ms = future_ms. A later
conversation or identity write then sets updated_at_ms = nowMS where
nowMS < created_at_ms, violating CHECK (updated_at_ms >= created_at_ms)
and quarantining the frame instead of projecting it.

Fix: in refreshConversation and resolveIdentity, after setting
UpdatedAtMS = nowMS, clamp it up to the existing row's CreatedAtMS when
the clock is behind. A new updated_at_clamped counter and Warn log make
the skew visible in /api/status without losing any data.

Regression test: TestWorkerPathsUpdatedAtClampedOnSkewedConversationCreate
feeds a message with a future provider timestamp (to produce a
conversation with created_at_ms = future), then feeds a conversation
snapshot at nowMS < future. Without the fix the second frame quarantines;
with it the frame stores, Quarantined stays 0, and UpdatedAtClamped > 0.

Closes #167

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@pgcath pgcath closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inbound identity-graph snapshots with updated_at_ms < created_at_ms are quarantined (data loss) since v2

1 participant