Skip to content

Fix: mirror funds only ever traded in a burst at startup - #13

Open
mpont91 wants to merge 1 commit into
ent0n29:mainfrom
mpont91:fix/mirror-ingestion-lag
Open

Fix: mirror funds only ever traded in a burst at startup#13
mpont91 wants to merge 1 commit into
ent0n29:mainfrom
mpont91:fix/mirror-ingestion-lag

Conversation

@mpont91

@mpont91 mpont91 commented Aug 27, 2026

Copy link
Copy Markdown

On first poll FundTradeListener looks back an hour; from then on the window is [lastPollTime, now] and lastPollTime is set to now at the end of every poll. The query selects on ts:

WHERE proxy_address IN (...)
  AND ts > toDateTime(from)
  AND ts <= toDateTime(to)

Trades do not reach ClickHouse the instant they happen. Measured on a live instance the lag between ts and ingested_at is a steady ~4.5 minutes (median 267s, p95 285s, max 296s). A trade therefore becomes visible several minutes after the highwater mark has passed its ts, and no window will ever contain it.

The only trades a fund ever sees are the ones already in the table when the startup lookback runs.

Evidence

Fund executions on a running instance, grouped by hour:

14:00   69      <- service restart
13:00   24      <- service restart
10:00  103      <- service restart
(nothing in between)

Meanwhile the index constituents traded ~1,400 times in the same 24 hours. Between restarts the funds are inert.

Fix

Window on ingested_at, which is when a row becomes visible, so nothing can arrive behind the mark. Signals still carry the real trade timestamp, so the delay a fund sees is the pipeline lag and nothing more.

pollForTrades is also synchronized. MultiFundConfiguration.pollPsiMirrorFunds runs at a fixed 1s rate over every fund; when a round takes longer than that the next starts on another thread, both read the same lastPollTime, and the same trade is queued twice. It shows up in the logs as one signal emitted by two scheduling threads a millisecond apart — two mirrored orders for one trader action.

After

Executions now arrive continuously rather than only after a restart, across all six mirror funds.

FundTradeListener advanced its highwater mark to now() on every poll and
selected trades whose ts fell in that window. Trades reach ClickHouse
minutes after they happen, so by the time one lands the mark is already
past its ts and no window can contain it. Only the startup lookback ever
saw anything.

The window is over ingested_at now, and pollForTrades is synchronized
because the multi-fund scheduler overlaps invocations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant