Skip to content

protocol: bound the socket writes and stop sending on a broken frame (ibx#254) - #363

Open
userFRM wants to merge 1 commit into
deepentropy:mainfrom
userFRM:fix/write-timeouts
Open

protocol: bound the socket writes and stop sending on a broken frame (ibx#254)#363
userFRM wants to merge 1 commit into
deepentropy:mainfrom
userFRM:fix/write-timeouts

Conversation

@userFRM

@userFRM userFRM commented Jul 30, 2026

Copy link
Copy Markdown

Problem

No socket in the engine had a write timeout — set_write_timeout was called nowhere in src/. Reads were bounded; writes were not.

The hot loop is one thread driving all three transports, and every outbound message goes out through a blocking write on it: order submits, cancels, subscriptions, and the heartbeats themselves. A peer that stops draining without closing blocks that thread, so liveness cannot be evaluated, shutdown cannot be serviced, and a reconnect cannot be polled. The failure is self-defeating: the heartbeat exists to detect a dead peer, and it is the write that blocks on it.

What this changes

One construction path. Both constructors reach the socket through from_stream, so a transport cannot be given a different set of timeouts by being built down a different path, and there is a single place the bound can be dropped from.

A broken frame ends the transport; a stalled one does not. Outbound frames are HMAC-chained, so a timed-out write needs more care than a retry. write_frame tracks how much of the frame reached the socket:

  • part of it went out — the peer holds a prefix it cannot verify and every later frame would be signed from state it does not share. That connection is finished, and every subsequent send is refused rather than putting more on the wire.
  • none of it went out — nothing reached the wire, the chain is intact, and the frame can go again. A peer that is merely slow does not cost the transport.
  • interrupted by a signal — retried, counted as neither.

The bound is on a single write syscall rather than a whole frame, so a peer draining a trickle keeps resetting it. The case it exists for is the peer making no progress at all, which trips the first timeout; a peer that stalls indefinitely reaches the liveness deadlines, which is where it belongs.

The keep-up-to-date historical request no longer advances its signing IV for a frame that never went out. That IV is derived from the frame, so committing it before the send desynchronises the chain and the next authentic frame no longer verifies. It is now held across the send and committed only when the frame is away, along with the send timestamp the liveness clock reads.

What this does not do

It does not treat an abandoned write as a disconnect. That belongs with the reconnect paths, and three of them do not currently support it — each reachable today through the read-side liveness path, none introduced here:

Until those are fixed, recovery stays on the existing route — the read deadlines — and the write bound is what stops the thread being held meanwhile.

Tests

  • a_write_timeout_is_configured_on_both_constructors — the bound is set, the read cadence is unchanged, and a fresh connection writes.
  • the_write_bound_stays_within_the_liveness_cadence — the bound cannot outlast the liveness probe interval.
  • only_a_frame_that_partly_went_out_finishes_the_transport — stall with nothing sent is retryable; the same error mid-frame is not; a hard error never is.
  • a_failed_write_finishes_the_transportsend_raw, send_fix and send_fixcomp are all refused afterwards, and no sequence number is consumed.

Each fails by name against a compiling reversion of the production change it covers.

Closes #254.

Test plan

  • Mutation: removing set_write_timeout fails a_write_timeout_is_configured_on_both_constructors by name.
  • Mutation: raising the bound to 50s fails the_write_bound_stays_within_the_liveness_cadence.
  • Mutation: treating every error as recoverable fails only_a_frame_that_partly_went_out_finishes_the_transport.
  • Mutation: letting send_fixcomp bypass the latch, and letting the latch stop refusing, each fail a_failed_write_finishes_the_transport.
  • Both constructors reach the socket through one function, so the bound cannot be present on one transport and missing on another.
  • cargo check --offline clean on --lib, --lib --features python, --bins, --examples, and each integration target individually.
  • tests/ib_paper_compat compared against a clean checkout of the base commit — identical sorted diagnostic sets.
  • cargo test --offline --lib — only the two known config::expiry_tests failures, which fail on the base commit for missing legacy tzdata (fixed separately in config: resolve the legacy timezone names IB states its times in (ibx#335) #336).

@userFRM
userFRM force-pushed the fix/write-timeouts branch from 0c68016 to 622d28b Compare July 30, 2026 15:03
@userFRM userFRM changed the title engine: bound outbound writes so a stalled peer cannot wedge the hot loop (ibx#254) protocol: bound the socket writes and stop sending on a broken frame (ibx#254) Jul 30, 2026
@userFRM
userFRM force-pushed the fix/write-timeouts branch from 622d28b to e4cf674 Compare July 31, 2026 11:39
No socket in the engine had a write timeout — `set_write_timeout` was called nowhere. Reads were bounded; writes were not. The hot loop is one thread driving all three transports, and every outbound message goes out through a blocking write on it: order submits, cancels, subscriptions, and the heartbeats themselves. A peer that stops draining without closing blocks that thread, so liveness cannot be evaluated, shutdown cannot be serviced, and a reconnect cannot be polled. The failure is self-defeating — the heartbeat exists to detect a dead peer, and it is the write that blocks on it.

Both constructors now reach the socket through one function, so a transport cannot be given a different set of timeouts by being built down a different path, and there is a single place the bound can be dropped from.

A timed-out write needs more care than a retry, because outbound frames are HMAC-chained. `write_frame` tracks how much of the frame reached the socket, which is what separates a transport that can carry on from one that cannot. A frame that went out in part left the peer a prefix it cannot verify, and every later frame would be signed from state the peer does not share — that connection is finished, and every subsequent send is refused rather than putting more on the wire. A write that moved nothing put nothing on the wire: the chain is intact, the frame can go again, and a peer that is merely slow does not cost the transport. That reasoning holds only where the count is bytes on the socket, so it is confined to the raw transport — through TLS the count is plaintext the layer accepted, and a stall reporting none accepted can still have put part of a record in front of the peer. There a retry would leave a second frame behind half of the first, with sequence and signature state committed for a frame the peer never receives whole, so an order reported as failed could still arrive. A signal is retried rather than counted as either.

The bound is on a single write syscall rather than a whole frame, so a peer draining a trickle keeps resetting it. The case it exists for is the peer making no progress at all, which trips the first timeout; the peer that stalls indefinitely reaches the liveness deadlines, which is where it belongs.

The keep-up-to-date historical request committed its next signing IV before the send and discarded the result. That IV is derived from the frame, so committing it for a frame that never went out desynchronises the chain and the next authentic frame no longer verifies. It is now held across the send and committed only when the frame is away, along with the send timestamp the liveness clock reads. A refused send is also reported as one: the request used to be registered as pending and returned as sent, and that waiter is exempt from the idle sweep because a keep-up-to-date request is a subscription rather than a single answer — so nothing would ever answer it and nothing would ever expire it.

What this does not do is treat an abandoned write as a disconnect. That belongs with the reconnect paths, and three of them do not currently support it: HMDS sets its disconnected flag without releasing the connection, so its reconnect never runs; the farm's disconnect clears the very subscription list its reconnect resubscribes from; and a request whose write is refused stays recorded with no response and no rejection. Each is reachable today through the read-side liveness path and none is introduced here. Until they are fixed, recovery stays on the existing route — the read deadlines — and the write bound is what stops the thread being held meanwhile.

Closes deepentropy#254.
@userFRM
userFRM force-pushed the fix/write-timeouts branch from e4cf674 to 9997bc5 Compare July 31, 2026 12:17
userFRM added a commit to userFRM/ibx that referenced this pull request Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

engine: no write timeouts anywhere, so a stalled peer blocks the single hot-loop thread and wedges liveness, shutdown and reconnect

1 participant