protocol: bound the socket writes and stop sending on a broken frame (ibx#254) - #363
Open
userFRM wants to merge 1 commit into
Open
protocol: bound the socket writes and stop sending on a broken frame (ibx#254)#363userFRM wants to merge 1 commit into
userFRM wants to merge 1 commit into
Conversation
userFRM
force-pushed
the
fix/write-timeouts
branch
from
July 30, 2026 15:03
0c68016 to
622d28b
Compare
userFRM
force-pushed
the
fix/write-timeouts
branch
from
July 31, 2026 11:39
622d28b to
e4cf674
Compare
No socket in the engine had a write timeout — `set_write_timeout` was called nowhere. Reads were bounded; writes were not. The hot loop is one thread driving all three transports, and every outbound message goes out through a blocking write on it: order submits, cancels, subscriptions, and the heartbeats themselves. A peer that stops draining without closing blocks that thread, so liveness cannot be evaluated, shutdown cannot be serviced, and a reconnect cannot be polled. The failure is self-defeating — the heartbeat exists to detect a dead peer, and it is the write that blocks on it. Both constructors now reach the socket through one function, so a transport cannot be given a different set of timeouts by being built down a different path, and there is a single place the bound can be dropped from. A timed-out write needs more care than a retry, because outbound frames are HMAC-chained. `write_frame` tracks how much of the frame reached the socket, which is what separates a transport that can carry on from one that cannot. A frame that went out in part left the peer a prefix it cannot verify, and every later frame would be signed from state the peer does not share — that connection is finished, and every subsequent send is refused rather than putting more on the wire. A write that moved nothing put nothing on the wire: the chain is intact, the frame can go again, and a peer that is merely slow does not cost the transport. That reasoning holds only where the count is bytes on the socket, so it is confined to the raw transport — through TLS the count is plaintext the layer accepted, and a stall reporting none accepted can still have put part of a record in front of the peer. There a retry would leave a second frame behind half of the first, with sequence and signature state committed for a frame the peer never receives whole, so an order reported as failed could still arrive. A signal is retried rather than counted as either. The bound is on a single write syscall rather than a whole frame, so a peer draining a trickle keeps resetting it. The case it exists for is the peer making no progress at all, which trips the first timeout; the peer that stalls indefinitely reaches the liveness deadlines, which is where it belongs. The keep-up-to-date historical request committed its next signing IV before the send and discarded the result. That IV is derived from the frame, so committing it for a frame that never went out desynchronises the chain and the next authentic frame no longer verifies. It is now held across the send and committed only when the frame is away, along with the send timestamp the liveness clock reads. A refused send is also reported as one: the request used to be registered as pending and returned as sent, and that waiter is exempt from the idle sweep because a keep-up-to-date request is a subscription rather than a single answer — so nothing would ever answer it and nothing would ever expire it. What this does not do is treat an abandoned write as a disconnect. That belongs with the reconnect paths, and three of them do not currently support it: HMDS sets its disconnected flag without releasing the connection, so its reconnect never runs; the farm's disconnect clears the very subscription list its reconnect resubscribes from; and a request whose write is refused stays recorded with no response and no rejection. Each is reachable today through the read-side liveness path and none is introduced here. Until they are fixed, recovery stays on the existing route — the read deadlines — and the write bound is what stops the thread being held meanwhile. Closes deepentropy#254.
userFRM
force-pushed
the
fix/write-timeouts
branch
from
July 31, 2026 12:17
e4cf674 to
9997bc5
Compare
userFRM
added a commit
to userFRM/ibx
that referenced
this pull request
Aug 3, 2026
… drop the hardcoded debug dump
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
No socket in the engine had a write timeout —
set_write_timeoutwas called nowhere insrc/. Reads were bounded; writes were not.The hot loop is one thread driving all three transports, and every outbound message goes out through a blocking write on it: order submits, cancels, subscriptions, and the heartbeats themselves. A peer that stops draining without closing blocks that thread, so liveness cannot be evaluated, shutdown cannot be serviced, and a reconnect cannot be polled. The failure is self-defeating: the heartbeat exists to detect a dead peer, and it is the write that blocks on it.
What this changes
One construction path. Both constructors reach the socket through
from_stream, so a transport cannot be given a different set of timeouts by being built down a different path, and there is a single place the bound can be dropped from.A broken frame ends the transport; a stalled one does not. Outbound frames are HMAC-chained, so a timed-out write needs more care than a retry.
write_frametracks how much of the frame reached the socket:The bound is on a single write syscall rather than a whole frame, so a peer draining a trickle keeps resetting it. The case it exists for is the peer making no progress at all, which trips the first timeout; a peer that stalls indefinitely reaches the liveness deadlines, which is where it belongs.
The keep-up-to-date historical request no longer advances its signing IV for a frame that never went out. That IV is derived from the frame, so committing it before the send desynchronises the chain and the next authentic frame no longer verifies. It is now held across the send and committed only when the frame is away, along with the send timestamp the liveness clock reads.
What this does not do
It does not treat an abandoned write as a disconnect. That belongs with the reconnect paths, and three of them do not currently support it — each reachable today through the read-side liveness path, none introduced here:
maybe_spawn_hmds_reconnectreturns early forever (hmds: a liveness timeout leaves the connection in place, so the HMDS reconnect never runs #367);handle_disconnectclearsinstrument_md_reqs, the list its ownreconnectresubscribes from (farm: reconnect resubscribes from the collection disconnect clears, so every farm reconnect comes back with no L1 data #368);Until those are fixed, recovery stays on the existing route — the read deadlines — and the write bound is what stops the thread being held meanwhile.
Tests
a_write_timeout_is_configured_on_both_constructors— the bound is set, the read cadence is unchanged, and a fresh connection writes.the_write_bound_stays_within_the_liveness_cadence— the bound cannot outlast the liveness probe interval.only_a_frame_that_partly_went_out_finishes_the_transport— stall with nothing sent is retryable; the same error mid-frame is not; a hard error never is.a_failed_write_finishes_the_transport—send_raw,send_fixandsend_fixcompare all refused afterwards, and no sequence number is consumed.Each fails by name against a compiling reversion of the production change it covers.
Closes #254.
Test plan
set_write_timeoutfailsa_write_timeout_is_configured_on_both_constructorsby name.the_write_bound_stays_within_the_liveness_cadence.only_a_frame_that_partly_went_out_finishes_the_transport.send_fixcompbypass the latch, and letting the latch stop refusing, each faila_failed_write_finishes_the_transport.cargo check --offlineclean on--lib,--lib --features python,--bins,--examples, and each integration target individually.tests/ib_paper_compatcompared against a clean checkout of the base commit — identical sorted diagnostic sets.cargo test --offline --lib— only the two knownconfig::expiry_testsfailures, which fail on the base commit for missing legacy tzdata (fixed separately in config: resolve the legacy timezone names IB states its times in (ibx#335) #336).