Skip to content

engine: park the hot loop while every transport is down (ibx#399) - #401

Open
userFRM wants to merge 1 commit into
deepentropy:mainfrom
userFRM:fix/disconnected-loop-spins
Open

engine: park the hot loop while every transport is down (ibx#399)#401
userFRM wants to merge 1 commit into
deepentropy:mainfrom
userFRM:fix/disconnected-loop-spins

Conversation

@userFRM

@userFRM userFRM commented Aug 3, 2026

Copy link
Copy Markdown

Summary

  • The hot loop busy-polls the farm, auth and historical sockets and never yields. While a session is up that is the design: a poll that finds a tick returns it with no syscall in the way, and core_id exists so the spin can own a core.
  • With every transport down there is nothing to poll. Each pass runs the same three polls, gets nothing, and starts again — measured at roughly 925,000 iterations a second, which is one core pinned at 100% for the length of the disconnect.
  • Parking 1 ms in that state, and only that state, leaves the connected path byte for byte as it was. Nothing waits on the loop meanwhile: reconnects are already scheduled on the ibx#218 backoff in seconds, and shutdown is read once per pass.

On the reported cause

The report puts the CPU on the reconnect scheduler retrying in a tight loop. The timestamps in its own log are two minutes apart:

09:48:11.537  CCP auto-reconnect skipped: no credentials
09:50:21.157  CCP auto-reconnect skipped: no credentials

That is the 60-second re-check in maybe_spawn_ccp_reconnect doing its job. The spin is the loop body itself, and it is there whether or not credentials are cached — a connected engine spins too, which is intended, and a disconnected one spins for nothing, which is not.

Closes #399.

Test plan

  • cargo test --lib — 803 pass. The two config::expiry_tests failures are pre-existing on main and identical on a pristine checkout of 9367845.
  • a_loop_with_every_transport_down_does_not_spin drives run() with all three transports marked down, shuts it down after 60 ms and asserts the iteration count. Counting iterations rather than CPU time is what separates a parked loop from a spinning one on any machine.
  • Mutation: deleting the sleep takes that test from ~60 iterations to 55,554 in the same window, and it fails by name.
  • Not covered: the laptop sleep/resume path the report describes. The guard is on the disconnected flags, so it does not depend on how the disconnect happened.

The loop busy-polls three sockets and never yields. That is the point while a session is up: a poll that finds a tick returns it without a syscall in the way, and `core_id` exists so the spin can own a core.

With every transport down there is nothing to poll. Each pass runs the same three polls, gets nothing from any of them, and starts again. Measured here at roughly 925,000 iterations a second, which is one core at 100% for as long as the disconnect lasts. On a laptop that is the fans and the battery, which is how it gets noticed.

Parking one millisecond in that state and only that state leaves the connected path untouched. Nothing is waiting on the loop either: reconnects are already scheduled on a backoff measured in seconds, and shutdown is read once per pass.

The report this comes from attributes the CPU to the reconnect scheduler warning in a tight retry. The timestamps in it show those warnings two minutes apart, which is the 60-second re-check working as intended. The spin is the loop itself and is present whether or not credentials are cached.

`a_loop_with_every_transport_down_does_not_spin` counts iterations rather than CPU time, which is what separates a parked loop from a spinning one on any machine. Removing the sleep takes it from 60 to 55,554 in the same window.

Closes deepentropy#399.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rust EClient fails to reconnect and causes 100% CPU infinite spin-loop when connection is lost

1 participant