Skip to content

fix(auth): keep long-lived clients current and coalesce token refreshes - #1102

Merged
codyde merged 1 commit into
masterfrom
fix/config-lock-warning-storm
Aug 14, 2026
Merged

fix(auth): keep long-lived clients current and coalesce token refreshes#1102
codyde merged 1 commit into
masterfrom
fix/config-lock-warning-storm

Conversation

@codyde

@codyde codyde commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Problem

A long-running railway ca / railway code session fills the screen with:

warning: timed out waiting for config lock; proceeding without it

Three causes:

  1. The TUI's bearer is frozen at startup. GQLClient::new_authorized bakes the token into default_headers; the TUI holds one client for its whole run. Once the token expires, every request 401s — permanently, since the 401 retry builds a throwaway client and leaves the long-lived one stale.
  2. Each 401 forced a token refresh under the config file lock. fs2 locks are per open file description, so the TUI's own tasks contend like separate processes. Bursts (25s auto-refresh, 1.5s watch ticks, per-env sweeps) queued on the lock until the 10s timeout fired, printing into a raw-mode alternate screen.
  3. probe_session_liveness refreshed unconditionally, bypassing the AlreadyFresh check. Backboard rotates and reuse-detects the refresh token on every refresh, so N simultaneous 401s = N rotations, N-1 presenting a consumed token → grounds for revoking the whole grant.

Cause 3 is a hard logout caused by our own concurrency, not just cosmetic. Disabling the new coalescing makes the burst test report the real number:

assertion `left == right` failed: the burst must cost one token rotation between them, not 8
  left: 8    right: 1

Fix

Cause Change
1 Set authorization per request from disk; run ensure_valid_token before sending, not after being refused. reqwest applies a default header only when the request hasn't set that key, so long-lived clients pick up refreshes. Fast path is a timestamp compare on an already-read config — no new I/O.
2 In-process refresh gate taken ahead of the file lock, so at most one task waits on it. Capped at 15s: a hung token endpoint can hold it for the full ~90s retry budget. Waiters fall back to stored credentials and do not refresh on their own.
3 10s window in which a forced refresh stands down because one just completed. A burst now costs one rotation.
reporter::warn defers output while a TUI owns the terminal, flushing on exit with a repeat count. Wired into set_terminal_owned, so all TUIs get it from their existing restore_terminal. Five other TUIs never set that flag; they do now.

New: post_graphql_public. Since post_graphql now attaches a bearer and a reqwest::Client can't be asked whether it was built with one, the template lookup/search queries that must go out unauthenticated declare it at the call site. The existing "public template lookup sends no auth headers" test caught this.

Testing

1122 tests pass, 5 consecutive clean runs, 0 clippy errors. whoami and templates search verified against the real API.

New in auth_sim.rs:

  • a stale client sends the rotated token through the real send path
  • a second refusal reuses the first one's refresh
  • 8 concurrent refusals rotate the token exactly once (fails with 8 if coalescing is removed)

MockBackboard now records each request's authorization header.

Risks

  • Touches every GraphQL request in the CLI, not just ca. Main risk; worth a canary.
  • 10s recency window: a grant revoked within 10s of a refresh reports OAuthInsufficientGrant instead of "log in again". Self-heals. Easy to tighten.
  • post_graphql_public is opt-in, no compile-time guard. A future public call site that forgets it sends a redundant bearer. Contained — all ~230 GraphQL call sites target get_backboard(), so it's never a cross-host leak. A newtype over reqwest::Client would make it checkable; out of scope here.
  • Logout in another terminal doesn't disarm a running TUI — with no token on disk, the client's baked bearer is still used. Pre-existing; unchanged.

🤖 Generated with Claude Code

A `railway ca` or `railway code` session left open long enough filled the
screen with "timed out waiting for config lock; proceeding without it".
Three things compounded to produce it.

The TUI builds one authorized client at startup and keeps it for the whole
run, and `GQLClient::new_authorized` bakes the bearer into that client's
default headers. Once the access token expires, every request it sends
carries a dead one — forever, because the 401 handler builds a throwaway
client for its retry and leaves the long-lived one just as stale.

Every one of those 401s then forced a token refresh while holding the
config file lock. `fs2` locks are per open file description, so the TUI's
own tasks contended exactly as separate processes would: a burst of
refusals queued on the lock, each waiter sat through the earlier ones'
token round-trips, blew the lock's 10s timeout, and printed. Into a raw-mode
alternate screen with no cursor discipline, which is why it smeared.

Worse than the spam: `probe_session_liveness` refreshed unconditionally,
with no "someone already did it" check. Backboard rotates and reuse-detects
the refresh token on every refresh, so N simultaneous 401s meant N
rotations, N-1 of them presenting an already-consumed token — grounds for
revoking the whole grant. A hard logout caused by nothing but our own
concurrency. The new burst test fails with 8 rotations if the coalescing is
removed.

So:

- Set the bearer per request from the config on disk, and refresh before
  sending rather than after being refused. Long-lived clients now pick up
  refreshes instead of going permanently stale. `reqwest` applies a default
  header only when the request has not already set that key.

- Add an in-process refresh gate ahead of the file lock, plus a 10s window
  in which a forced refresh stands down because one just happened. At most
  one task waits on the file lock, and a burst costs one rotation. The gate
  wait is capped at 15s: a hung token endpoint can hold it for the full
  90s retry budget, and stalling every other request behind that would be
  worse than what it replaced. Waiters proceed with stored credentials and
  deliberately do not refresh on their own.

- Hold warnings back while a TUI owns the terminal and flush them when it
  exits, deduplicated with a repeat count. Wired into `set_terminal_owned`,
  so every TUI gets it from its existing `restore_terminal`. Five other
  TUIs never set that flag and could be smeared the same way; they do now.

`post_graphql_public` is new: `post_graphql` now attaches a bearer, and a
`reqwest::Client` cannot be asked whether it was built with one, so the
template lookup/search queries that must go out unauthenticated say so at
the call site. The existing test asserting public template lookups send no
auth headers is what caught this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codyde codyde added the release/patch Author patch release label Aug 14, 2026
@codyde
codyde merged commit 69e6405 into master Aug 14, 2026
7 of 8 checks passed
@codyde
codyde deleted the fix/config-lock-warning-storm branch August 14, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/patch Author patch release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant