Broker reports a rotate-token failure as "explicit workspace key was rejected", misdirecting diagnosis

[khaliqgant]

Summary

When agent-name reclaim fails at the rotate_agent_token step,
startup_single_session_set_from_sources relabels the error as the workspace
key being rejected. The workspace key was accepted; a later call on a different
credential class was not. This cost six days of misdiagnosis on a production
outage, with the investigation aimed at credential provisioning while the key was
never the problem.

What the operator sees

Error: failed to initialize relaycast session
Caused by:
    0: explicit workspace key from AGENT_RELAY_WORKSPACE_KEY was rejected
    1: Agent token required (at_live_...)

Cause 0 asserts the key was rejected. Cause 1 is the real reason, and it is about
a different credential entirely. Every reading of cause 0 sends you to check the
key, its scope, and how it is provisioned - all of which are fine.

Mechanism

In crates/broker/src/relaycast/auth.rs, admit_agent_registration (:985) has a
409 branch:

let existing = relay.get_agent(name).await...?;      // 200 with the workspace key
...
let token_response = relay.rotate_agent_token(&existing.name).await...?;  // 401

That 401 propagates out of register_agent_with_workspace_key. The caller at
:551 matches Err(error) if is_auth_rejection(&error) and, because
candidate.explicit_join is set, wraps it with:

"explicit workspace key from {} was rejected"

is_auth_rejection only tests the HTTP status, so a 401 from the rotate call is
indistinguishable from a 401 on the registration call, and both get the
registration wording.

Suggested fix

Attribute the failure to the call that produced it. The registration-rejected
wording should apply only when register_agent itself was rejected; a failure
from get_agent or rotate_agent_token inside the conflict branch should say so,
naming the credential class the endpoint actually wanted. Cause 1 already carries
the right information - it is cause 0 that overrides it with a confident wrong
answer.

This is a diagnosis fix, not a repair: the underlying reason a broker cannot
rotate its own token with a workspace key is filed against relaycast.

Reproduction

With broker 11.8.1, run init --persist --state-dir <dir> twice with the same
--instance-name and the same state dir, using a valid workspace key. The first
boot succeeds; the second produces the error above. A deliberately invalid key
produces Invalid API key instead, confirming the second-boot failure is not
about key validity.

[khaliqgant]

Adding the reproduction table, and reframing what the underlying defect is — the mislabel in the issue body is the symptom; this is the thing that is actually broken.

The real defect: a broker holding only a workspace key cannot complete its own documented identity-reclaim path

runtime/session.rs calls startup_session_set_with_identity on every boot and has no persisted-token resume branch. So a broker that reuses a stable instance name always gets 409 agent_already_exists for its own prior registration. The reclaim branch in auth.rs:985 then does exactly what it was designed to do — it proves ownership via the identity_key derived from the durable state dir — and immediately calls rotate_agent_token, which the engine gates behind an at_live_ agent token the broker cannot mint from a workspace key.

The ownership proof succeeds. The step after it refuses on a credential class the caller cannot hold, precisely because it is trying to re-establish one. The path cannot succeed for its intended caller.

Reproduction table

Published @agent-relay/broker-darwin-arm64@11.8.1 (npm, not a workspace build), matching the deployed stack @agent-relay/factory@0.1.68 -> harness-driver 11.8.1 -> broker 11.8.1.

arm	env	result
A	workspace key + RELAY_NODE_ID/RELAY_NODE_TOKEN	starts, relay ready
B	node env only, no workspace key	starts — but creates a new workspace
C	workspace key only, first boot	starts, relay ready
D	invalid rk_ key (control)	fails: Invalid API key
E	= C repeated: second boot, same --instance-name, same --state-dir	fails: the production error

Arm E, verbatim:

Error: failed to initialize relaycast session
Caused by:
    0: explicit workspace key from AGENT_RELAY_WORKSPACE_KEY was rejected
    1: Agent token required (at_live_...)

The control (D) returns a different error, so the experiment discriminates rather than reporting one failure mode for every input. Only name reuse moves the result. Arm A carries a node token and still succeeds, so the node credential is not implicated: build_relay_client takes only the passed workspace key, and RELAY_NODE_TOKEN is consumed later by resolve_cached_node_token for /v1/node/ws.

Direct confirmation of the failing step, against a throwaway agent:

POST /v1/agents/<throwaway>/rotate-token   Authorization: Bearer rk_live_...
-> 401 {"code":"unauthorized","message":"Agent token required (at_live_...)"}

Credential-class vocabulary on GET /v1/agents, showing the workspace key is otherwise fully accepted on the same resource: valid rk_live_ -> 200; at_live_ -> 401 Workspace key required; nt_live_ -> 401 node_token_forbidden; invalid rk_ -> 401 Invalid API key.

Why sidestepping it is not a fix

The obvious workaround is a unique instance name so registration stays on the 201 path. Two problems, both learned the hard way in production tonight:

1. It does not hold. A name unique per boot still collides on every retry within that boot — the control plane respawns roughly every 40s, the first spawn registers the name, and every retry after that lands back on 409 -> reclaim -> rotate -> 401. Net effect: the first spawn of each boot works, the rest fail identically.
2. It leaks rows. Every fresh name is a permanent agent record. The node roster in the same deployment is already carrying ~1225 undeletable rows; adding an agent row per boot repeats that mistake in a second table.

So the workaround cannot be made correct by making the name more unique — a per-spawn name leaks faster and still never lets a broker reclaim an identity it demonstrably owns.

Suggested direction

Either let a workspace key authorize rotate-token (see AgentWorkforce/relaycast-cloud#83 — it grants no new capability class, since a workspace key can already create agents and receives a token in that response), or give the broker a way to obtain a token for a name whose identity_key it has already proven in step 3.

Until one of those exists, any broker with a stable instance name and a durable state directory is permanently unable to restart. It is invisible until the second boot, because the first registration takes the 201 path and never touches rotate.

[khaliqgant]

The relabelling also hides intermediate refusals, not just the final one — and that is the sharper cost

Adding this because it is the concrete reason a real production outage stayed misdiagnosed, and it strengthens the case beyond "the message names the wrong request".

The conflict branch in admit_agent_registration (auth.rs:985) makes up to three calls: register_agent → get_agent → rotate_agent_token. Any 401 from any of them is caught by is_auth_rejection at auth.rs:551 and rewritten as:

explicit workspace key from AGENT_RELAY_WORKSPACE_KEY was rejected

So an operator sees one string that asserts the workspace key was rejected, when in fact the workspace key authenticated calls 1 and 2 successfully and call 3 refused on a different credential class entirely. Every intermediate outcome is erased.

Why that mattered in practice. Two independent investigators, working the same outage from different angles, each built a plausible and wrong theory that fit the string — "the key is unscoped for agents", "relaycast tightened agent auth" — because the string is all production surfaces. Both theories were killed in minutes once the underlying calls were probed directly:

probe	result
POST /v1/agents, new name, workspace key	201, returns an at_live_ token
same, bogus rk_ key (control)	401 Invalid API key — a different error
POST /v1/agents/<name>/rotate-token, workspace key	401 Agent token required (at_live_...)

The registration path was never the problem. The message said it was.

This gets worse as the surrounding code grows. PATCH /v1/agents/:name/legacy-identity now exists and has its own refusal codes (agent_identity_already_claimed, a not-offline refusal, agent_identity_claim_conflict). Any of those that surfaced as a 401 through this branch would be flattened into the same workspace-key string. The more recovery paths the engine adds, the more distinct failures this one line collapses into a single misleading sentence.

Suggested shape: attribute to the call that failed. Only use the registration wording when register_agent itself was rejected; for get_agent or rotate_agent_token, say which call refused and which credential class it wanted. The cause chain already carries the right information — it is cause 0 overriding it with a confident wrong answer that does the damage.

The value here is purely diagnostic, and that is exactly the point: nothing about the fix changes behaviour, but it is the difference between a one-hour diagnosis and a multi-day one.