[Feature]: re-evaluate a routing profile when the selected candidate fails terminally

[brunoflma]

Area

Proxy and routing

What are you trying to accomplish?

I want a policy/<id> request to survive a terminal failure from the candidate it selected — specifically a 429 on a provider with no key pool, or a 400 that is really a provider-side defect rather than a malformed request.

The concrete shape: I route through a profile whose candidates span several providers. The evaluator picks candidate A. Candidate A returns 429 (quota exhausted) or 400. I want the request to be re-evaluated against the remaining eligible candidates and served by B, rather than the error reaching the client.

That is the reason I reached for routing profiles in the first place. On a single-provider setup the failure is just a failure; the value of declaring three candidates is that one of them can cover for another.

What prevents this today?

The policy evaluator runs exactly once, on the inbound routing decision, and never again for that request.

resolvePolicyProfileId has a single call site in the whole request path (src/router.ts:507 on v2.14.0), and re-entry is prevented deliberately:

// Only explicit requests reach this branch; concrete recursive targets skip policy
// resolution entirely (bypassCombos) so an alias matching a selected candidate can
// never recurse …
const policyId = !bypassCombos ? resolvePolicyProfileId(config, modelId) : null;

Once a concrete candidate is chosen, the profile is out of the picture. What exists downstream is scoped narrower than the profile:

• 429 triggers key-pool rotation within the same provider — src/server/responses/core.ts:3422: "rotate to the next pool key … and retry the SAME request once per remaining key. OAuth/forward providers and single-key pools return null immediately." It never moves to a different candidate.
• 400 has no failover path at all. Reasonable as a general rule, since 400 is nominally a client error — but it means a provider that answers 400 for its own reasons is indistinguishable from a genuinely malformed request.

So the profile's remaining influence on failures is indirect and slow: errors degrade a candidate's health, which lowers its score on future requests. On my install the gap between first and second candidate is 0.1256, and the effective weight of health in the total is 0.387, so the leader has to lose about 0.33 of health — roughly 98% → 66% success — before the order flips. That is recovery from sustained collapse, not from the request that just failed.

I am not claiming the current behaviour is a bug. Selection-time-only is a coherent design. But it is not discoverable from the configuration surface: a profile that lists three candidates and weights health reads like a failover chain, and it is not one.

What should OpenCodex do?

On a terminal failure from the selected candidate, re-run the policy evaluator with that candidate excluded, and serve the request from the next eligible one. Bounded by the candidate list, so at most N attempts and no recursion.

Minimum viable scope, if the full shape is too broad:

1. 429 when the provider has no key pool to rotate (today: a hard stop, even with two other candidates sitting idle).
2. An opt-in list of statuses treated as candidate-terminal, so 400 can be included by operators who know their provider emits it spuriously, without changing the default for everyone.

What I would want preserved: the existing bypassCombos guard against recursion, and the dry-run contract, so an operator can still see which candidate would be chosen first.

If failover is deliberately out of scope for profiles, then the documentation is the ask instead: state plainly that a profile chooses before the request and does not react to its outcome. I read the reference page and the dry-run output and still expected otherwise.

Example usage or interface

Today, with TR rate-limited and two healthy candidates configured:

POST /v1/responses   model=policy/fast-free
  → evaluator picks TR/moonshotai/kimi-k3-free   (highest score)
  → upstream 429, single-key provider, no pool to rotate
  → 429 reaches the client
  → google-antigravity/gemini-3.6-flash and opencode-zen/deepseek-v4-flash-free were both
    eligible in the same evaluation and are never tried

Desired:

POST /v1/responses   model=policy/fast-free
  → evaluator picks TR/moonshotai/kimi-k3-free
  → upstream 429, no pool rotation available
  → re-evaluate with TR excluded → google-antigravity/gemini-3.6-flash
  → 200

Opt-in shape for the second half, so the default stays conservative:

{
  "routingProfiles": {
    "fast-free": {
      "candidates": [ "…" ],
      "failover": { "onStatus": [429, 503], "maxCandidates": 3 }
    }
  }
}

Alternatives or workarounds

• apiKeyPool — what I use today, and it does cover 429 for the providers where I hold several keys. It cannot help a single-key or OAuth provider, and it never crosses to a different model.
• upstreamFailoverThreshold — account-level, on the native OpenAI forward path. Different axis: it fails over between accounts of one provider after consecutive transient failures, not between profile candidates.
• Client-side retry with a different model. Works, and is arguably where this belongs. The reason I am asking here anyway: the proxy already holds the candidate list, the health evidence, and the exclusion logic. The client has none of that and would be re-deriving a decision the evaluator already made a moment earlier.
• Waiting for health to shift the order. Measured above — needs a ~33-point health collapse, so it does not address a single failed request.

Additional context

Measured on 2.14.0, Windows 11. Score components from POST /api/routing-profiles/dry-run on a three-candidate profile:

google-antigravity/gemini-3.6-flash    cp=1.000 health=0.985 quota=0.3 cost=0.3  total=0.8134
nvidia/nemotron-3-ultra-550b-a55b      cp=0.667 health=0.966 quota=0.3 cost=0.3  total=0.6879
opencode-zen/deepseek-v4-flash-free    cp=0.333 health=0.958 quota=0.3 cost=0.3  total=0.5665

Two observations from that table, offered as data rather than as separate asks:

quota and cost are 0.3 for all three candidates in every evidence shape I tried, including with an explicit costPerMTokUsd in the request. Weighting them appears to have no effect on ordering — only configuredPriority (list order) and health move, and health spreads just 0.019 across the three.

unknownEvidence.capability: "penalize" does not lower the score. With contextWindow: 900000, a candidate that lacks modelContextWindows keeps a byte-identical total (0.8061478786883453 with and without the evidence) and only gains an exclusions entry. It stays ahead of the one candidate that actually satisfies the requirement. "penalize" reads like it will reorder; it does not.

Checks

• I searched existing issues and documentation.
• This request describes a concrete OpenCodex workflow rather than merely naming a desired technology.
• I removed secrets and personal data.

[Ingwannu]

Confirmed: your description matches current dev. A routing profile is a deterministic selection-time policy, not a request-attempt failover chain. The evaluator chooses one concrete route, and the recursion guard intentionally removes the profile from the child dispatch. Cross-candidate retry today belongs to combos; key/account rotation remains narrower and provider-local.

The scoring observations are also consistent with the documented contract:

• unknownEvidence.capability: "penalize" currently behaves like allow because capability does not yet have a score component; the reference docs state this explicitly.
• Cost and quota can affect ordering only when candidate-scoped evidence exists. Request costPerMTokUsd is not a candidate price source, and an unbound OAuth/account candidate intentionally remains unknown until account selection.

The failover direction is valuable, but I would not add implicit cross-provider replay to profiles as proposed without a larger contract. It needs to preserve request-body replay safety, stream-commit boundaries, account/key retry ordering, combo interaction, per-attempt usage/health accounting, exact candidate exclusion traces, and non-retryable 4xx behavior. A generic operator list containing 400 is especially unsafe because it can replay malformed or policy-rejected requests across paid providers.

The conservative minimum I would accept is one of these two scoped directions:

1. Document profiles as selection-only and compose them with an explicit combo/failover target when outcome-based retry is required.
2. Design a separate opt-in profile failover phase limited initially to pre-stream 429/502/503/connection failures, with a strict attempt cap and immutable exclusion trace. Any 400 retry must use a provider-specific classified error contract, not status alone.

I am keeping this open as an enhancement/design item. It should not be implemented by wiring resolvePolicyProfileId back into the recursive child route; that would violate the existing recursion and trace invariants.