verification-stack-e2e asserts on whichever of two readiness timeouts wins a race

[khaliqgant]

What happens

verification-stack-e2e failed on PR #327 with:

AssertionError [ERR_ASSERTION]: The input did not match the regular expression
/Service web readiness probe never became ready within 4s/u. Input:
'Service web workload deployment/web never became ready within 4s: kubectl exited with 1:
 error: timed out waiting for the condition'

PR #327 touches only orphan-recovery label gating in src/orchestrator/factory.ts. It cannot reach a Kubernetes readiness probe. This is the job's own timing fragility.

Why it happens

src/environments/verification-stack-deployer.ts can fail readiness in two places, and both spend the same readiness.timeoutSeconds budget:

• line 359 — the workload rollout wait: Service <n> workload <w> never became ready within Ns: <kubectl error>
• line 435 — the HTTP readiness probe: Service <n> <label> probe never became ready within Ns (last error: …)

test/e2e/stack-deployer.e2e.ts builds a deliberately-unready service by pointing the probe at /never-ready with timeoutSeconds: 4, then asserts the probe message at line 99:

assert.match(readinessFailure.message, /Service web readiness probe never became ready within 4s/u)

But the rollout wait runs first and shares that 4s. On a slow kind cluster the pod has not finished rolling out inside 4s, so the rollout wait times out and throws the line-359 message — and the probe the test is actually exercising is never reached.

Note the test's other assertions all passed: stage === 'readiness' and service === 'web' were both correct, and the failure was still bounded. Only the message regex distinguishes the two paths.

Why it is worth fixing rather than re-running

The test intends to prove that an unready readiness probe fails closed and is bounded. Today it can pass for the wrong reason: if the rollout is slow it exercises the rollout timeout instead, and if the rollout is fast it exercises the probe. It is asserting on whichever of two mechanisms happened to win a race, which makes both a red run and a green run weak evidence.

Suggested fix

Any of:

1. Give the rollout wait and the probe separate budgets, so the probe timeout is the only one that can expire in this fixture.
2. Assert on a discriminator that does not depend on which timeout won — e.g. keep stage/service and add an explicit cause/kind field on StackDeploymentError distinguishing workload-rollout from readiness-probe.
3. If both outcomes are genuinely acceptable for this fixture, widen the regex to accept either and say so in a comment — but that weakens what the test proves.

(1) or (2) is preferable; (3) documents the ambiguity rather than removing it.

Frequency

Only failure of this job in the last 12 CI runs (the other 11 across main, fix319-marker-boundary, fix/heartbeat-atomic-write, fix/313-closure-authority all passed) — so it is infrequent, not benign. Filed by Factory Lead while getting #327 green.