The gap
.github/workflows/web-e2e.yml:172-173 runs:
- name: Install Playwright Chromium
run: pnpm --filter @cipherbox/web-e2e exec playwright install --with-deps chromium
That step intermittently hangs for 10-15+ minutes in a job that otherwise completes in about two minutes. It has no timeout-minutes, no retry, and no cache, so a hang burns the runner until a human notices and cancels it.
Evidence — the healthy case is 24 seconds
Same workflow, same day, same runner image.
| Run |
Step 16 duration |
Step 17 (the actual suite) |
Whole job |
| PR #1298, run 32257383084 |
24s, success |
ran, 34s, success |
2m 12s |
| PR #1300, run 32270890206 attempt 1 |
8m 39s, cancelled |
never ran, skipped |
cancelled |
| PR #1300, run 32270890206 attempt 2 |
14m 11s, cancelled |
never ran, skipped |
cancelled |
| PR #1300, run 32270890206 attempt 3 |
11m+, cancelled |
never ran, skipped |
cancelled |
So this is not a case of the install being inherently expensive. It normally costs 24 seconds. It intermittently stops making progress altogether.
Why it matters
- Three consecutive attempts on one PR never reached the test suite. Step 17 was
skipped every time, so the gate reports on a diff it never exercised.
- A cancelled job surfaces as
Web E2E Smoke Result = failure at the roll-up job, which reads as a test failure and costs triage time on an unrelated diff.
- With no
timeout-minutes, the hang is unbounded — it ends only when someone cancels it by hand.
- Every PR in the current wave needs this gate.
Likely cause
--with-deps performs an apt-get install of OS packages in addition to downloading the browser. Either half can stall: an apt mirror that stops responding, or the browser CDN stalling a connection that has no read timeout. The two halves are not separable today, so the logs cannot even tell us which one hangs.
Fix
- Bound it. Put
timeout-minutes on the step so a hang fails in minutes instead of burning the runner until noticed.
- Separate the halves —
playwright install chromium (download) from playwright install-deps chromium (apt) — so the next occurrence names which one stalls.
- Cache the browser binaries at
~/.cache/ms-playwright, keyed on the resolved Playwright version, so the download half is usually skipped entirely. This is a smaller win than it first appears (24s), but it removes one of the two hang surfaces.
- Consider a retry around the step, since the failure is intermittent rather than deterministic.
Checkable condition
- the step is bounded by an explicit timeout, and a hang fails the job in minutes rather than running unbounded
- a cache hit at an unchanged Playwright version skips the browser download
- the download and the system-dependency install are separate steps, so a future hang is attributable
- the suite itself is unaffected, and a cache miss or version bump still installs correctly
The gap
.github/workflows/web-e2e.yml:172-173runs:That step intermittently hangs for 10-15+ minutes in a job that otherwise completes in about two minutes. It has no
timeout-minutes, no retry, and no cache, so a hang burns the runner until a human notices and cancels it.Evidence — the healthy case is 24 seconds
Same workflow, same day, same runner image.
skippedskippedskippedSo this is not a case of the install being inherently expensive. It normally costs 24 seconds. It intermittently stops making progress altogether.
Why it matters
skippedevery time, so the gate reports on a diff it never exercised.Web E2E Smoke Result = failureat the roll-up job, which reads as a test failure and costs triage time on an unrelated diff.timeout-minutes, the hang is unbounded — it ends only when someone cancels it by hand.Likely cause
--with-depsperforms anapt-getinstall of OS packages in addition to downloading the browser. Either half can stall: anaptmirror that stops responding, or the browser CDN stalling a connection that has no read timeout. The two halves are not separable today, so the logs cannot even tell us which one hangs.Fix
timeout-minuteson the step so a hang fails in minutes instead of burning the runner until noticed.playwright install chromium(download) fromplaywright install-deps chromium(apt) — so the next occurrence names which one stalls.~/.cache/ms-playwright, keyed on the resolved Playwright version, so the download half is usually skipped entirely. This is a smaller win than it first appears (24s), but it removes one of the two hang surfaces.Checkable condition