Skip to content

ci: the web-e2e Playwright install intermittently hangs for 10+ minutes, unbounded and uncached #1309

Description

@FSM1

The gap

.github/workflows/web-e2e.yml:172-173 runs:

- name: Install Playwright Chromium
  run: pnpm --filter @cipherbox/web-e2e exec playwright install --with-deps chromium

That step intermittently hangs for 10-15+ minutes in a job that otherwise completes in about two minutes. It has no timeout-minutes, no retry, and no cache, so a hang burns the runner until a human notices and cancels it.

Evidence — the healthy case is 24 seconds

Same workflow, same day, same runner image.

Run Step 16 duration Step 17 (the actual suite) Whole job
PR #1298, run 32257383084 24s, success ran, 34s, success 2m 12s
PR #1300, run 32270890206 attempt 1 8m 39s, cancelled never ran, skipped cancelled
PR #1300, run 32270890206 attempt 2 14m 11s, cancelled never ran, skipped cancelled
PR #1300, run 32270890206 attempt 3 11m+, cancelled never ran, skipped cancelled

So this is not a case of the install being inherently expensive. It normally costs 24 seconds. It intermittently stops making progress altogether.

Why it matters

  • Three consecutive attempts on one PR never reached the test suite. Step 17 was skipped every time, so the gate reports on a diff it never exercised.
  • A cancelled job surfaces as Web E2E Smoke Result = failure at the roll-up job, which reads as a test failure and costs triage time on an unrelated diff.
  • With no timeout-minutes, the hang is unbounded — it ends only when someone cancels it by hand.
  • Every PR in the current wave needs this gate.

Likely cause

--with-deps performs an apt-get install of OS packages in addition to downloading the browser. Either half can stall: an apt mirror that stops responding, or the browser CDN stalling a connection that has no read timeout. The two halves are not separable today, so the logs cannot even tell us which one hangs.

Fix

  1. Bound it. Put timeout-minutes on the step so a hang fails in minutes instead of burning the runner until noticed.
  2. Separate the halvesplaywright install chromium (download) from playwright install-deps chromium (apt) — so the next occurrence names which one stalls.
  3. Cache the browser binaries at ~/.cache/ms-playwright, keyed on the resolved Playwright version, so the download half is usually skipped entirely. This is a smaller win than it first appears (24s), but it removes one of the two hang surfaces.
  4. Consider a retry around the step, since the failure is intermittent rather than deterministic.

Checkable condition

  • the step is bounded by an explicit timeout, and a hang fails the job in minutes rather than running unbounded
  • a cache hit at an unchanged Playwright version skips the browser download
  • the download and the system-dependency install are separate steps, so a future hang is attributable
  • the suite itself is unaffected, and a cache miss or version bump still installs correctly

Metadata

Metadata

Assignees

No one assigned

    Labels

    comp:ciCI gates, workflows, test/perf harnessv2-buildv2 rewrite build slice

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions