fix(tests): give the Chromium imagegen suites room on CI - #87
Merged
Conversation
Each imagegen render launches a fresh headless Chromium. The 30s budget holds locally, where the whole run.test.ts file finishes in under 2s, but on a loaded two-core CI runner a single launch has already blown past it and failed a run on main. Raise the three Chromium-backed suites to 120s, matching the budget the Next dev-server e2e suite already uses. The tests do not hang, they are just slow under contention, so the higher ceiling costs nothing on a healthy run.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI run #376 failed on the PR branch for #86:
The test is unrelated to that PR's change, and the same suite passed on the main push run minutes later, so it is a flake rather than a regression.
Diagnosis
renderCardlaunches a fresh headless Chromium per render and closes it in afinally, so nothing leaks. Fonts are inlined as data URIs and the OpenAI call is stubbed, so neither the network nor an external service is in the path. What is left is CPU: on a two-core GitHub runner, with the other suites in the samebun testinvocation competing (including one that boots a Next dev server), a single Chromium launch can exceed 30s.Two supporting details:
run.test.tsfile finishes in 1.8s, a 16x margin under the old ceiling. A budget that generous locally and still failing on CI is a budget set against the wrong machine.Change
Raise
setDefaultTimeoutfrom 30s to 120s in the three Chromium-backed suites (imagegen/integration/run,imagegen/integration/render,imagegen/e2e/cli), matching whattests/e2e/llm-endpoints.test.tsalready uses for its dev server. These tests do not hang, so the higher ceiling costs nothing on a healthy run and only buys headroom on a slow one.Verification
bun test209 pass / 0 fail,bun run checkclean,bun run typecheckclean.Separate flake, not addressed here
While running
bun test tests/imagegenunder load I hit a different intermittent failure, three cases inimagegen/e2e/cli.test.tsdying onTypeError: undefined is not an object (evaluating 'result.stdout.toString'), meaningBun.spawnSynccame back with no stdout. It reproduces independently of this change and the file already carries a comment about async spawns deadlocking once the Chromium tests have run in the same process. Repeat runs of the same directory on both main and this branch came back clean, so it needs its own investigation rather than a blind fix.