Native arm64 for dtest: Chrome for Testing 153, CI on ARM runners - #589
Merged
Conversation
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Screenshot diffs detected
|
Chrome for Testing began publishing linux-arm64 at 153 - chrome AND chromedriver - so the amd64 pin is no longer forced. .dev/cft-version moves to 153.0.8010.5 (the first Beta with it; a fixed version string, so nothing floats), the Dockerfile and bin/setup-test-env pick the CfT platform from the build/host arch instead of hardcoding linux64, and both `platform: linux/amd64` pins come out of compose so the container follows the host. The win is the emulation tax: the container was running amd64 under emulation on an ARM Mac, which is what mem_limit: 4g and the "Chrome 152 needs >2g under amd64 emulation (OOM tab crashed at 2g)" note were paying for. Verified native: uname -m reports aarch64, Chrome for Testing 153.0.8010.5, chromedriver matching, Debian 13. CI moves to ubuntu-24.04-arm in the same commit - free for a public repo - so both sides stay on one architecture. Doing it together matters: bumping 152 -> 153 invalidates the linux/ baselines by itself, so splitting the work would pay for a full re-record twice. Two things this needed that were not in the plan: - Gemfile.lock carried aarch64-linux-musl (Alpine) but not aarch64-linux (glibc), so nokogiri had no native candidate for a Debian ARM container. Added via `bundle lock --add-platform aarch64-linux`. That gap would have broken the ARM runners too. - bin/dc's DOCKER_DEFAULT_PLATFORM=linux/arm64/v8 export is deleted rather than kept-and-now-correct. It never affected the test services (they pinned linux/amd64 and won) and it is what made four separate "must be an arch problem" diagnoses look plausible on 2026-08-22. MEASURED, and the reason this is safe: after changing BOTH the architecture and the Chrome major version, 47 of 55 screenshots still match baselines recorded on amd64 Chrome 152. The 8 that do not are exactly the mobile codeblocks family that already differed before this change, at the same magnitudes - so that divergence is neither arch nor Chrome version. It is the remaining Debian-container vs Ubuntu-runner font stack, and it closes only by running CI inside this image (see .okf/build/rendering-stack.md). Re-record linux/ via an update-baselines dispatch ON MASTER after this merges - recorder and tester must see the same tree. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Defects first, because they were silent: - `bundle config set --local` wrote /app/.bundle/config, and the `..:/app` bind mount replaces /app at runtime, so none of those settings reached the running container. BUNDLE_PATH/JOBS/RETRY happened to be ENV as well and covered for it; `without development:assets` did not. Now BUNDLE_WITHOUT alongside the others. Proven: `bundle config get without` in the container reports "Set via BUNDLE_WITHOUT: [:development, :assets]". - compose declared `node_modules:` and `bundle:` named volumes that no service referenced. Removed. Then the optimizations, per current Docker guidance: - `# syntax=docker/dockerfile:1`, so cache-mount syntax does not depend on bin/dc exporting DOCKER_BUILDKIT. - apt cache mounts on /var/cache/apt and /var/lib/apt, with `rm -f /etc/apt/apt.conf.d/docker-clean` FIRST - Debian's hook deletes the downloaded debs, so without removing it the cache mount does nothing. The old `rm -rf /var/lib/apt/lists/*` came out with it: once the lists live in a mount, purging them is wrong rather than redundant. - Both VOLUME lines dropped. `VOLUME /opt/bundle` was the harmful one - a persisted volume there serves stale gems after any image rebuild; `bundle check` now reports satisfied from the image layer. The node_modules shield moved to an explicit ANONYMOUS mount in compose, which keeps the host's macOS binaries out (354 entries present in-container) without persisting stale modules between runs. Not claimed: a rebuild speedup. The apt cache only pays when that layer re-runs and no cold-vs-warm comparison was measured. The compose `hugo` service was checked and KEPT - bin/build drives production builds through it, so it is not vestigial as first assumed. dtest after the change: same 8 codeblocks failures as before, no regression. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
…sserted The rendering-stack concept and STATUS.md both still described a stack that no longer exists: "wait for Chrome 153 Stable, then go all-ARM", with a two-stack table claiming x86_64 on both sides and Chrome 152. Paul overrode the pin-to-Stable policy on 2026-08-21 and the migration shipped, so the trigger row moves out of "Blocked on Paul" into Now/WIP with the re-record as its next step. The substantive change is a retraction. The previous OKF entry concluded the 8 mobile/blog/special/codeblocks failures were caused by Debian-vs-Ubuntu, and #589's commit message repeated it. That conclusion is not established: local ARM (Debian) and CI ARM (Ubuntu) fail the SAME 8 keys, which reads as confirmation but cannot discriminate, because both are compared against baselines recorded on the old amd64/Chrome-152 stack - every hypothesis predicts red on both. That is the fifth instance of the exact failure shape the concept already catalogues four of: a mechanical-sounding cause asserted without measuring the thing it names. It is now recorded as the fifth, with the discriminating experiment written down - re-record on master, then run local bin/dtest; green means the distro never mattered, same 8 red means it does. Also fixes an indexing gap: rendering-stack.md was never listed in .okf/build/index.md, so the concept was unreachable by progressive disclosure from the section index. Bundle validates with zero ERRORs (--strict still exits 1 on the pre-existing known-red warning set, tracked in the 2608 README). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
…ified Both of these were left behind by #589 and both point a reader the wrong way. bin/dtest justified discarding local screenshot candidates by arch: "bin/dc pins DOCKER_DEFAULT_PLATFORM=linux/arm64/v8 while the committed linux/ baselines are recorded by CI on amd64, so a candidate produced here is never committable". #589 deleted that export and moved CI to arm64, so the stated reason is now false while the conclusion is still right. That combination is the dangerous one: a reader who checks the premise finds the arches now match and concludes local candidates ARE committable, which is one `git add -A` from corrupting the baseline set. Rewritten to name the reason that actually still holds - the base OS is the last unpinned rendering variable. docs 20.10 O3 recorded "the 10 red Linux baselines are NOT stale - they are emulation drift: green on CI-native amd64, red only under local Apple-Silicon emulation." Emulation ended on 2026-08-22 and the reds did not, so emulation was never the cause. Marked withdrawn; the operational rule it carried (never re-record locally, trust CI) is unchanged and still correct. No behaviour change - comments and prose only. bash -n clean, unit 289 runs / 6185 assertions / 0 failures, hugo-build green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Codex review caught this on the branch and it is a real hole. Dropping the
`platform: linux/amd64` pins made Compose follow the HOST arch, which is only
equivalent to CI's arm64 when the host happens to be ARM. On an Intel Mac or
x86_64 Linux box the container would render amd64 Chrome and compare it against
test/fixtures/screenshots/linux/ - which CI records on ubuntu-24.04-arm. Local
bin/dtest and the required CI check would then disagree by construction, on a
gate CLAUDE.md marks BLOCKING. Worse, the comment on the `t` service asserted
the guarantee ("Host arch, matching CI's arm64 runners") that the code only
delivers on an ARM host, so a reader had no reason to look.
Fix is a pin to arm64 - CI's arch - on the two services that build screenshot
candidates (`t`, `sh`). This is not the amd64 pin coming back: it pins to the
arch that records the baselines rather than to the wrong one. On an ARM Mac the
pin IS the host arch, so the emulation win the migration was for is untouched;
it costs only an x86 host, which emulates arm64 and is correct-but-slow instead
of fast-and-wrong. `hugo` is deliberately left following the host - it serves
pages and produces no screenshots.
bin/setup-test-env (bare-metal Linux) keeps following the host, because there
is nothing to pin to: an x86 CPU cannot execute arm64 Chrome. That host is
structurally unable to reproduce the baselines, so it now says so on stderr
rather than emitting a suite of false reds that someone could "fix" by
re-recording.
VERIFIED, not assumed: `docker compose config` shows platform linux/arm64 on
both services and none on hugo; the container reports `aarch64` and Chrome for
Testing 153.0.8010.5 under the pin; the x86 branch of setup-test-env was
exercised directly and fires the warning; bash -n clean on the script; unit
289 runs / 6185 assertions / 0 failures; hugo-build green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
…tent Adds the rule the Codex finding produced: pin the test container to CI's arch, never to "the host's". The two coincide only while every developer is on the same architecture, so "follow the host" is a guarantee that expires silently on the first machine that differs - and no suite fails to announce it, because the divergent host is one nobody has yet. That is why review caught it and gates did not, which is the part worth remembering. Also repairs inconsistencies my own earlier edit introduced in the same file: the section still said "Four wrong explanations" after a fifth was documented below it, explanation #1 described the container in the present tense as x86_64 when it is now arm64, and a cross-reference pointed "below" at something that had moved above it. The frontmatter description still advertised four. Bundle validates: 0 ERRORs (--strict exits 1 on the pre-existing known-red warning set only, tracked in the 2608 README). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Both close questions my own comments had left open, which is the defect worth fixing here - a comment that reads "left until measured" is an invitation to reopen, and the next session would have spent time re-deciding. "Our last stack is arm until we will find that cannot support it" (Paul, 2026-08-22). arm64 is THE stack - container, CI, baselines - not a preference and not a trial. Reversed only on evidence something genuinely CANNOT support it. Two things that do not qualify, named explicitly because both are close to hand: "an x86 box would be slower" is a cost, not an inability; and the 8 unexplained codeblocks reds are a measurement job, not grounds to retreat - they predate the migration and survived it unchanged, so they are not evidence against ARM at all. That loose end is exactly what would otherwise invite a retreat to the old stack, so it is called out where the decision is recorded. This also sharpens the compose comment from the previous commit, which framed x86 as a supported-but-slower path. It is not a supported rendering host; the pin is the policy. `mem_limit: 4g` stays, settled rather than pending. The prior comment said "left at 4g until measured rather than guessed down", which invited exactly the optimization Paul has now ruled out. Headroom on a test container is free, and an OOM-killed Chrome is expensive twice over because it presents as a rendering difference rather than as an OOM. Recorded in .okf/build/rendering-stack.md, STATUS.md and ruflo memory (decision-arm64-is-the-stack) so a cold session finds it before relitigating. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chrome for Testing began publishing
linux-arm64at 153 (chrome and chromedriver), so the amd64 pin is no longer forced.What changed
.dev/cft-version→ 153.0.8010.5 (first Beta with ARM; a fixed version string, nothing floats) · Dockerfile andbin/setup-test-envpick the CfT platform from the arch instead of hardcodinglinux64· bothplatform: linux/amd64pins removed so the container follows the host · CI →ubuntu-24.04-arm(free for public repos).Verified native:
aarch64, Chrome for Testing 153.0.8010.5, chromedriver matching, Debian 13. That removes the emulation tax — which is whatmem_limit: 4gand the "Chrome 152 needs >2g under amd64 emulation (OOM tab crashed at 2g)" note were paying for.Local and CI move together deliberately: bumping 152→153 invalidates the
linux/baselines by itself, so splitting the work would pay for a full re-record twice.The measurement that makes this safe
After changing both the architecture and the Chrome major version, 47 of 55 screenshots still match baselines recorded on amd64 Chrome 152.
The 8 that don't are exactly the
mobile/blog/special/codeblocks/*family that already differed before this change, at the same magnitudes (0.036–0.063). So that divergence is neither arch nor Chrome version — it's the remaining Debian-container vs Ubuntu-runner font stack, and it closes only by running CI inside this image (rendering-stack).Rendering turns out to be far more stable across arch and browser version than the "drift" folklore assumed; the distro is the variable that actually moves pixels.
Two things the plan didn't anticipate
Gemfile.lockhadaarch64-linux-musl(Alpine) but notaarch64-linux(glibc), so nokogiri had no native candidate for a Debian ARM container. Fixed withbundle lock --add-platform aarch64-linux— that gap would have broken the ARM runners too.bin/dc'sDOCKER_DEFAULT_PLATFORM=linux/arm64/v8export is deleted, not kept-and-now-correct. It never affected the test services (they pinnedlinux/amd64and won) and it's what made four separate "must be an arch problem" diagnoses look plausible.After merge
Re-record
linux/via anupdate-baselinesdispatch on master — recorder and tester must see the same tree. Expect the 8 codeblocks keys plus whatever the 152→153 bump moves.🤖 Generated with Claude Code
https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ