Conversation
A transient connection-refused from the staging CAS during Bazel's remote-capabilities query fails the build outright (2026-10-01, run 36815211094: the Redis store tester went red on the TraceMachina#2826 merge push and green on the next 14 main runs — pure endpoint blip, unrelated change blamed). Two gaps closed: - tools/bazel-retry.sh: the transient pattern now also matches 'Connection refused', 'Failed to query remote execution capabilities', and gRPC 'UNAVAILABLE:' (still gated on ^ERROR: lines, max 3 attempts). - native-bazel.yaml: the Linux/macOS test arms and the three Redis store tester invocations go through bazel-retry instead of raw bazel. The Windows arm is left raw for now (different startup-flag shape).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A transient
Connection refusedfrom the staging CAS endpoint during Bazel's remote-capabilities query fails a lane outright. Concretely: on the #2826 merge push (run 36815211094, jobRedis store tester), the build died withFailed to query remote execution capabilities: ... Connection refused: cas-*:443(exit 34) before any test ran — and the same lane passed on the next 14mainruns. A pure endpoint blip put a red X on an unrelated scheduler PR.Two gaps closed:
tools/bazel-retry.shexisted for exactly this class but its transient pattern only covered HTTP fetch failures (403/429/5xx,Connection reset/timed out); it would not have matched this gRPC failure. The pattern now also matchesConnection refused,Failed to query remote execution capabilities, and gRPCUNAVAILABLE:— still gated on^ERROR:lines, still max 3 attempts with backoff, so a real test failure does not retry.Bazel Nativeworkflow called rawbazel(only the Nix and LRE workflows used the wrapper). The Linux/macOS test arms and the three Redis store tester invocations now go through./tools/bazel-retry.sh. The Windows arm is left raw in this pass — it passes a startup option (--output_user_root) ahead of the subcommand and deserves its own verification.How was this verified?
The new pattern was checked against the actual failing log of run 36815211094: its
ERROR:lines matchConnection refused/Failed to query remote execution capabilities, so the wrapper would have retried instead of failing the job. Shellcheck/shfmt and YAML validation pass via the repo pre-commit hooks.tools/bazel-retry.shis committed with the executable bit (100755), so the./tools/...invocation works on the runner. Not yet observed over a live endpoint blip — by nature that needs the next one to occur.Risk
Low. Retry only triggers when an
^ERROR:line matches the transient pattern; deterministic build/test failures still fail on the first attempt. Worst case is up to two extra attempts (≈15 s backoff plus rebuild time) during a genuine prolonged outage before the lane fails anyway. The Windows arm's behavior is unchanged.AI assistance
An agent (Claude Code) performed the failure triage and drafted this change and description; I reviewed every line and the gating pattern against the original failure log.