RQ-71-MUSL (#1349): ship a static musl Linux asset, prove it builds on every PR — and correct a release doc that has been false since v0.58

.github/workflows/ci.yml

@@ -5751,3 +5751,87 @@ jobs:
           set -euo pipefail
           python3 scripts/oracle_evidence.py "$ORACLE_EVIDENCE_JSONL" --min-oracles 1
 
+
+  # RQ-71-MUSL (#1349): PROVE the portable release asset before promising it.
+  #
+  # `release.yml` only runs on a tag push, so a new target added there is
+  # UNVERIFIED until the tag — and changing the release workflow between an RC
+  # and its tag is the unreviewed late change this process exists to prevent
+  # (v0.70 shipped the 2.39 floor rather than make one). This job builds the
+  # SAME target with the SAME feature set on every PR, and runs the issue's own
+  # two acceptance commands on the result, so the release.yml matrix entry is
+  # verified by construction rather than on the day it matters.
+  #
+  # MEASURED on the published v0.70.0 x86_64 asset: `ELF 64-bit LSB pie
+  # executable, dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2`,
+  # max GLIBC_ symbol GLIBC_2.39 — excluding Ubuntu 22.04 (2.35), Debian 12
+  # (2.36), RHEL 9 / AL2023 (2.34), Ubuntu 20.04 (2.31) and Alpine entirely.
+  # These binaries feed the varve `pulseengine` realm's rolling layer, whose
+  # portability is the MAXIMUM floor across its payloads.
+  #
+  # Deliberately NOT a required context: it is new, and a required name that has
+  # never run deadlocks merges. It reddens the PR that breaks it, which is the
+  # job's whole purpose.
+  musl-portable-asset:
+    name: Portable musl asset builds and is static (#1349)
+    # SELF-HOSTED on purpose. 56 of this file's 67 jobs target `ubuntu-latest`,
+    # which RQ-61-CICAP measured as THE bottleneck: runs queue for hours beside
+    # idle self-hosted machines, because the constraint is the GitHub-hosted
+    # quota and not the fleet. A new job defaulting to ubuntu-latest makes that
+    # worse for every other lane, so this one goes to the pool that has capacity.
+    runs-on: [self-hosted, linux, x64, rust-cpu]
+    steps:
+      - uses: actions/checkout@v7
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          targets: x86_64-unknown-linux-musl
+      - uses: Swatinem/rust-cache@v2
+        with:
+          key: musl-portable
+      # `musl-gcc` is the linker driver rustc invokes for this target IF it is
+      # not self-containing the link. Whether rust's bundled musl crt +
+      # `rust-lld` make this unnecessary for a pure-Rust workspace is UNVERIFIED
+      # here — it cannot be checked on this macOS host (no musl std installed,
+      # no nightly for `--print target-spec-json`), so it is not asserted either
+      # way. Installed only when absent and never fatal: if the link genuinely
+      # needs a driver that is missing, the BUILD step below says so loudly,
+      # which is the honest place for that answer rather than a guess here.
+      - name: Install musl toolchain if absent
+        run: |
+          if command -v musl-gcc >/dev/null 2>&1; then
+            echo "musl-gcc already present: $(command -v musl-gcc)"
+          elif command -v apt-get >/dev/null 2>&1; then
+            sudo apt-get update && sudo apt-get install -y musl-tools || \
+              echo "musl-tools install failed; letting the build be the judge"
+          else
+            echo "no apt-get on this runner; letting the build be the judge"
+          fi
+      # The SAME command release.yml runs for this target. If these diverge the
+      # job stops testing the thing it claims to test.
+      - name: Build synth for x86_64-unknown-linux-musl
+        run: cargo build --release --target x86_64-unknown-linux-musl -p synth-cli --features verify
+      # The PROPERTY, read from the ELF — not `file`'s wording. The first
+      # version of this grepped `file` for "statically linked", which rustc's
+      # musl target never produces: its spec sets crt-static-default AND
+      # static-position-independent-executables, so the link is `-static-pie`
+      # and `file` says "static-pie linked". That gate would have REFUSED A
+      # CORRECT ARTIFACT (v0.71 cold review). "No loader" is exactly "no
+      # PT_INTERP", which is read directly and needs no `file` and no `strings`.
+      - name: The issue's acceptance property, on the built artifact
+        run: python3 scripts/elf_static_check.py target/x86_64-unknown-linux-musl/release/synth
+      # POTENCY: the SAME check must FAIL on the gnu build, or it proves nothing
+      # about this binary — it would pass on anything. This now exercises BOTH
+      # assertions (PT_INTERP and GLIBC_), which the previous version did not:
+      # its gnu block never asserted the loader was present, so the musl
+      # `interpreter` assertion had no negative control (v0.71 cold review).
+      - name: Potency — the SAME check must FAIL on the gnu target
+        run: |
+          set -euo pipefail
+          cargo build --release --target x86_64-unknown-linux-gnu -p synth-cli --features verify
+          if python3 scripts/elf_static_check.py \
+               target/x86_64-unknown-linux-gnu/release/synth; then
+            echo "FAIL(potency): the gnu build passed the static check, so the"
+            echo "musl result above is vacuous — it would pass on anything."
+            exit 1
+          fi
+          echo "potency OK: the gnu build is correctly REFUSED by the same check"

.github/workflows/release.yml

@@ -55,6 +55,28 @@ jobs:
             os: ubuntu-latest
             archive: tar.gz
             cross: true
+          # RQ-71-MUSL (#1349). The gnu assets above are built on ubuntu-latest
+          # and carry a GLIBC_2.39 floor — MEASURED on the published v0.70.0
+          # x86_64 asset: `ELF 64-bit LSB pie executable, dynamically linked,
+          # interpreter /lib64/ld-linux-x86-64.so.2`, max GLIBC_ symbol
+          # GLIBC_2.39. That excludes Ubuntu 22.04 (2.35), Debian 12 (2.36),
+          # RHEL 9 and Amazon Linux 2023 (2.34), Ubuntu 20.04 (2.31), and
+          # Alpine/distroless-static entirely. These binaries are ingested into
+          # the varve `pulseengine` realm's rolling layer, and a layer's
+          # portability is the MAXIMUM floor across its payloads, so one 2.39
+          # payload sets the floor for everything pinned beside it.
+          #
+          # This target is ADDITIVE on purpose: the gnu assets keep their
+          # names, their consumers and their attestation path unchanged, and
+          # the static musl build is published ALONGSIDE as the portable
+          # payload. Statically linked, so it has no loader and no glibc floor
+          # at all rather than a lower one. `--features verify` is pure Rust
+          # here (no build.rs, no build-dependencies anywhere in the
+          # workspace), which is why this needs no cross container.
+          - target: x86_64-unknown-linux-musl
+            os: ubuntu-latest
+            archive: tar.gz
+            musl: true
           # x86_64-apple-darwin cross-compiles on the arm64 macos-14
           # runner — matches pulseengine/rivet and pulseengine/witness
           # (both Rust-workspace CLIs, the closest analogs to synth).
@@ -77,6 +99,13 @@ jobs:
         with:
           key: release-${{ matrix.target }}
 
+      # RQ-71-MUSL (#1349): `musl-tools` supplies `musl-gcc`, the linker driver
+      # rustc invokes for this target. The workspace has no C dependencies and
+      # no build.rs, but the target's own libc.a still links through it.
+      - name: Install musl toolchain
+        if: matrix.musl
+        run: sudo apt-get update && sudo apt-get install -y musl-tools
+
       - name: Install cross
         if: matrix.cross
         run: cargo install cross --git https://github.com/cross-rs/cross --locked
@@ -89,6 +118,30 @@ jobs:
         if: matrix.cross
         run: cross build --release --target ${{ matrix.target }} -p synth-cli --features verify
 
+      # RQ-71-MUSL (#1349): the ISSUE'S OWN two acceptance commands, executed
+      # on the artifact this job just built, BEFORE it is packaged. A portable
+      # asset that is silently dynamic, or that picked up a glibc symbol, must
+      # redden the release rather than ship and be discovered by a consumer —
+      # which is exactly how the 2.39 floor was found in the first place. Run
+      # before `strip`, because stripping is what would hide the symbols.
+      # RQ-71-MUSL (#1349): the ISSUE'S acceptance property, on the artifact
+      # this job just built, BEFORE it is packaged — and read from the ELF
+      # rather than from `file`'s wording.
+      #
+      # CORRECTED after the v0.71 cold review, and this one mattered: the first
+      # version grepped `file` for "statically linked", which rustc's musl
+      # target NEVER produces (its spec sets crt-static-default AND
+      # static-position-independent-executables, so the link is `-static-pie`
+      # and `file` reports "static-pie linked"). The gate would have refused a
+      # CORRECT artifact — and `create-release` has `needs: [build-binaries]`,
+      # so that is not one missing archive, it is NO RELEASE AT ALL. A gate
+      # that cannot pass, in the workflow that publishes the release.
+      #
+      # Runs before `strip`, because stripping is what would hide the symbols.
+      - name: Verify the musl asset has no loader and no glibc floor
+        if: matrix.musl
+        run: python3 scripts/elf_static_check.py "target/${{ matrix.target }}/release/synth"
+
       - name: Strip binary
         if: ${{ !matrix.cross }}
         run: strip "target/${{ matrix.target }}/release/synth" 2>/dev/null || true

artifacts/release-v0.71/RQ-71-ISSUESCOPE.yaml

@@ -95,5 +95,5 @@ artifacts:
       priority: should
       verification-track: differential
       issue: "#1250"
-      landed: "#1250"
+      landed: "#1250 (PR #1356)"
       done-when: "contains:.github/workflows/ci.yml:test_issue_closure_check.py"

artifacts/release-v0.71/RQ-71-MUSL.yaml

@@ -1,62 +1,117 @@
 artifacts:
   - id: RQ-71-MUSL
     type: system-req
-    title: "Linux release assets are glibc-2.39-only, so one payload sets the portability floor for the whole varve pinned layer"
+    title: "Linux release assets were glibc-2.39-only, so one payload set the portability floor for the whole varve pinned layer — a static musl asset now ships alongside"
     description: >
-      #1349, 2026-09-22. Measured on the v0.69.0 Linux x86_64 release asset,
-      downloaded from this repo: `ELF 64-bit LSB pie executable, dynamically
-      linked, interpreter /lib64/ld-linux-x86-64.so.2`, and
-      `strings -a synth | grep -oE 'GLIBC_[0-9.]+' | sort -V | tail -1` gives
-      **GLIBC_2.39**.
-
-      That excludes Ubuntu 22.04 (2.35), Debian 12 bookworm (2.36), RHEL 9 and
-      Amazon Linux 2023 (2.34), Ubuntu 20.04 (2.31), and Alpine / distroless-static
-      entirely.
-
-      WHY IT IS NOT MERELY A CONVENIENCE. These binaries are ingested into the varve
-      `pulseengine` realm's rolling layer — the pinned toolchain an air-gapped or
-      enterprise consumer installs. A layer's portability is the MAXIMUM floor
-      across its payloads, so a single 2.39 payload sets the floor for everything
-      pinned beside it.
-
-      v0.70 SHIPS WITH THIS TOO, AND SAID SO. The issue arrived on the day of the
-      v0.70 cut. Changing the release workflow between an RC and its tag is exactly
-      the unreviewed late change this process exists to prevent, so v0.70's assets
-      carry the same 2.39 floor and its release report states that plainly rather
-      than leaving a consumer to discover it.
-
-      REFUTING COMMANDS TO RUN BEFORE SCOPING, not after. First: inspect the
-      v0.70.0 asset the moment it publishes, with the same two commands — if the
-      floor is already lower, the runner image moved under us and the premise needs
-      re-measuring. Second: `grep -n 'musl' .github/workflows/*.yml` — measured at
-      the v0.70 cut, the only hit is `fuzz-smoke.yml`, so no musl RELEASE target
-      exists today; publishing an artifact that is already built would be a much
-      smaller lane than adding a target, and the two must not be confused.
-
-      HONEST RISK: this is release engineering, not compiler work, and it will look
-      cheaper than it is. A second target means a second set of assets, a second
-      path through sigil attestation, and a varve layer decision about which
-      payload is canonical. Measure before promising.
-
-      THE MACHINE'S HALF OF THIS IS `contains:.github/workflows/release.yml:musl`. The full obligation is:
-      a Linux release asset is published whose dynamic-loader floor is verified by the issue's own two commands (file + the GLIBC_ strings scan) to run on Ubuntu 22.04, Debian 12 and RHEL 9, or a musl-static asset is published alongside; and the release process documents which asset varve's layer pins
-      The predicate cannot express all of that — it checks that the artefact
-      EXISTS, not that it is right. It is still worth having: v0.70 measured
-      that four releases running declared 100% `manual:` done-when, which R3
-      can never fire on, and printed that as though it were evaluation.
-
-    status: proposed
+      #1349, 2026-09-22. Both refuting commands were run again on this tree
+      before anything was scoped, and BOTH CONFIRMED the premise rather than
+      narrowing it.
+
+      FIRST: the v0.70.0 asset, the moment it published — if the floor had
+      already dropped, the runner image moved and the premise needed
+      re-measuring. It had not. Downloaded from this repo and measured with the
+      issue's own two commands: `ELF 64-bit LSB pie executable, dynamically
+      linked, interpreter /lib64/ld-linux-x86-64.so.2`, and the maximum of 17
+      distinct `GLIBC_` version symbols is **GLIBC_2.39**.
+
+      SECOND: `grep -n 'musl' .github/workflows/*.yml` — one hit, in
+      `fuzz-smoke.yml`, and it is there to FORCE THE GNU TARGET because
+      cargo-fuzz defaults to musl. So no musl release target existed, and this
+      is the larger of the two lanes the plan distinguished: adding a target,
+      not publishing one already built.
+
+      A 2.39 floor excludes Ubuntu 22.04 (2.35), Debian 12 (2.36), RHEL 9 and
+      Amazon Linux 2023 (2.34), Ubuntu 20.04 (2.31), and Alpine /
+      distroless-static entirely. These binaries are ingested into the varve
+      `pulseengine` realm's rolling layer, and a layer's portability is the
+      MAXIMUM floor across its payloads, so one 2.39 payload sets the floor for
+      everything pinned beside it.
+
+      WHAT SHIPPED. `x86_64-unknown-linux-musl`, statically linked — no loader
+      and no glibc floor at all, rather than a lower one. ADDITIVE on purpose:
+      the gnu assets keep their names, their consumers and their attestation
+      path unchanged, so nothing that works today changes. The workspace has no
+      `build.rs` and no `[build-dependencies]` anywhere, which is why this needs
+      no cross container.
+
+      THE PART THAT IS NOT RELEASE PLUMBING, and the reason this lane is not as
+      cheap as it looks: `release.yml` runs ONLY on a tag push, so a target added
+      there is UNVERIFIED until the tag — and changing the release workflow
+      between an RC and its tag is precisely the unreviewed late change this
+      process exists to prevent (v0.70 shipped the 2.39 floor rather than make
+      one). So a per-PR CI job builds the SAME target with the SAME feature set
+      and runs the issue's own two acceptance commands on the result. The
+      release.yml entry is verified by construction, not on the day it matters.
+
+      WITH A POTENCY CONTROL, because "is statically linked" and "has no GLIBC_
+      symbols" are assertions that could pass on anything: the same job builds
+      the GNU target on the same runner and asserts those checks FAIL there. If
+      they ever pass on both, the musl check proves nothing and the job says so.
+
+      A FALSE DOC STATEMENT FOUND ON THE WAY, and corrected in place rather than
+      left. `docs/release-process.md` said the `verify` feature is "**not**
+      enabled" in release builds and that enabling it "remains a deliberate
+      follow-up decision". That decision was TAKEN in v0.58 — RQ-58-SHIPVERIFY
+      (#1000, PR #1002) put `--features verify` on both the native and `cross`
+      build lines — so the sentence had been false for thirteen releases.
+      Measured on the shipped v0.70.0 asset: 19 `ordeal` byte-occurrences
+      (case-sensitive; 21 case-insensitively — the earlier text said "20",
+      which is neither, and the v0.71 cold review caught it) and ZERO
+      occurrences of the degraded-path message, so `synth verify` works in a
+      released binary. Nobody would have found this from the workflow alone;
+      it surfaced because this lane had to read what the release actually
+      builds.
+
+      THE COLD REVIEW FOUND THIS LANE'S GATE COULD NOT PASS, and it is recorded
+      rather than quietly fixed. The acceptance step asserted
+      `file "$BIN" | grep -q 'statically linked'`. rustc's musl target NEVER
+      produces that string: its spec sets `crt-static-default` AND
+      `static-position-independent-executables`, so the link is `-static-pie`
+      and `file` reports "static-pie linked". The gate would have REFUSED A
+      CORRECT ARTIFACT — and `create-release` has `needs: [build-binaries]`, so
+      that is not one missing archive, it is NO RELEASE AT ALL. A gate that
+      cannot pass, in the workflow that publishes the release, in the lane whose
+      whole point is that the asset is checkable.
+
+      The fix is not a better string. `file`'s wording is a presentation detail
+      of a tool this repo does not control; the property is "asks no loader to
+      run it", which is exactly "no PT_INTERP program header".
+      `scripts/elf_static_check.py` reads it from the ELF with nothing beyond
+      the stdlib, and is validated BOTH ways before shipping: it REFUSES the
+      real published v0.70.0 gnu asset (PT_INTERP=1, 17 GLIBC_ occurrences,
+      rc=1) and ACCEPTS a synthesised static-pie shape (rc=0).
+
+      The potency control was widened at the same time: the gnu leg now runs the
+      SAME check and must fail it, so both assertions have a negative control.
+      Previously the gnu block never asserted a loader was present, leaving the
+      musl `interpreter` assertion uncontrolled.
+
+      AND THE JOB WAS PUT ON THE POOL THAT HAS CAPACITY. 56 of ci.yml's 67 jobs
+      target `ubuntu-latest`, which RQ-61-CICAP measured as THE bottleneck —
+      runs queue for hours beside idle self-hosted machines, because the
+      constraint is the GitHub-hosted quota, not the fleet. This job now runs on
+      `[self-hosted, linux, x64, rust-cpu]`, and its apt step is conditional and
+      non-fatal: whether rust's bundled musl crt makes `musl-tools` unnecessary
+      is UNVERIFIED here (it cannot be checked on this macOS host) and is
+      therefore not asserted — the build step is left to answer it loudly.
+
+      RESIDUAL, NAMED. arm64 Linux is still glibc-floored — only the x86_64 musl
+      asset is published. An `aarch64-unknown-linux-musl` asset needs the `cross`
+      path and its own measurement, and is NOT claimed here.
+
+    status: implemented
     release: v0.71
     tags:
       - release-engineering
-      - downstream-consumer
+      - downstream-blocker
       - feature-loop
     links:
       - type: derives-from
         target: BR-001
     fields:
-      req-type: constraint
+      req-type: process
       priority: should
       verification-track: differential
       issue: "#1349"
-      done-when: "contains:.github/workflows/release.yml:musl"
+      landed: "#1349 (PR #1357)"
+      done-when: "contains:.github/workflows/ci.yml:musl-portable-asset"