Skip to content

ci: fix nuttx builds failing with 'no space left on device'#4999

Open
matthargett wants to merge 1 commit into
bytecodealliance:mainfrom
rebeckerspecialties:fix/nuttx-enospc-rework
Open

ci: fix nuttx builds failing with 'no space left on device'#4999
matthargett wants to merge 1 commit into
bytecodealliance:mainfrom
rebeckerspecialties:fix/nuttx-enospc-rework

Conversation

@matthargett

@matthargett matthargett commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Problem

compilation on nuttx is intermittently red on main and across PRs — every build_iwasm_on_nuttx leg fails with no space left on device, in two places: the container image pull (Docker pull failed, the dominant mode) and during the build while compiling the from-source bloaty (protobuf/abseil).

Root cause

It's a container: job on the large apache-nuttx-ci-linux image, and it also builds google/bloaty (recursive protobuf/abseil submodules) from source on every matrix leg for the size report. A container job pulls its image at job init, before any step runs, so no step can free disk first — and the runner's default free space plus that from-source build doesn't fit.

Fix

  • Drop container:. Run on ubuntu-latest, remove the unused preinstalled toolchains (~25 GB) as step 1, then run the NuttX configure+build inside the apache image via docker run with the workspace mounted. The image is pulled after disk is freed (verified ~119 GB free), so both modes are fixed.
  • Build bloaty once + cache it. A small build_bloaty job compiles the pinned bloaty in the image and caches the binary, keyed on BLOATY_REF — bump the ref and the cache key changes, so it rebuilds and refreshes automatically. The matrix needs: build_bloaty and restores the prebuilt binary, so 63 protobuf builds collapse to one. The size report (bloaty -d compileunits --source-filter wamr/libc-builtin/libc-wasi) is byte-for-byte unchanged — the MCU binary-size regression signal is fully preserved.
  • Drop a duplicated wamr_config_option matrix entry (it was listed twice), removing a redundant leg.

Verification (on a fork)

  • Free-disk gives /dev/root 145G 26G 119G 18% /; the apache image pulls; all board matrix legs build and link, no ENOSPC.
  • build_bloaty builds once and Cache saved with key: bloaty-<ref>-ubuntu-latest; the matrix legs Cache restored from key: bloaty-<ref>-... (restore, not rebuild) and the size report prints.

CI cost

Measured on the fork runs: building bloaty (protobuf/abseil/…) from source adds ~3 min to every matrix leg (~8–9.5 min per leg vs ~5.7 min with a cache restore), and it was being done on all 56 legs. Doing it once (~7.6 min, and ~free on a warm cache) instead saves roughly 56 × 3 ≈ 170 Actions-minutes per run; dropping the duplicated matrix leg saves ~7 × 5.7 ≈ 40 more — about ~200 Actions-minutes per nuttx run.

At the current ~18 nuttx runs/week on this repo that's roughly ~3,600 Actions-minutes (~60 hours) per week (warm cache; a bloaty-ref bump adds one ~8-min rebuild that run). This is on top of no longer wasting minutes on the ENOSPC-failed runs.

build_iwasm_on_nuttx ran as a `container:` job on the large
apache-nuttx-ci-linux image and built google/bloaty (protobuf/abseil
submodules) from source on every matrix leg for the size report. A
container job pulls its image at job init, before any step can free space,
and the runner's default free space plus that from-source build didn't fit,
so legs intermittently failed with "no space left on device" (both at image
pull and during the bloaty compile). This is flaky on main.

- Drop `container:`; run on ubuntu-latest, remove the unused preinstalled
  toolchains (~25 GB) first, then run the NuttX configure + build inside the
  apache image via `docker run`. The image is pulled after disk is freed and
  everything writes to the mounted host workspace, so it fits (verified
  ~119 GB free on the runner).
- Build bloaty once in a `build_bloaty` job and cache it, keyed on
  BLOATY_REF (bump the ref -> cache miss -> rebuild + refresh). The matrix
  restores the prebuilt binary instead of every leg recompiling bloaty and
  protobuf; the `bloaty -d compileunits --source-filter` output is unchanged.
- Drop a duplicated wamr_config_option matrix entry (it was listed twice).

Verified on a fork run: 119 GB free after cleanup, image pulls, all board
matrix legs build and link, bloaty is built once + cached and the matrix
restores it, and the size report prints - no "no space left on device".
@matthargett
matthargett force-pushed the fix/nuttx-enospc-rework branch from 468ec5a to 82a6d39 Compare July 17, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant