koga: fix two bugs that let the generated lisp-info go stale or corrupt

[dg1sbg]

Two related bugs in how the generated lisp-info files are produced. Both let a build proceed with generated data that does not match the source tree, and they interact, so they are fixed together here. Fixes #1823 and #1828.

Originally submitted as two PRs (#1826 and #1829); combined at review request.

---

1. The generator almost never runs (#1828)

build …/generated/features.sexp …/runtime-packages.lisp …:
    generate-lisp-info | boehmprecise/iclasp             <- implicit dep: iclasp only

build boehmprecise/iclasp: link …/main.o || boehmprecise/lib/libclasp.so
                                                         <- ORDER-ONLY dep (||)

generate-lisp-info runs iclasp, which loads libclasp at runtime via rpath and dumps the packages, functions, variables, classes and type map it finds there — so its output depends on libclasp's contents. But the only path to that library is through iclasp, which depends on it order-only: in ninja that means "build it first if missing, but changes to it do not make me dirty". iclasp is just main.o plus a link, so it rarely changes on its own.

Declaring iclasp as an implicit input shows the dependency was intended; it simply names the wrong artifact.

generated stamp: 2026-08-03 23:42:47
libclasp.so:     2026-08-04 00:05:50        <- newer
$ touch build/boehmprecise/lib/libclasp.so && ninja -C build
ninja rc=0   generated stamp: 2026-08-03 23:42:47    <- unchanged
$ ninja -C build -n boehmprecise/generated/runtime-packages.lisp
ninja: no work to do.

Worse in practice: walking a single tree through eight branches, rebuilding at each, left the generated files at their original timestamp and byte sizes throughout — including across a branch adding C++ sources and therefore new scraped symbols. Eight images, each built from another branch's lisp-info.

2. When it does run, it can leave a stale tail (#1823)

The generator opens its eight outputs with :if-exists :overwrite and never closes them. :overwrite writes over an existing file without truncating, so shorter new content leaves the previous version's tail past the new EOF.

Whether that breaks loudly or silently depends on whether the leftover bytes happen to parse. Loudly: the build dies in compile-bytecode-image.lisp with an Eclector reader error whose position is a few bytes short of the file size, behind a wall of WARNING: Unknown variable ...: treating as special that is only the reader recovering and points nowhere near the cause. Silently: the build succeeds on mismatched definitions with no diagnostic at all. Measured — after switching to a shorter branch without clearing, runtime-packages.lisp kept 178 B and runtime-functions.lisp 214 B of the previous branch, ninja returned 0, and the suite ran to completion reporting success.

:supersede alone does not fix this. It is not required to replace the old file until the stream is closed, and this script closes none of its eight. Verified in clasp, writing 5 bytes over a 1000-byte file:

	resulting size
:supersede, no close	1000
:supersede + close	5
:overwrite + close	1000

So each stream is bound and closed as well as switched to :supersede. Only let and close are used, since this script runs in primitive Clasp before any image loads.

Why together

They are opposite halves of the same failure. Today the truncation bug is rare precisely because the generator seldom runs; fixing the dependency alone would make it fire constantly. Fixing the truncation alone leaves the files stale. Ordered here so the truncation fix precedes the dependency fix within the branch.

The truncation risk applies to a reused build tree — local development, CI caches, git bisect — where a longer file is already on disk. Fresh CI runners build from scratch and have nothing to leave a tail, which matches what CI showed for the dependency fix in isolation: clean builds, no corruption.

Verification, x86-64 Linux / LLVM 18

Truncation: padding runtime-packages.lisp with 500 junk bytes and regenerating yields exactly the clean 88383 bytes, where before all 500 survived. Generated script confirmed to carry 8 :supersede and 8 close.

Dependency: the edge now reads generate-lisp-info | boehmprecise/iclasp boehmprecise/lib/libclasp.so; touching the library marks the generator dirty and the files regenerate.

Both: full build clean, regression suite 1963 / TEST_EXIT=0, identical to baseline. CI additionally exercised the truncation change through a complete clasp/ubuntu-latest/bytecode build and baseline regression run.

lib-filename already returns .a under --static-linking, so the single call covers both linkage modes.

[dg1sbg]

Cross-referencing #1828 / #1829, since these two interact and the merge order matters.

#1829 fixes a separate bug: the generated lisp-info depends on libclasp only through an order-only edge, so in practice the generator almost never re-runs — editing Lisp or C++ sources leaves the generated files entirely stale.

That has a direct bearing on this PR:

• Today, the stale-tail bug fixed here is rarely triggered, because the generator seldom runs at all. What you usually get instead is wholly stale generated data (generate-lisp-info never regenerates: its dependency on libclasp reaches it only through an order-only edge #1828).
• Once koga: depend the generated lisp-info on libclasp, not just iclasp #1829 lands, the generator will run whenever libclasp changes — i.e. constantly. Every one of those runs is an opportunity for the :overwrite-without-truncate behaviour fixed here to corrupt a file, whenever the new content is shorter than the old.

So this PR should land before or together with #1829, otherwise fixing the dependency edge will make the truncation bug substantially more frequent rather than less.

Worth noting the two also explain each other's evidence: I originally hit the corruption fixed here while switching branches in a reused tree, and could not reproduce it reliably afterwards — because whether the generator runs at all is itself unreliable, for the reason #1829 fixes.

[dg1sbg]

Note on the CI reds here — every one has been a known pre-existing flake, in three separate tests across two runs, and none is attributable to this change.

Run 1 — cando/ubuntu-latest/native: sole failure WEAK-KEY-AND-VALUE-WEAKNESS, Successes: 1962, 2 aborted compilation units. That is #1814; I posted frequency data there (seven Linux jobs across five unrelated PRs in one day).

Run 2 (after a retrigger) — two reds:

• clasp/ubuntu-latest/bytecode: build and regression suite both passed (Build=success, Successes: 1963, exactly baseline). The failing step was Run ANSI tests: 29 failures with 1 unexpected failure, the unexpected one being PRINT.DOUBLE-FLOAT.4 — prin1 prints some double-floats with too few digits to read back the same float #1816, the double-float print round-trip bug that makes ninja ansi-test exit 1 at random.
• cando/ubuntu-latest/native: WEAK-KEY-AND-VALUE-WEAKNESS again, Successes: 1962.

Meanwhile clasp/ubuntu-latest/native and clasp/macos-latest/bytecode both passed on run 2.

Run 2 is the more useful of the two for reviewing this PR, because clasp/ubuntu-latest/bytecode is the configuration that exercises the generator this change touches, and it shows a clean full build plus a baseline regression suite with the patched generator in place.

For completeness about my own testing: I verified the truncation semantics directly — padding runtime-packages.lisp with 500 junk bytes, regenerating, and confirming it returns to exactly the clean 88383 bytes — but had not run a full build-and-suite with the patched generator locally. CI has now done that.

Not retriggering further: two runs have produced three flake failures across two separate known issues, so another attempt is a coin flip rather than information.