reproducible=true build is not bit-identical across different host machines

[manfred-kaiser]

Summary

A fully-pinned, reproducible = true build (all *_sha256 pins set, pylock/build_pylock hash-verified) produces different .AppImage bytes when built on two different machines, even though every resolved input is proven byte-identical. Reproducibility does hold machine-to-machine when the machine is held constant (two independent GitHub Actions runs of the same workflow produce an identical hash), so the gap is specifically "same machine class" vs. "different machine class" reproducibility.

Found while packaging ssh-mitm as an AppImage (the flagship real-world user of this feature) - see ssh-mitm/ssh-mitm, commits around 2026-09-02 (search history for "repro investigation" in that repo for the full trail of debug commits/CI runs this issue is based on).

What's proven

• CI-to-CI: two independent ubuntu-latest GitHub Actions runs of the exact same commit/workflow produce a bit-identical .AppImage (same sha256, twice).
• Local-to-local: two independent builds on the same local machine (openSUSE Tumbleweed, btrfs) also produce a bit-identical .AppImage.
• Local vs. CI: the two above do not match each other.

Environment (local vs. CI)

	Local	CI (ubuntu-latest)
OS	openSUSE Tumbleweed	Ubuntu 24.04
Filesystem	btrfs	ext4 (typical)
CPU	Intel i7-7700 (Kaby Lake)	AMD (has sse4a)
CPU flags relevant to zstd	sse2 sse4_1 sse4_2 bmi1 bmi2 avx2	sse2 sse4_1 sse4_2 sse4a bmi1 bmi2 avx2 (identical superset)
Cores	8	4
appimagetool binary	same file, sha256-pinned	same file, sha256-pinned

What's been ruled out (each with a real test, not a guess)

1. Compile-time SIMD dispatch (facebook/zstd#4099/#4230) - the gate that bug touches is #if defined(ZSTD_ARCH_X86_SSE2), resolved once at compile time. Since both machines run the exact same sha256-pinned appimagetool binary, this can't differ between runs. Confirmed irrelevant by reading the zstd source diff directly.
2. Runtime CPU feature dispatch (BMI2/AVX2 runtime CPUID probing, referenced in zstd's own BMI2-dispatch work) - checked /proc/cpuinfo on both machines directly in CI; bmi2/avx2 are present on both.
3. mksquashfs -processors N (thread count) - local has 8 cores, CI has 4. Forced -processors 1 on both (via --mksquashfs-opt); no change to either machine's hash.
4. File/packing order - forced a fully deterministic, LC_ALL=C-sorted -sort priority file (covering all ~11k AppDir entries, tested to actually take effect - it does change the hash) on both machines. Still produced two different (though each individually stable) hashes.
5. Locale (LC_COLLATE) - local shell runs de_DE.UTF-8, CI runs (presumably) C/C.UTF-8. Rebuilding locally with LC_ALL=C produced no change from the non-LC_ALL=C baseline - rules out locale-dependent sorting inside mksquashfs itself (it did explain an artifact in our own comparison tooling's sort calls, see next point, but not the actual mksquashfs output).
6. AppDir content identity - extracted both a local- and a CI-built .AppImage (--appimage-extract) and diffed the full trees (diff -rq): zero differences. Independently re-verified via a full-tree sha256sum manifest, correctly LC_ALL=C-sorted on both sides to avoid the locale-sort artifact above: zero differences, 9109/9109 files matching.
7. Symlink targets - find -type l was invisible to the content-hash manifest (find -type f only). Collected readlink output for all 1049 symlinks on both sides: zero differences.
8. File permission bits - found and fixed a real discrepancy here (many files were -rwxrwxr-x/-rw-rw-r-- locally vs. -rwxr-xr-x/-rw-r--r-- on CI, source not fully identified - umask reads 0022 on both, so it isn't the interactive shell's umask; possibly the python-build-standalone tarball's own stored permission bits combined with a different extraction-time umask context). Rewrote every local file's mode to exactly match CI's manifest (verified 0 diff after). Rebuilding after this fix still produced a different hash from CI. So this is a real bug (arguably appimage.ctl should normalize permission bits the same way it already normalizes mtimes via _normalize_mtimes, since two logically-equal AppDirs can currently have different mode bits depending on the build host's extraction-time umask/context) but it is not sufficient to explain the squashfs-level divergence.
9. zstd version (AppImage/appimagetool's own CI pins alpine:3.19, which ships zstd 1.5.5-r8 - predates the 1.5.7 release containing the #4230 fix) - built mksquashfs 4.6.1 from source, statically linked against a freshly-built zstd 1.5.7, and swapped it into the extracted appimagetool AppDir (usr/bin/mksquashfs, found via execvp("mksquashfs", ...) + AppRun's own PATH-prepending, confirmed in AppImage/appimagetool's source). Built this exact same binary on CI and downloaded it as an artifact; running the CI-built binary against my local AppDir reproduced my own local hash (not CI's) - proving the mksquashfs binary itself isn't the variable, only which AppDir/machine processes it. Building independently on both machines (different host GCC each time - 16.2.0 locally vs. Ubuntu's) still gave two different hashes, but that test is confounded by the compiler difference and isn't conclusive on its own.
10. squashfskit (squashfskit/squashfskit) - the exact fork NixOS/nixpkgs#40144 documents successfully using to fix this same class of bug (their words: "mksquashfs is nondeterministic ... even when passing -no-fragments and -processors 1"), and that reproducible-builds.org's own guidance points to. Built it (commit ae0d656, 2019) from source on both machines, statically linked against the same fresh zstd 1.5.7, swapped into appimagetool. Still produced two different hashes.

What's left unexplained

Content, permissions, symlinks, sort order, and thread count are all proven identical or forced identical between the two builds. The appimagetool binary is proven byte-identical (when using the stock pinned one) or built from the exact same pinned source both times (when using a custom mksquashfs). Every documented, community-recommended fix for this general class of squashfs non-determinism (squashfskit, fresh zstd, -processors 1, -sort) has been tried and none of them close the gap for this specific machine pair.

Remaining untested candidates, roughly in order of plausibility:

• Kernel-version-dependent behavior of some syscall mksquashfs relies on (e.g. getdents64 ordering semantics, readahead/mmap behavior) that isn't controlled by any of the above.
• Filesystem-type-dependent behavior (btrfs locally vs. ext4 on CI) surfacing through something other than plain directory-entry order (already ruled out via -sort).
• glibc-version-dependent codegen/behavior baked into the statically-linked mksquashfs/zstd binary at build time (both the squashfskit and fresh-zstd tests built independently per-machine with each machine's own host GCC/glibc, which is a real confound neither test controlled for).

Practical takeaway for now

Reproducibility does hold for the actual release pipeline (GitHub Actions building on ubuntu-latest, twice, bit-identical) - that's what matters for "can someone else verify this exact release was built from this exact source," which is the realistic threat model most projects care about. What doesn't currently hold is "any two arbitrary machines produce the same bytes," which reproducible = true's docstring currently implies more broadly ("reproducible across machines and over time").

Suggested follow-ups

1. Fix item 8 above regardless of the rest (normalize permission bits in _normalize_mtimes or a sibling function) - it's a real, if insufficient, correctness gap.
2. Soften the reproducible/docs/reproducible-builds.md language to be explicit that the byte-identical guarantee is currently only demonstrated same-machine-class (e.g. "the same CI runner image"), not universally cross-machine, until this is actually root-caused.
3. If anyone wants to pick this up: the most promising unexplored angle is a byte-level diff of the two squashfs images (not their extracted content, which is identical) to see exactly which structure differs - superblock fields, fragment table, block layout - which would narrow down whether this is a metadata field or genuinely different compressed bytes for identical block content.

[manfred-kaiser]

Update: squashfskit tested, and a new, more precise finding

Tried the actual squashfskit fork (the one NixOS/nixpkgs#40144 used successfully, commit ae0d656), statically linked against a freshly-built zstd 1.5.7. Still produced a different hash from CI.

But going deeper (prompted by re-reading AppImageKit#929, the issue this project's own code already cites) turned up something important: SomberNight in that 2018 thread describes the exact same symptom - past some file-count threshold, appimagetool produces "one out of two hashes randomly" on repeated runs of the identical input, on a single machine. That's a different claim than "machine A differs from machine B" - it's "this is non-deterministic per invocation, full stop," which a two-machine comparison can't tell apart from a real cross-machine effect without enough samples.

So: ran the squashfskit build four times in a row, same AppDir, same machine, same everything (-processors 1 set). Got three different hashes in three consecutive runs. Tried again with ASLR disabled (setarch -R) - still different every time. squashfskit is not stable even locally in this setup.

Then ran the stock, hash-pinned appimagetool's own bundled mksquashfs (Alpine 3.19, zstd 1.5.5, squashfs-tools 4.6.1 - the one actually used in the real build/release pipeline) four times in a row, same conditions: identical hash, all four times.

So the instability isn't inherent to squashfs-tools in general - it's specific to the squashfskit build in this setup, most likely because I linked 2019-era squashfskit source against a 2025 zstd (1.5.7) it was never tested against. That combination is itself untested/unvalidated upstream, and pairing old wrapper code with a newer library than it shipped with is a plausible new source of bugs distinct from the original cross-machine question.

Practical conclusion, revised: squashfskit is not a safe drop-in fix here and should not be recommended without further validation against its original contemporary zstd version. The stock, sha256-pinned appimagetool remains the better choice - it's proven deterministic on repeated same-machine runs (both locally and on CI, independently), which is what the actual release pipeline relies on. The original cross-machine (not run-to-run) divergence documented above stands unexplained.

[manfred-kaiser]

Checked squashfs-tools#140 (mkfs-time not honoring SOURCE_DATE_EPOCH pre-4.7) too - not it either. Parsed the actual squashfs superblock bytes directly (magic hsqs + inode_count + mkfs_time fields) out of a real built AppImage: mkfs_time = 0, matching SOURCE_DATE_EPOCH=0. That's correctly honored even in the 4.6.1 we use.

[manfred-kaiser]

Tested another strong candidate found via research: squashfs-tools 4.7 (June 2025) removed the "fragment block stall" - a mechanism added in 4.4 (2019) specifically to force deterministic file ordering, at a real perf cost. The 4.7 changelog describes it as replaced by a genuinely reproducible ordering algorithm rather than the old timing-based stall.

Built mksquashfs 4.7.5 statically (zstd-only, same recipe as appimagetool's own CI), swapped it into the pinned appimagetool the same way as before. This time I shipped the exact same binary to both machines (not rebuilt per-machine) to eliminate any compiler/glibc-at-build-time confound entirely.

Result: CI-internal stability holds (b190202... on two independent CI runs), local-internal stability holds too, but CI != local, same as with the stock tool. So the pre-4.7 stall mechanism isn't the cause either - ruling out fragment ordering/scheduling as a suspect entirely, since 4.7's replacement isn't timing-based at all.

Since this test used the identical binary object code on both machines, this also rules out build-time compiler/glibc differences as a contributor - whatever differs must come from something at run time (kernel, runtime glibc, filesystem, or memory layout) feeding into mksquashfs's processing, not from how the tool itself was built.

(Tested via a personal fork + temp CI workflow, not on this repo, to keep the real CI history clean.)

[manfred-kaiser]

Found a real, concrete structural cause for part of the divergence: did a byte-level superblock comparison of the two squashfs images (not just decompressed-content diff, which is blind to this).

inode_table_start was byte-identical between a local and a CI build, meaning the actual compressed data/fragment blocks matched exactly - but xattr_id_table_start differed: the local build (openSUSE Tumbleweed) had a real xattr table, CI had none (-1). Root cause: openSUSE labels every file with security.selinux="unconfined_u:object_r:user_home_t:s0" at the filesystem level, and mksquashfs faithfully preserves xattrs by default - baking the build host's own SELinux label into the packaged AppImage. Ubuntu's runner has no such xattr, so nothing gets embedded there. This is invisible to a decompressed-content diff since xattrs aren't part of file content.

This is a real bug independent of reproducibility too - a portable AppImage shouldn't carry the build host's security-context labels at all. Confirmed the fix: appimagetool --mksquashfs-opt -no-xattrs removes the xattr table entirely on both sides.

However, testing local-vs-CI again with -no-xattrs on both (same pinned appimagetool, xattr confound eliminated): still not identical. This time inode_table_start itself differs by ~3.2KB between local and CI, meaning the actual compressed data+fragment region has a real size difference now, not just a metadata-table shift. CI-internal stability re-confirmed (two independent workflow runs, identical hash). So there's still a genuine local-vs-CI difference at the compressed-data level once xattrs are out of the picture.

Suggested fix regardless of the rest: appimagectl should pass -no-xattrs (or normalize/strip xattrs during AppDir assembly, same idea as the existing _normalize_mtimes/_normalize_permissions) so the build host's filesystem security labels never leak into the image.

[manfred-kaiser]

Found and confirmed a real, separate bug: same-machine reproducibility (not just cross-machine) breaks whenever the AppDir is freshly reassembled (new inode numbers on btrfs), even with identical content/order/xattrs/thread-count. Root-caused it to mksquashfs's duplicate-detection code (duplicate()/pre_duplicate() in mksquashfs.c), which pre-filters candidate duplicates by comparing already-compressed block sizes before doing a full checksum+byte comparison - any variance there changes which files get deduplicated vs. recompressed, cascading into a very different final image even though the actual file content is 100% identical.

Confirmed with a controlled test: two fully independent AppDir reassemblies on the same machine, identical -no-xattrs -sort <forced-order> -processors 1, gave two different hashes. Adding -no-duplicates on top: two more independent reassemblies now give the exact same hash. This is a genuine, reproducible squashfs-tools finding, worth reporting upstream separately from this issue.

However: -no-duplicates (even combined with -no-fragments, eliminating cross-file interaction entirely) does not close the local-vs-CI gap - tested cross-machine with both flags active, still two different hashes. So there are at least two independent causes: one now identified and fixable (duplicate-detection sensitivity, fixed by -no-duplicates), and one still unexplained that's specifically machine-dependent and survives even with every file getting its own dedicated, independently-compressed block.

Also directly verified in the zstd v1.5.7 source that the BMI2 dispatch hypothesis from earlier in this thread is wrong: ZSTD_encodeSequences_bmi2() and ZSTD_encodeSequences_default() call the identical shared _body function - provably byte-identical output, only different instruction selection for speed. The only other CPU-dependent path (ZSTD_row_matchMaskGroupWidth, gated by #if defined(ZSTD_ARCH_X86_SSE2)) is resolved at compile time, so it can't vary between runs of the same binary. zstd's own maintainer confirms (facebook/zstd#2949): deterministic given same input/params/version, no machine caveat. So whatever's left is not zstd's compression algorithm itself.

Also fully re-verified (fresh, on the current -no-xattrs/-sort build pair, not just the earlier baseline) that decompressed content is still 100% identical: full recursive diff, full sha256 manifest of all 9109 files, symlink targets, permissions - zero differences anywhere. The divergence is purely in how identical bytes get packed, never in what bytes exist.

[manfred-kaiser]

Cross-referencing everything found in this investigation, in case it's useful to anyone tracking related reports:

Directly matching our symptom:

• mksquashfs: fix data race when spilling blocks to disk in duplicate checking plougher/squashfs-tools#362 - confirmed, unreleased (milestone 4.7.6) data race in duplicate-checking when a file is too big for buffer space and blocks get spilled to disk + read back. Very close to what we found by testing -no-duplicates, though its specific trigger (sparse blocks) doesn't apply to our AppDir (checked - no sparse files present).
• Number of symbolic links/uncompressed size reported changes between invocations of mksquashfs on the same tree plougher/squashfs-tools#176 - independent report of file/symlink counts changing between mksquashfs invocations on an unchanged tree. Turned out to be the source tree changing between runs in that case (hardlinks/live backup), not a squashfs-tools bug - but worth linking since the surface symptom matches ours closely.
• FATAL ERROR: BUG in get_virt_disk, 1991996551 not found plougher/squashfs-tools#360 - referenced by #362 as possibly the same underlying race, FATAL ERROR: BUG in get_virt_disk.

Compression-side, checked and ruled out (with reasoning, in case someone else goes down this path):

• [reproducibility] Zstd varies compressed output based on the presence of SSE / NEON in some cases facebook/zstd#4099 - the SSE/NEON compressed-output-varies issue. Verified in the v1.5.7 source that the only BMI2-gated path (ZSTD_encodeSequences_bmi2) is provably byte-identical to its default counterpart (same shared body function), and the only other CPU-gated path (ZSTD_row_matchMaskGroupWidth, #if defined(ZSTD_ARCH_X86_SSE2)) is a compile-time, not runtime, gate.
• [Question] Is zstd compression determinist ? facebook/zstd#2949 - zstd maintainer confirms determinism given same input/params/version, no machine caveat.

Already tried as a fix candidate:

• squashfskit/squashfskit - the fork referenced by Reproducible squashfs NixOS/nixpkgs#40144 and reproducible-builds.org's own guidance. Tested it (built from source, linked against fresh zstd 1.5.7) - introduced its own local run-to-run instability with that zstd version, not a safe fix as tested.

Full writeup of the investigation is in the issue body and the comments above.

[manfred-kaiser]

Full pipeline breakdown, with the call chain and an honest assessment of what's actually been verified this investigation vs. what's assumed/untested:

#	Step	Started by	What happens	Reproducibility
1	Config resolution	appimage.ctl build()	Reads pyproject.toml [tool.appimage], validates pins	Deterministic (pure text processing)
2	Load Python runtime	_prepare_python()	Download/cache, sha256-verified against python_sha256	Byte-identical guaranteed (hash pin)
3	Tarball extraction	_prepare_python() (tarfile)	Extracts the Python tarball into AppDir/python	Deterministic given an identical tarball, but not checked: whether tarfile's extraction carries mtimes/permissions 1:1 from the archive before step 8 normalizes them
4	Install dependencies	pip (subprocess), against pylock.toml	Hash-verified dependency install	Wheel contents byte-identical guaranteed (hash pin) - but not checked: whether .dist-info/RECORD or similar metadata embeds absolute paths that could vary by pip version
5	Build+install the project's own wheel	Build backend (hatchling, via build_pylock)	Builds the project itself into a wheel	Not individually verified - hatchling is generally considered reproducible, but wasn't tested in this investigation
6	Compile bytecode	_compile_pyc() -> compileall (subprocess)	.pyc files, --invalidation-mode unchecked-hash	Hash-based (no mtime embedding), but an untested, open suspicion: co_filename in the bytecode can embed the absolute build path if compileall is invoked with a full path instead of a relative/stripped one - local (/home/user/...) vs. CI (/home/runner/...) are necessarily different paths!
7	Copy assets	_copy_assets()/_copy_extra_files()	Icon, .desktop, AppRun, extra files	Deterministic (plain byte copy)
8	Normalize mtimes	_normalize_mtimes()	All mtimes -> SOURCE_DATE_EPOCH	Verified deterministic
9	Normalize permissions	_normalize_permissions()	Clears group/other-write bits	Verified deterministic (our fix)
10	Resolve appimagetool/runtime	_resolve_appimagetool()/_resolve_runtime_file()	Download/cache, sha256-verified	Byte-identical guaranteed, and directly verified: statically linked, no dynamic system-library dependency
11	Packaging invocation	subprocess.run([appimagetool,...])	Calls appimagetool with generated arguments	Arguments themselves deterministic
12	AppDir validation	appimagetool (own process)	Checks desktop file, AppStream metadata	Deterministic
13	mksquashfs invocation	appimagetool -> execvp (PATH-based, bundled usr/bin)	Assembles generated mksquashfs arguments, invokes it	Not fully audited: whether appimagetool itself adds extra, possibly time-/environment-dependent flags (e.g. -mkfs-time) automatically
14	Directory tree traversal	mksquashfs (internal)	readdir()/stat() over the AppDir	Open gap: the shipped fix does not use -sort - processing order relies on plain readdir(), which is filesystem-dependent. Our -sort test only forced this manually, not on the real pipeline's default path
15	Duplicate detection	mksquashfs (internal), pre_duplicate()/duplicate()	Content-size pre-filter + byte comparison	Now disabled (-no-duplicates) - verified as a cause of non-determinism on AppDir reassembly, same machine
16	Block compression	mksquashfs -> ZSTD_compressCCtx()	Each 128 KB block compressed individually	Code-verified deterministic for identical input (BMI2/SSE2 paths are provably byte-identical or compile-time-fixed) - yet we still observe divergence, so either the actual per-block input isn't identical after all, or there's a still-undiscovered factor
17	Fragment packing	mksquashfs (internal)	Small file tails packed into shared blocks	Not disabled in the shipped fix (only tested manually combined with -no-duplicates, which did not isolate/close the cross-machine gap - but was never tested alone)
18	Write metadata tables	mksquashfs (internal)	Inode/directory/fragment/id tables written compressed	Directly depends on steps 14 + 16
19	Concatenate runtime+squashfs	appimagetool	Appends the squashfs image to the runtime ELF	Deterministic
20	Embed MD5 digest	appimagetool	Patches a 16-byte MD5 into a reserved slot in the runtime	Circularly deterministic - fully depends on steps 16-18
21	zsyncmake invocation	appimagetool -> execvp	Generates the .zsync control file	Not checked at all in this investigation whether zsyncmake itself is deterministic

The two rows I'd check next: #6 (path embedding in .pyc files) and #14 (traversal order without -sort) - both are plausible, not-yet-tested explanations that fit the observed pattern (local != CI, but each internally stable) exactly.

[manfred-kaiser]

Correction to the table above: steps 1-9 (AppDir assembly, including bytecode compilation) are not open/unverified - I mis-stated that. We've already repeatedly diffed the fully decompressed AppDir content between local and CI builds (diff -rq + a full SHA256 manifest across all 9109 files, several times, across several test configurations), and it has always matched exactly, byte for byte. That check covers .pyc file bytes, .dist-info/RECORD, everything steps 1-9 produce - if any of those steps embedded a different absolute path or otherwise produced different bytes on local vs. CI, it would have shown up there, and it never did.

So:

• No module named 'appimage.build' #3 (tarball extraction before normalization) - moot regardless, since normalization runs after and is independently verified.
• Is it possible to use local python instead of downloading? #4 (pip/RECORD embedded paths) - refuted by the content-identical proof.
• Is libfuse3 supported? #5 (hatchling build reproducibility) - refuted by the content-identical proof.
• chore(deps): bump actions/checkout from 4 to 7 #6 (.pyc path embedding) - refuted by the content-identical proof.

All of steps 1-9 should read ✅ verified deterministic (content-identical, confirmed repeatedly).

This actually sharpens the investigation usefully: it rules out the entire AppDir-assembly side of the pipeline. The divergence is fully confined to steps 10 onward - specifically the squashfs packaging (#13-18), since that's the only place left where two provably-identical inputs produce a different result.

[manfred-kaiser]

One more correction: #14 (traversal order) isn't an open candidate either - it was already tested in combination. The -no-duplicates cross-machine test earlier in this thread had an explicit, fully forced -sort (identical file order on both sides) together with -no-duplicates active, and still produced different hashes (CI 693e2c8... vs. local b41010c...). So order was controlled at the same time duplicate-detection was controlled, and the divergence persisted.

With content (1-9), xattrs, duplicate-detection, and processing order all independently controlled and confirmed identical, and it still diverges - the remaining mystery is now narrowed specifically to block compression (#16) or fragment packing (#17) inside mksquashfs itself, with no controllable input-side explanation left standing.

[manfred-kaiser]

Checked the remaining open rows from the table above:

#13 (appimagetool's automatic flags): Read the source directly (AppImage/appimagetool/src/appimagetool.c, sfs_mksquashfs()). The flag list is fully deterministic for fixed inputs: -comp zstd -root-owned -noappend -b 128K, plus -wildcards -ef .appimageignore only if that file exists (relative to cwd, which is always our stable dist_dir). -mkfs-time 0 is only added when SOURCE_DATE_EPOCH is not set in the environment - we always set it, so appimagetool relies on mksquashfs's own env-var handling instead (already verified earlier via direct superblock byte inspection: mkfs_time=0 correct on both sides). Not a source of variance.

#21 (zsyncmake): Not actually a useful check for root-causing this - .zsync is purely derived from the (already-differing) AppImage bytes. Even with a perfectly deterministic zsyncmake, the .zsync would necessarily differ once its source file differs. Downstream symptom, not a possible cause.

#17 (fragment packing): Only ever tested in combination with -no-duplicates (which alone was already insufficient). Never isolated. But since -no-duplicates + -sort + -no-fragments together still diverge, it's not standing alone as an explanation either.

Net result: every step except #16 (the actual ZSTD_compressCCtx() call per block) is now either verified deterministic or ruled out by a combined test. #16 is the one contradiction left: source-code review and the zstd maintainer's own statement both say it should be deterministic for identical input, yet we observe divergence with content, order, xattrs, and duplicate-detection all independently controlled. Continuing with a per-block compressed-size trace (built mksquashfs with -DSQUASHFS_TRACE) on the minimal reproducer to see whether the actual bytes going into a specific block call are identical or not.

[manfred-kaiser]

Important correction to the minimal-reproducer finding above: I found a real bug in my own test setup, not in appimage.ctl or mksquashfs.

The squashfs-repro CI workflow installed pip install "appimage==3.0.0" from PyPI. I just checked that package directly (fresh venv, pip install appimage==3.0.0) - it's the old, unfixed version: no -no-xattrs, no -no-duplicates, plain cmd = [str(appimagetool_bin), "--runtime-file", str(staged_runtime)]. Locally, I was using my manually-patched, reinstalled venv copy with today's fixes baked in. So the earlier "even the fully embedded-toolchain minimal reproducer still diverges" result was comparing a fixed local build against an unfixed CI build - not a valid same-configuration comparison, and doesn't actually demonstrate anything about whether the current fixes hold up cross-machine.

Note this doesn't affect the earlier debug-nodup/debug-nodup-nofrag cross-machine tests in this thread - those constructed the --mksquashfs-opt flags manually in the CI workflow script itself, bypassing appimage.ctl's own command construction entirely, so they weren't exposed to this particular mismatch.

Re-running the minimal reproducer with both sides installing from the exact same git commit instead of PyPI, to get an actually valid comparison.

[manfred-kaiser]

Resolution for the minimal reproducer: with both sides installing the exact same committed wheel (built from commit 3b297ce, sha256 ed2d7bd6ad4e6f171d3e6a32d652eac6be46a174ba684e6406e75bb13f27cbe4, verified matching in the CI log) and the standard, unmodified, sha256-pinned appimagetool (no swap, no custom build), local and CI now produce the identical AppImage:

39493882e093651ad5e5cbe027f51b91ff9249745bc20dc09472952bfb97b90f

So the cross-machine divergence chased through this whole thread was not a squashfs-tools/zstd/kernel-level mystery after all - it was a bug in my own test methodology: the CI workflow for the minimal reproducer was installing appimage==3.0.0 from PyPI (the old, unfixed version - no -no-xattrs, no -no-duplicates at all, verified by installing it fresh into a clean venv), while local was using a manually-patched, reinstalled copy with today's fixes. Once both sides install literally the same wheel, -no-xattrs + -no-duplicates alone is sufficient for full cross-machine reproducibility on this reproducer - no -processors pin, no other workaround needed.

Practical consequence worth flagging: ssh-mitm's real build pipeline (hatch run appimage:build) resolves appimage>=3.0.0 from PyPI too. The fixes in this repo have never actually been exercised by that pipeline on CI, since nothing's been published yet. Worth cutting a new release once this is confirmed solid.

I'll re-verify the full ssh-mitm project (not just the minimal reproducer) with matched code on both sides next, to close this out properly.

[manfred-kaiser]

Root cause and resolution

The actual root cause of the cross-machine divergence chased through this whole thread: the appimage build-tool dependency itself was never pinned. Every downloaded artifact appimage.ctl touches (Python, appimagetool, the runtime stub, all project dependencies) was hash-pinned and verified byte-identical repeatedly - but the tool doing the pinning wasn't pinned itself. Local was running a locally-patched appimage with today's fixes; CI resolved appimage>=3.0.0 fresh from PyPI, getting the old, unfixed release. In hindsight this was foreseeable: a fully reproducible build requires every parameter in the chain to be pinned, and this project's own build-tool version was the one link that wasn't - once that's understood, "local and CI eventually diverge" isn't surprising, it's just a matter of finding which unpinned parameter did it. Confirmed by installing the exact same wheel on both sides: identical hash, no other workaround needed.

Fixed in 3.0.1, now on PyPI. Two real bugs shipped in that release:

1. mksquashfs preserves filesystem xattrs by default - build-host security labels (e.g. SELinux) were leaking into the packaged image. Fixed via -no-xattrs.
2. mksquashfs's duplicate-detection pre-filter (compressed-block-size comparison before a full byte check) made packaging sensitive to incidental per-build state, even from an unchanged, freshly-reassembled AppDir on the same machine. Fixed via -no-duplicates.

Documented both the pinning requirement and the exact effective command line in docs/reproducible-builds.md so this doesn't have to be rediscovered:

appimagetool --runtime-file <staged copy of runtime_sha256's file> \
  --mksquashfs-opt -no-xattrs \
  --mksquashfs-opt -no-duplicates \
  [-u <update_info>] \
  <AppDir> <app>-<arch>.AppImage

which itself invokes (from AppImage/appimagetool's own source, sfs_mksquashfs()):

mksquashfs <AppDir> <destination> \
  -offset <runtime size> -comp zstd -root-owned -noappend -b 128K \
  [-wildcards -ef .appimageignore]   # only if present
  [-mkfs-time 0]                     # only if SOURCE_DATE_EPOCH is unset
  <our --mksquashfs-opt flags>

Verified end-to-end on the real ssh-mitm project: with appimage==3.0.1 pinned by hash (not >=3.0.0), local and CI now produce a byte-identical .AppImage via the actual production workflow, no debug scaffolding involved.

Leaving this issue open for now since the xattr leak is worth reporting to AppImage/appimagetool upstream separately (general ecosystem issue, not appimage.ctl-specific), but the reproducibility question this issue was originally about is resolved.