ci: Run on Ubuntu 26.04 - #1577
Draft
adombeck wants to merge 9 commits into
Draft
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1577 +/- ##
==========================================
- Coverage 87.33% 87.28% -0.06%
==========================================
Files 124 121 -3
Lines 8909 8933 +24
Branches 111 114 +3
==========================================
+ Hits 7781 7797 +16
- Misses 1072 1079 +7
- Partials 56 57 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
adombeck
force-pushed
the
UDENG-10655-run-on-2604
branch
3 times, most recently
from
July 1, 2026 17:25
e316740 to
6e67e57
Compare
adombeck
added
the
e2e-tests
adombeck
force-pushed
the
UDENG-10655-run-on-2604
branch
5 times, most recently
from
July 6, 2026 17:06
3112d11 to
ade3ce9
Compare
denisonbarbosa
requested changes
Jul 6, 2026
denisonbarbosa
left a comment
denisonbarbosa
left a comment
Member
There was a problem hiding this comment.
It looks like the e2e tests did not run here, even though we added the tag. Can you double check what happened? Even if it's something that we didn't break in this PR, I think it's worth fixing it here to check if the e2e tests run fine on the new GitHub runner.
Contributor
Author
huh that's indeed strange. I'll investigate |
adombeck
marked this pull request as draft
July 6, 2026 22:03
adombeck
force-pushed
the
UDENG-10655-run-on-2604
branch
6 times, most recently
from
July 7, 2026 19:42
e1f4ec9 to
413cbb5
Compare
adombeck
force-pushed
the
UDENG-10655-run-on-2604
branch
from
July 8, 2026 17:44
2cdc2d5 to
1bab292
Compare
The Ubuntu 26.04 image is currently available in public preview. That means it can be used by specifically requesting it but the "ubuntu-latest" label still resolves to ubuntu-24.04. Using the new image will allow us to simplify a few things in our e2e-tests and also help me with #1529, so I'd like to make the switch now.
The go-sanity job was failing with
Error getting pkg-config variable "dynamic_uid_min": exit status 1
Go tests failed in CI with:
Error Trace: /home/runner/work/authd/authd/pam/integration-tests/modulehelpers_test.go:111
/opt/hostedtoolcache/go/1.25.11/x64/src/testing/testing.go:1308
/opt/hostedtoolcache/go/1.25.11/x64/src/testing/testing.go:1572
/opt/hostedtoolcache/go/1.25.11/x64/src/testing/testing.go:1898
Error: Received unexpected error:
rename /tmp/TestCLIChangeAuthTok1077107872/002/pam_authd_exectestclichangeauthtok.so-module.gcda /home/runner/work/authd/authd/raw-coverage/TestCLIChangeAuthTok.gcov/pam_authd_exectestclichangeauthtok.so-module.gcda: invalid cross-device link
Test: TestCLIChangeAuthTok
Messages: Teardown: Can't move coverage report data for c Library: rename /tmp/TestCLIChangeAuthTok1077107872/002/
The fix is to move the coverage data from a directory below /tmp to
another directory below /tmp, avoiding crossing filesystem boundaries.
In contrast to the project directory, the directory below /tmp is not
bind-mounted into the LXD containers, so we need to create it there
first for the tests running in LXD containers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sudo-rs (the default sudo on Ubuntu 26.04+) does not support -E. The flag was never necessary: none of the provisioning scripts set LIBVIRT_DEFAULT_URI, and virsh defaults to qemu:///system when run as root anyway. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
On Ubuntu 26.04+ runners systemd enables tmp.mount, which mounts /tmp as a tmpfs capped at 50% of RAM (typically ~7-8 GiB on GitHub-hosted runners). The e2e jobs write the VM qcow2 image and multiple libvirt external memory snapshot files (.mem, one per virsh snapshot-create-as call while the VM is running) into ARTIFACTS_DIR. Each .mem file can reach several GiB, and together with the qcow2 the total easily exceeds the tmpfs limit, producing: Unable to write ...e2e-runner-...-stable-installed.mem: No space left on device /var/tmp is on the real SSD rather than tmpfs, and unlike RUNNER_TEMP (which resolves to /home/runner/work/_temp) it is world-accessible (mode 1777). RUNNER_TEMP lives under /home/runner which has mode 700, causing virsh to fail when the libvirt-qemu user tries to open the qcow2 image: Cannot access storage file '...qcow2' (as uid:64055, gid:991): Permission denied Fix by setting ARTIFACTS_DIR and OUTPUT_DIR to /var/tmp paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When systemd-journal-remote is given a directory as --output, it names the journal files after the connection source. For vsock connections this produces names like remote-vsock:3:3094900709.journal — colons that GitHub Actions artifact upload rejects because they are invalid on NTFS. Pass a specific output file path (as the TCP path already does) so the journal is written to <vm-name>.journal, which contains no colons. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The "Provision authd and the broker" CI step fails on Ubuntu 26.04 with: socat E connect(5, AF=40 cid:4294967295 port:22, 16): No such device CID 4294967295 is VMADDR_CID_ANY, which libvirt reports for a shut-off VM. The VM ends up shut off because force_create_snapshot falls back to a disk-only snapshot when `virsh domstate` does not return "running", and reverting to a disk-only snapshot leaves the VM shut off. The VM is shutting down at snapshot time because cloud-init re-runs its power_state: poweroff directive on the second boot. cloud-init is an installed system package whose systemd services start on every boot. Detaching the NoCloud ISO only removes the datasource for the first boot; subsequent boots use the config cached in /var/lib/cloud/. Modules with frequency "always", such as power_state_change, re-run on every boot using that cached config. On Ubuntu 26.04 the cloud-final stage completes at ~19-20s into the boot — right when force_create_snapshot checks virsh domstate. Fix: once cloud-init has finished its initial setup (the VM is shut off after the first boot), disable it via /etc/cloud/cloud-init.disabled so it never re-runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…vert When virsh snapshot-revert reverts to a disk-only snapshot, the VM is left shut off (no memory state to resume from). On boot, cloud-init may still run its power_state module and shut the VM down again. provision-authd.sh: pre-create /etc/cloud/cloud-init.disabled before the first boot. If cloud-init still powers off the VM (e.g. because ext4 journal replay on the mid-run snapshot backing file undoes the guestfish write), wait for a clean shutdown, run guestfish again on the now-consistent filesystem, and reboot. The second guestfish pass is reliable because cloud-init has completed and flushed all writes. libprovision.sh: using guestfish on a qcow2 snapshot overlay (with a backing chain) fails on Ubuntu 26.04: libguestfs: error: appliance closed the connection unexpectedly. libguestfs: error: /usr/bin/qemu-system-x86_64 exited with error status 1. Instead, start the VM normally and treat a sync_time failure with the VM no longer running as a reliable signal that cloud-init ran its power_state module. Wait for a clean shutdown and reboot — cloud-init does not re-run after completing its first run (semaphore files remain on disk), so the second boot is stable. If sync_time fails while the VM is still running, propagate the error as a genuine failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
adombeck
force-pushed
the
UDENG-10655-run-on-2604
branch
from
July 14, 2026 08:37
1bab292 to
e794062
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Ubuntu 26.04 image is currently available in public preview. That means it can be used by specifically requesting it but the "ubuntu-latest" label still resolves to ubuntu-24.04.
Using the new image will allow us to simplify a few things in our e2e-tests and also help me with #1529, so I'd like to make the switch now.
UDENG-10655