Live boot ISO support

[Kekun]

Bring forward the appropriate bits (mostly initramfs and some systemd units from eos-boot-helper) so that we can boot Endless’s ISO images (one of the possible outputs of the image builder) into a live EOS session.

Trying to scope this. The ISO image builds already, and boots. It fails due to eos-image-boot-setup.service not running. This is included in the initramfs, but something must be going wrong.

Tested various ISO images built from eos-build-meta commit 0271426 with eos-image-builder commit 7e79bd2feb1d9775182d02f06ec0f98a6cd41698.

In all cases, it fails in the initramfs with errors awaiting two disk units One dev-disk-endless\x2dimage3.device, and one dev-disk-by\x2duuid-2025....device.

After 1m30s it drops to an error prompt which fails to login as root due to the root account being locked:

In an image built from 'main', I found no way to trigger the GRUB menu. It seems the menu timeout is 0 and no amount of hammering ESC could get me a menu.

I built a modified OSTree with the following change to files/grub-config/grub.cfg:

diff --git a/files/grub-config/grub.cfg b/files/grub-config/grub.cfg
index 0e263407..23f0c3e8 100644
--- a/files/grub-config/grub.cfg
+++ b/files/grub-config/grub.cfg
@@ -147,9 +147,7 @@ else
 fi
 
 # Show menu if we need to offer Windows, or if the last boot failed.
-if test ( "$found_windows" -o "$recordfail" ) ; then
-    set timeout=30
-fi
+set timeout=30
 
 # Increment loglevel if the last boot failed.
 if test "$recordfail" ; then

This allows me to see the final GRUB commandline:

The endless.image.device=UUID=2025-10-21... parameter is suspicious.

The ISO file doesn't have that UUID anywhere in the metadata I can see:

> isoinfo -d -i eos-master-amd64-amd64.251021-155213.base.iso 
CD-ROM is in ISO 9660 format
System id: 
Volume id: Endless-OS-Nightly-base
Volume set id: 
Publisher id: ENDLESS OS FOUNDATION LLC
Data preparer id: XORRISO-1.5.4 2021.01.30.150001, LIBISOBURN-1.5.4, LIBISOFS-1.5.4, LIBBURN-1.5.4
...

In the source grub.cfg the following fragment exists:

    set root=(loop_img,gpt3)
    probe -u $eosimage --set=image_dev_uuid
    set kparams="endless.image.device=UUID=$image_dev_uuid endless.image.path=$image_path nohibernate"

There are various bits of code in https://github.com/endlessm/eos-boot-helper which consume the endless.image.device parameter, including eos-esp-generator and the dracut/image-boot module. I don't know how this works yet.

I haven't yet managed to get a shell in the initramfs, due to root account being locked. Tomorrow I will try if running passwd in the initramfs build element causes dracut to not lock the root account. Then I should be able to use rd.shell and rd.break to debug (currently these are useless as they can't open an actual shell anyway)

[dsd]

blkid can be used to test the codepaths used to create the links like /dev/disk/by-uuid.

Testing latest eos6 release iso:

# blkid eos-eos6.0-amd64-amd64.250610-092703.base.iso 
eos-eos6.0-amd64-amd64.250610-092703.base.iso: BLOCK_SIZE="2048" UUID="2025-06-10-10-24-44-00" LABEL="Endless-OS-6-0-7-base" TYPE="iso9660" PTTYPE="dos"
# losetup -f  --show -P eos-eos6.0-amd64-amd64.250610-092703.base.iso 
/dev/loop0
# blkid /dev/loop0*
/dev/loop0: BLOCK_SIZE="2048" UUID="2025-06-10-10-24-44-00" LABEL="Endless-OS-6-0-7-base" TYPE="iso9660" PTTYPE="dos"
/dev/loop0p1: BLOCK_SIZE="2048" UUID="2025-06-10-10-24-44-00" LABEL="Endless-OS-6-0-7-base" TYPE="iso9660" PTTYPE="dos"
/dev/loop0p2: SEC_TYPE="msdos" UUID="7ECB-FCC4" BLOCK_SIZE="512" TYPE="vfat"

[dsd]

In an image built from 'main', I found no way to trigger the GRUB menu. It seems the menu timeout is 0 and no amount of hammering ESC could get me a menu.

Checking the same eos6 iso, I confirm that it seems not to work in a UEFI VM with GNOME Boxes. I'm pretty sure it works on real hardware, may be a VM issue.

It does work in GNOME Boxes booting the eos6 iso in bios mode and holding Shift. And then rd.break does work there (just have to press Enter when prompted)

Thanks! Confirmed the ISO does have the UUID correct:

~/eos7/import-vm: blkid eos-master-amd64-amd64.251022-100053.base.iso
eos-master-amd64-amd64.251022-100053.base.iso: BLOCK_SIZE="2048" UUID="2025-10-22-10-06-17-00" LABEL="Endless-OS-Nightly-base" TYPE="iso9660" PTTYPE="dos"
~/eos7/import-vm: sudo losetup -f --show -P eos-master-amd64-amd64.251022-100053.base.iso
/dev/loop0
~/eos7/import-vm: sudo blkid /dev/loop0*
../../../../dev/loop0: BLOCK_SIZE="2048" UUID="2025-10-22-10-06-17-00" LABEL="Endless-OS-Nightly-base" TYPE="iso9660" PTTYPE="dos"
../../../../dev/loop0p1: BLOCK_SIZE="2048" UUID="2025-10-22-10-06-17-00" LABEL="Endless-OS-Nightly-base" TYPE="iso9660" PTTYPE="dos"
../../../../dev/loop0p2: SEC_TYPE="msdos" UUID="C996-D8B8" BLOCK_SIZE="512" TYPE="vfat"

Also managed to log in as root at the error prompt; which required the following patch to dracut:

diff --git a/modules.d/68systemd-sysusers/module-setup.sh b/modules.d/68systemd-sysusers/module-setup.sh
index 477b8e7..ca70040 100755
--- a/modules.d/68systemd-sysusers/module-setup.sh
+++ b/modules.d/68systemd-sysusers/module-setup.sh
@@ -28,4 +28,7 @@ install() {
     # set read and write permission for the current user
     [[ -f "$initdir/etc/gshadow" ]] && chmod u+rw "$initdir/etc/gshadow"
     [[ -f "$initdir/etc/shadow" ]] && chmod u+rw "$initdir/etc/shadow"
+
+    # TEMP: Set a root password
+    systemd-firstboot --root="$initdir" --force --root-password "root"
 }

Once able to debug, it's clear that kernel module loading isn't working -- the initramfs is missing libkmod.so which systemd & udev need to dlopen(). We use dracut 108, which decides what libraries to include based on special ELF metadata that is systemd provides since v257. We also downgraded to systemd v256 due to the PAYG patches.

One would think the above issue would also break the regular disk images, but those seem to work fine either way. Not sure why.

Fixed the initramfs by rebasing the Endless patches on systemd 257.10.

It boots much further but still doesn't reach GDM / Gnome Shell.

Various errors in the journal that show issues.

Missing /etc/machine-id

systemd warns that the machine cannot boot:

Oct 22 13:21:10 eos systemd[1]: Detected first boot.
Oct 22 13:21:10 eos systemd[1]: System cannot boot: Missing /etc/machine-id and /etc is mounted read-only.
Oct 22 13:21:10 eos systemd[1]: Booting up is supported only when:
Oct 22 13:21:10 eos systemd[1]: 1) /etc/machine-id exists and is populated.
Oct 22 13:21:10 eos systemd[1]: 2) /etc/machine-id exists and is empty.
Oct 22 13:21:10 eos systemd[1]: 3) /etc/machine-id is missing and /etc is writable.

Boot continues anyway.

NetworkManager later complains:

Oct 22 13:21:29 eos NetworkManager[770]: <error> [1761139289.2116] /etc/machine-id: no valid machine-id. Use fake one based on secret-key:>

We discussed this previously and opened #95. I will do that.

eos-esp-generator fails

The systemd-gpt-auto-generator probably should fail, as eos-esp-generator apparently does the same thing.

I suspect eos-esp-generator should not fail, though -- maybe missing some label on the ESP volume?

Oct 22 13:21:10 eos systemd-gpt-auto-generator[380]: EFI loader partition unknown, exiting.
Oct 22 13:21:10 eos systemd-gpt-auto-generator[380]: (The boot loader did not set EFI variable LoaderDevicePartUUID.)
Oct 22 13:21:10 eos kernel: Guest personality initialized and is inactive
Oct 22 13:21:10 eos kernel: VMCI host device registered (name=vmci, major=10, minor=122)
Oct 22 13:21:10 eos kernel: Initialized host personality
Oct 22 13:21:10 eos kernel: NET: Registered PF_VSOCK protocol family
Oct 22 13:21:10 eos kernel: zram: Added device: zram0
Oct 22 13:21:10 eos eos-esp-generator[369]: Generator failed:
                                            Traceback (most recent call last):
                                              File "/usr/lib/systemd/system-generators/eos-esp-generator", line 983, in <module>
                                                main()
                                              File "/usr/lib/systemd/system-generators/eos-esp-generator", line 969, in main
                                                esp_mount = generator.get_esp_mount()
                                                            ^^^^^^^^^^^^^^^^^^^^^^^^^
                                              File "/usr/lib/systemd/system-generators/eos-esp-generator", line 446, in get_esp_mount
                                                esp_part = self._get_esp_part()
                                                           ^^^^^^^^^^^^^^^^^^^^
                                              File "/usr/lib/systemd/system-generators/eos-esp-generator", line 598, in _get_esp_part
                                                esp_disk_scheme = esp_disk_part['PART_ENTRY_SCHEME']
                                                                  ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
                                            KeyError: 'PART_ENTRY_SCHEME'
Oct 22 13:21:10 eos (sd-exec-[368]: /usr/lib/systemd/system-generators/eos-esp-generator failed with exit status 1.

/var/tmp missing or read-only

Various services fail due to missing /var/tmp. Not sure if this is due to missing /etc/machine-id or something else...

Oct 22 13:21:10 eos systemd-tmpfiles[482]: All rules containing unresolvable specifiers will be skipped.
Oct 22 13:21:10 eos (resolved)[481]: Failed to create destination mount point node '/run/systemd/mount-rootfs/var/tmp', ignoring: Read-onl>
Oct 22 13:21:10 eos (resolved)[481]: Failed to mount /run/systemd/unit-private-tmp/var-tmp to /run/systemd/mount-rootfs/var/tmp: No such f>
Oct 22 13:21:10 eos (resolved)[481]: systemd-resolved.service: Failed to set up mount namespacing: /var/tmp: No such file or directory
Oct 22 13:21:10 eos (resolved)[481]: systemd-resolved.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-resolved: No suc>
Oct 22 13:21:10 eos audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=systemd-resolved comm="sy>
Oct 22 13:21:10 eos systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE
Oct 22 13:21:10 eos systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Oct 22 13:21:10 eos systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution.

Oct 22 13:21:10 eos (emd-oomd)[527]: Failed to create destination mount point node '/run/systemd/mount-rootfs/var/tmp', ignoring: Read-onl>
Oct 22 13:21:10 eos kernel: zram0: detected capacity change from 0 to 2017280
Oct 22 13:21:10 eos (emd-oomd)[527]: Failed to mount /run/systemd/unit-private-tmp/var-tmp to /run/systemd/mount-rootfs/var/tmp: No such f>
Oct 22 13:21:10 eos (emd-oomd)[527]: systemd-oomd.service: Failed to set up mount namespacing: /var/tmp: No such file or directory
Oct 22 13:21:10 eos (emd-oomd)[527]: systemd-oomd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-oomd: No such file o>
Oct 22 13:21:10 eos systemd[1]: systemd-oomd.service: Main process exited, code=exited, status=226/NAMESPACE
Oct 22 13:21:10 eos systemd[1]: systemd-oomd.service: Failed with result 'exit-code'.
Oct 22 13:21:10 eos systemd[1]: Failed to start systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer.

zram fails

zram again!

Oct 22 13:21:28 eos zram-generator[730]: Error: Device or resource busy (os error 16)
Oct 22 13:21:28 eos systemd[1]: systemd-zram-setup@zram0.service: Failed with result 'exit-code'.
Oct 22 13:21:28 eos systemd[1]: Stopped systemd-zram-setup@zram0.service - Create swap on /dev/zram0.
Oct 22 13:21:28 eos systemd[1]: eos-enable-zram.service: Failed with result 'exit-code'.
Oct 22 13:21:28 eos systemd[1]: Failed to start eos-enable-zram.service - swap with zram.
Oct 22 13:21:28 eos (udev-worker)[742]: zram0: /usr/lib/udev/rules.d/50-disk-bfq.rules:1 Failed to write ATTR{/sys/devices/virtual/block/z>

eos-transient-setup fails

Something is trying to write to /usr/local, which seems very bogus. Possibly missing some environment variable?

Oct 22 13:21:29 eos eos-transient-setup[692]: /usr/sbin/eos-transient-setup:86   write_dconf_compile    INFO: Installing new DConf profile>
Oct 22 13:21:29 eos eos-transient-setup[692]: Traceback (most recent call last):
Oct 22 13:21:29 eos eos-transient-setup[692]:   File "/usr/sbin/eos-transient-setup", line 174, in <module>
Oct 22 13:21:29 eos eos-transient-setup[692]:     main()
Oct 22 13:21:29 eos eos-transient-setup[692]:   File "/usr/sbin/eos-transient-setup", line 167, in main
Oct 22 13:21:29 eos eos-transient-setup[692]:     setup.write_dconf_compile()
Oct 22 13:21:29 eos eos-transient-setup[692]:   File "/usr/sbin/eos-transient-setup", line 87, in write_dconf_compile
Oct 22 13:21:29 eos eos-transient-setup[692]:     os.makedirs(os.path.dirname(USER_PROFILE_PATH), exist_ok=True)
Oct 22 13:21:29 eos eos-transient-setup[692]:   File "<frozen os>", line 215, in makedirs
Oct 22 13:21:29 eos eos-transient-setup[692]:   File "<frozen os>", line 215, in makedirs
Oct 22 13:21:29 eos eos-transient-setup[692]:   File "<frozen os>", line 215, in makedirs
Oct 22 13:21:29 eos eos-transient-setup[692]:   File "<frozen os>", line 225, in makedirs
Oct 22 13:21:29 eos eos-transient-setup[692]: OSError: [Errno 30] Read-only file system: '/usr/local'

GDM segfaults

Not clear what happens here, but the PAM failure looks interesting.

Oct 22 13:29:30 eos systemd[1]: Starting gdm.service - GNOME Display Manager...
Oct 22 13:29:30 eos systemd[1]: Started gdm.service - GNOME Display Manager.
Oct 22 13:29:30 eos gdm-launch-environment][2647]: pam_systemd(gdm-launch-environment:session): Failed to get user record: Input/output error
Oct 22 13:29:30 eos gdm-launch-environment][2647]: pam_unix(gdm-launch-environment:session): session opened for user gnome-initial-setup(uid=979) by (uid=0)
Oct 22 13:29:30 eos /usr/libexec/gdm-wayland-session[2655]: dbus-daemon[2655]: [session uid=979 pid=2655] Activating service name='org.freedesktop.systemd1' requested by ':1.0' (uid=979 pid=2652 comm="/usr/libexec/gdm-wayland-session dbus-run-session " label="unconfined")
Oct 22 13:29:30 eos /usr/libexec/gdm-wayland-session[2655]: dbus-daemon[2655]: [session uid=979 pid=2655] Activated service 'org.freedesktop.systemd1' failed: Process org.freedesktop.systemd1 exited with status 1
Oct 22 13:29:30 eos /usr/libexec/gdm-wayland-session[2652]: Unable to register display with display manager
Oct 22 13:29:30 eos gdm-launch-environment][2647]: pam_unix(gdm-launch-environment:session): session closed for user gnome-initial-setup
Oct 22 13:29:30 eos gdm[2642]: Gdm: GdmDisplay: Session never registered, failing
Oct 22 13:29:30 eos gdm[2642]: Gdm: Child process -2652 was already dead.
Oct 22 13:29:40 eos gdm-launch-environment][2668]: pam_systemd(gdm-launch-environment:session): Failed to get user record: Input/output error
Oct 22 13:29:40 eos gdm-launch-environment][2668]: pam_unix(gdm-launch-environment:session): session opened for user gnome-initial-setup(uid=979) by (uid=0)
Oct 22 13:29:40 eos gdm[2642]: Gdm: GdmSession: no session desktop files installed, aborting...
Oct 22 13:29:40 eos systemd[1]: gdm.service: Main process exited, code=dumped, status=5/TRAP
Oct 22 13:29:40 eos systemd[1]: gdm.service: Killing process 2668 (gdm-session-wor) with signal SIGKILL.
Oct 22 13:29:40 eos systemd[1]: gdm.service: Killing process 2671 (gdbus) with signal SIGKILL.
Oct 22 13:29:40 eos systemd[1]: gdm.service: Failed with result 'core-dump'.
Oct 22 13:29:40 eos systemd[1]: gdm.service: Triggering OnFailure= dependencies.

The core dump isn't saved.

Fixes/issues for these.

1. /etc/machine-id: create an empty /etc/machine-id, as planned in Create an empty /etc/machine-id file and avoid passing systemd.firstboot=no on kernel commandline #95. Fix tested, PR in Create an empty /etc/machine-id file #185
2. eos-esp-generator: eos-esp-generator fails in ISO image #183
3. /var/tmp: The error from systemd-resolved appears to be due to missing /var/tmp. However, /dev/loop1p3 is mounted at /var/ and contains /tmp. There is a unit eos-live-boot-overlayfs-setup.service which sets up a rw overlayfs on /var/tmp. That happens Before local-fs.target, but systemd-resolved doesn't declare any dependency on that target so is probably completing earlier. More investigation needed. systemd-resolved and systemd-oomd fail in the live ISO #187
4. zram: It looks like systemd-zram-setup@zram0.service already set up zram, causing eos-enable-zram to fail with "write error: Device or resource busy". We can remove the latter service in Remove eos-enable-zram service eos-boot-helper#426.
5. eos-transient-setup: eos6 has a symlink /usr/local -> /var/local, which would make it writable. PR in Add a symlink /usr/local -> /var/local #186 (requires testing)
6. GDM crash. Appears to be solved by adding /etc/machine-id.