Rescue me always ::

My arch install broke because of a module rename. All I would see was something like

Failed to open lvm
(initramfs) Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

See journal for logs - yada yada

Since root was unable to be mounted, no journal persisted. Hence, I cannot see the journal logs, can I ?

Anyway, the fix was simple. Get a rescue boot media like grml and regenerate initramfs using lvm2 instead of sd-lvm2.

Now, I was able to get grml on a pen drive to work on this machine but I’m not always this lucky (all my usb ports are bricked and only work when the shrodinger’s cat meows).
Also, I’m too lazy to plug a pen drive next time. Since I just used grml via a pendrive, can I have it boot from the hard disk too ?

grml’s website says: no

Is it possible to install Grml to harddisk?
No. If you want to install a Debian system take a look at grml-debootstrap (or use the Debian Installer instead).

Well, not every no is a no.

grml uses grub as bootloader with the following entry:

cfgboot/grub/grmlsmallamd64_default.cfg

menuentry "grml-small-amd64 2025.08" {
    set gfxpayload=keep
    echo 'Loading kernel...'
    linux   /boot/grmlsmallamd64/vmlinuz apm=power-off boot=live live-media-path=/live/grml-small-amd64/ bootid=grmlsmallamd64202508 "${loopback}" ${kernelopts} nomce 
    echo 'Loading initrd...'
    initrd  /boot/grmlsmallamd64/initrd.img
}

It sets kernel parameters to enable live boot via boot=live but no such kernel parameter exists.
It turns out, init programs inside the initramfs can look for these parms and set the system for bootup.

One such program is live-boot by debian.

live-boot is a collection of scripts providing hooks for the initramfs-tools, used to generate an initramfs capable of booting live systems, such as those created by live-build. This includes the live system ISOs, netboot tarballs, and USB stick images.

Arch docs also mention how to have grml directly on my esp (/boot).

Anyway, I don’t have enough space on the esp to store entire grml (~500M) and it doesn’t mention how can I point the live-media-path to a different partition. So, lets see how live-boot works.

live-boot⌗

live-boot project on commit ff8867c does the following:

Try 60 times, 1 per second, to find a live-bootable file system

sh9990-main.sh

# Scan local devices for the image
i=0
while [ "$i" -lt 60 ]
do
    livefs_root=$(find_livefs ${i})

    if [ -n "${livefs_root}" ]
    then
        break
    fi

    sleep 1
    i=$((i + 1))
done

yes, you’ve 60 seconds to make your livefs available

Unless LIVE_MEDIA is specified on command line, check removable then non removable devices

sh9990-misc-helpers.sh

find_livefs ()
{   ...
    devices_to_scan="$(removable_dev 'sys') $(non_removable_dev 'sys')"
    ...
    for dev in $(subdevices "${sysblock}")
    do
        if check_dev "${dev}"
        then
            return 0
        fi
    done
...
}

Mount whatever is mountable (also tries lvm, iso etc) and check inside

sh9990-misc-helpers.sh

check_dev ()
{ ...
    mount -t ${fstype} -o ro,noatime "${devname}" ${mountpoint} || continue
    ...
    if is_live_path ${mountpoint} && \
        ([ "${skip_uuid_check}" ] || matches_uuid ${mountpoint})
    then
        echo ${mountpoint}
        return 0
    else
        umount ${mountpoint} 2>/dev/null
    fi
   ...
}

If a file with .squashfs or other fs extension exist inside a directory called live/ (LIVE_MEDIA_PATH="live" in 0001-init-vars.sh) boot from here.

bash9990-misc-helpers.sh

is_live_path()
{
    DIRECTORY="${1}/${LIVE_MEDIA_PATH}"
    for FILESYSTEM in squashfs ext2 ext3 ext4 xfs dir jffs
    do
        if ls "${DIRECTORY}/"*.${FILESYSTEM} > /dev/null 2>&1
        then
            return 0
        fi
    done
    return 1
}

And finally mount the live filesystem’s root to /root via setup_unionfs.

sh9990-main.sh

    if [ -n "${MODULETORAMFILE}" ] || [ -n "${PLAIN_ROOT}" ]
    then
        setup_unionfs "${livefs_root}" "${rootmnt?}"
    else
        mac="$(get_mac)"
        mac="$(echo "${mac}" | sed 's/-//g')"
        mount_images_in_directory "${livefs_root}" "${rootmnt}" "${mac}"
    fi

and btw, if you’re wondering (like I did) and greping the entire codebase (like I did) to find out where $rootmnt is set, you won’t find it. It is set in initramfs’s init script:

shinit

export rootmnt=/root

So, whenever an initramfs with live-boot loads, it checks everything it possibly can, starting with removable media, to find a /live directory and mounts file the ending with squashfs (or similar) to /root.

grml explicitly set their live-media-path=/live/grml-small-amd64/ so it doesn’t look for live/ but looks for this path instead.

The picture is clear, I just need to:

Boot a linux kernel with boot=live live-media-path=/live/grml-small-amd64/ parameters
Have an initramfs with live-boot package
Copy /live/grml-small-amd64/ from grml’s iso image to any of my partitions.

Encryption ?⌗

Warning: dead-end

live-boot seems to have some luks code as well, would it be able to detect a /live dir in my luks -> lvm -> xfs setup ?

At the top of live-boot script, it tries to decrypt partitions.

sh9990-main.sh

	if [ -x /scripts/local-top/cryptroot ]
	then
		/scripts/local-top/cryptroot
	fi

Let’s ensure if luks tools are present in grml’s initramfs. First extract it and then grep for luks

; cpio -idmv < boot/grmlsmallamd64/initrd.img
; grep -r 'luks'

Nothing. The directory structure also doesn’t make sense. Where’s the rest of the directories ?

; tree -L 6 .
.
└── usr
    └── lib
        └── modules
            └── 6.12.41+deb13-amd64
                └── kernel
                    ├── arch
                    ├── crypto
                    ├── drivers
                    ├── fs
                    ├── lib
                    ├── net
                    └── sound

13 directories, 0 files

Only kernel modules are here. It first I thought, maybe the cpio was not extracted properly. After several tries I observed compressed cpio size > extracted !? It doesn’t make sense

; file boot/grmlsmallamd64/initrd.img
boot/grmlsmallamd64/initrd.img: ASCII cpio archive (SVR4 with no CRC)

; cpio -idmv < boot/grmlsmallamd64/initrd.img
; du -sh .
32M     .

; ls -lh boot/grmlsmallamd64/initrd.img
-rw-r--r-- 1 neo staff 48M Aug 15 23:06 boot/grmlsmallamd64/initrd.img

Something must be wrong with the img file, look deeper:

; binwalk -e boot/grmlsmallamd64/initrd.img

-------------------------------------------------------------------------------------------------------------------------
DECIMAL                            HEXADECIMAL                        DESCRIPTION
-------------------------------------------------------------------------------------------------------------------------
0                                  0x0                                CPIO ASCII archive, file count: 1296
30874112                           0x1D71A00                          XZ compressed data, total size: 18760580 bytes
-------------------------------------------------------------------------------------------------------------------------
[+] Extraction of cpio data at offset 0x0 completed successfully
[+] Extraction of xz data at offset 0x1D71A00 completed successfully
-------------------------------------------------------------------------------------------------------------------------

Analyzed 1 file for 85 file signatures (187 magic patterns) in 1.5 seconds

So, there’s 2 concatenated archives present inside the same file, and kernel even extracts it properly !

There’s cpio inside the xz too:

; cd extractions/initrd.img.extracted/
; file 1D71A00/decompressed.bin
1D71A00/decompressed.bin: ASCII cpio archive (SVR4 with no CRC)

Anyway, I found my missing files

; cpio -idmv < 1D71A00/decompressed.bin
; ls
0  1D71A00  bin  conf  cryptroot  etc  init  lib  lib64  run  sbin  scripts  usr

Although, files in the initramfs I get dropped in after failed boot of grml (with only initramfs and no live-media-path) vs what I see here are different. Idk how.
In the failed boot terminal, there’s no /scripts/local-top/cryptroot but, in the extraction, there is. Maybe I’ll look at this in the future.

So, encryption -> no.

Simpleton way: extra partition⌗

Let’s be a simple human being and just get a normal partition and have the /live/ directory there.

I’ll have to resize a vol, create a new partition and create a filesystem on it.
I’ve created one /dev/sdb4 and formatted it with ext4.
Then,

Copy the /live directory from the iso to the new partition

[root@Journey hritik]# mount /dev/sdb4 /mnt
[root@Journey hritik]# cp -r live /mnt/.
[root@Journey hritik]# ls /mnt/
live  lost+found
[root@Journey hritik]# ls /mnt/live/
grml-small-amd64

Copy the kernel and initramfs to the esp (/boot)

[root@Journey hritik]# cp boot/grmlsmallamd64/vmlinuz /boot/vmlinuz-rescue
[root@Journey hritik]# cp boot/grmlsmallamd64/initrd.img /boot/initrd-rescue.img

Finally, create a bootloader entry for grml kernel and initramfs. I use systemd-boot, so, here’s how it looks:

title Rescue (grml)
linux /vmlinuz-rescue
initrd /initrd-rescue.img
options apm=power-off boot=live live-media-path=/live/grml-small-amd64/ nomce keyboard=dvorak grml2ram ssh=rescue

I’ve used a few extra parameters documented in the grml cheatcodes.

We’ve sucessfully installed grml on a hard disk, although this is non persistent (can be made persistent). So, exactly like a live boot but from a hard disk partition.

Notes⌗

Boot from grml in bootloader entry but with a pen drive plugged in⌗

This shows a warning that bootid.txt is present but not specified in the kernel parameters – indicating that it picked up the /live directory from the pendrive (as it’s a removeable media and was scanned first).

Although, after a successfull boot into grml, lsblk shows the internal disk partition (sdb4) is mounted for live media. I’m not sure when this pivot happened.

Does a lookup for `/live` across all devices make it possible to hijack live-boot ?⌗

Certainly.

Rescue me always

live-boot⌗

Encryption ?⌗

Simpleton way: extra partition⌗

Notes⌗

Boot from grml in bootloader entry but with a pen drive plugged in⌗

Does a lookup for /live across all devices make it possible to hijack live-boot ?⌗

Does a lookup for `/live` across all devices make it possible to hijack live-boot ?⌗