I put a NVMe in my desktop and now my server won't reboot

After my VPS at OVH went up in flames, I switched it to GNU Guix. That VPS had lived through many hours of trial and errors, misconfigurations, reconfigurations, globally installed hand-built software, etc. Despite having kept careful notes, rebuilding it from scratch was just not a option.

At that point I had already installed GNU Guix as a package manager on my desktop, and the switch to it as the whole OS on my server was the promise that as long as I kept a copy of my operating-sytem configuration (a single Scheme file), and a backup of my data, I could be up and running again in minutes should my system elect to upload itself to an actual cloud again.

I devised a small configuration that I called ovh-minimal, that started this way:

...
(file-systems
 (cons (file-system
         (mount-point "/")
         (device "/dev/sda2"))))
...

OVH provided at the time a small DOS partition to put your bootloader in, and then the main partition, so / had to be on /dev/sda2.

A few years later, I started selling GNU Guix VPSes and when installing one for a new client I discovered that OVH had foregone the first partition, and now / ought to be on /dev/sda1.

I couldn't just switch to "/dev/sda1" in my configuration, because I still needed to support my and my early client's old configuration. But one good side of having your configuration language be an actual programming language (instead of YAML or JSON or ini or conf or god forbid XML) is that I was able to work my way around the problem quite easily:

(file-systems
 (cons (file-system
         (mount-point "/")
         (device
          (if (access? "/dev/sda2" F_OK)
              ;; As of 2022-08-28 OVH vps don't have a bios boot partition anymore
              ;; but support is needed for VPSs created before this date
              "/dev/sda2" "/dev/sda1"))

The configuration now looks for / on /dev/sda2 only if /dev/sda2 exists.

This worked for many months. Months during which I switched from guix system reconfigure to guix deploy.

guix system reconfigure computes the operating sytem from the declaration on the VPS.
guix deploy computes the operating system from the declaration on my much more powerful desktop machine, and then sends the diff to the VPS: faster, easier, more reliable (the VPS often falls short, RAM-wise).

Months later, I installed a NVMe on my desktop machine.

Can you see where this is going ?

Months later again, I rebooted my VPS, but it did not come back online. I logged into the VNC and saw the dreaded guile prompt that GNU Guix throws you into when it can't boot. The error message was something about the operating system missing from /dev/sda1.

One good thing about GNU Guix, is that I just had to reboot and select an old version of the system at the GRUB prompt in order to boot again (Guix lets you roll-back to any previous generation of your system, unless you explicitly garbage collect it). But the version I had to switch back to was quite old. In fact it was the version before I installed a NVMe on my desktop.

When I rebooted, I diffed the working system configuration with the latest one, saw no relevant difference, but noticed, I don't remember exactly how, that / was a mount of /dev/sda2. 2, not 1. So why was the system trying to boot from /dev/sda1 ?

Then it all clicked.

I had a NVMe on my desktop, so now, on my desktop there was no longer any /dev/sda* but /dev/nvme0n*. As I was using guix deploy, the access? in the scheme code was running on my desktop, and not on the server, and so it selected the wrong branch and configured the server to boot from /dev/sda1, which on my ancient server was the DOS boot partition, and not the main Linux partition.

And thus: my server won't reboot because I installed a NVMe on my desktop.