Kexec into an in-memory emergency system
LGPL-3.0 License
Okay, your system is completely broken, and you need to umount /
or something like that.
What do you do?
One of our servers had a broken root filesystem (btrfs, don't judge me).
Online recovery was not possible, so the filesystem needed to be unmounted which is not possible for the root fs.
Additionally, as errors were detected, the kernel decided to mount it read only and didn't let me remount it as rw
.
IPMI? Yes, I had the password in my password store but not the username.
So the only logical solution was to kexec into an emergency system.
This code is what I used.
It recovers all IP addresses as well as SSH host and user keys from the old system and kexecs into a new one - entirely in-memory.
The emergency
script (found in the repository root) will SSH over and execute the following things:
.tar.xz
with a small nix store and a kexec
script) from the files in this repository locally on the machine you're executing this code on
configuration.nix
kexec
-related features are imported from kexec.nix
kexec
script (see below)mkdir
/nix
and /tmp
. If the don't already exist and your root fs is read-only, you have a problem this project can't fixtmpfs
on /tmp
because there might not be one alreadyscp
the emergency image over and extract it/nix
using overlayfs
The kexec
script (found in kexec.nix
) will do the following:
kexec
into the kernel from the emergency image while using the new initrdThe script that is packed into the initrd of the new system will do the following:
networking.localCommands
so the interfaces are availableIf you set the environment variable EMERGENCY_DUMP_NETWORK
to 1
, all IPs, routes, and nameservers will be placed in the emergency_ips
, emergency_routes
, and emergency_nameservers
files, respectively.
$ ./emergency root@somehost
# or
$ ./emergency somebody@somehost
If it doesn't work for you, I'm sorry. I can probably not help you, but if you're able to fix something, feel free to create a PR.
The code is based on clever's kexec nix-test (found here).
The code is licensed under the LGPL3.