nomad task driver that uses firecracker to start micro-vms
APACHE-2.0 License
nomad task driver for creating Firecracker micro-vms.
Note: The last version of firecracker that works with this nomad plugin is : 0.25.2, more work is needed to make it work with latest releases.
Install(and compile) the firecracker-task-driver binary and put it in plugin_dir and then add a plugin "firecracker-task-driver" {}
line in your nomad config file.
go get github.com/cneira/firecracker-task-driver
cp $GOPATH/bin/firecracker-task-driver YOURPLUGINDIR
Then in your nomad config file, set
plugin "firecracker-task-driver" {}
In developer/test mode(nomad agent -dev
) , plugin_dir is unset it seems, so you will need to mkdir plugins and then copy the firecracker-task-driver binary to plugins and add a plugins_dir = "path/to/plugins"
to the above config file.
then you can run it like:
nomad agent -dev -config nomad.config
For more details see the nomad docs.
{
"name": "default",
"cniVersion": "0.4.0",
"plugins": [
{
"type": "ptp",
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.127.0/24",
"resolvConf": "/etc/resolv.conf"
}
},
{
"type": "firewall"
},
{
"type": "tc-redirect-tap"
}
]
}
Example : exposing port 27960 on micro-vm
{
"name": "microvms2",
"cniVersion": "0.4.0",
"plugins": [
{
"type": "ptp",
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "192.168.127.0/24",
"resolvConf": "/etc/resolv.conf"
}
},
{
"type": "firewall"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"runTimeConfig": {
"portMappings":
[ { "hostPort": 27960, "containerPort": 27960, "protocol": "udp" }
] }
},
{
"type": "tc-redirect-tap"
}
]
}
In this example with outside world connectivity for your vms. The name of this network is default and this name is the parameter used in Network on the task driver job spec. Also the filename must match the name of the network, and the suffix .conflist.
We need to an ext4 root filesystem to use as disk and an uncompressed vmlinux image, the process on how to generate them is described here.
Leveraging ZFS zvols to expose rootfs to firecracker is really simple, and zfs has a lot of benefits.
First download a template image, for example from OpenVZCentos7
Now create a ZVOL to host this tarball
$ zfs create -V 1G zpool/centos7vm
$ mkfs.ext4 /dev/zvol/zpool/centos7vm
$ mount -t ext4 /dev/zvol/zpool/centos7vm /mnt
$ tar xfvz centos-7-x86_64-minimal.tar.gz -C /mnt
$ zfs snapshot zpool/centos7vm@final
Now just use your new zvol as your BootDisk For example:
job "example3" {
datacenters = ["dc1"]
type = "service"
group "test" {
restart {
attempts = 0
mode = "fail"
}
task "test01" {
driver = "firecracker-task-driver"
config {
Vcpus = 1
Mem = 128
KernelImage= "/home/cneira/kernel-images/vmlinux.bin"
BootDisk = "/dev/zvol/vms/centos7vm"
Network = "default"
}
}
}
}
When the microvm starts a file will be created in /tmp/ with the following name -, for example : /tmp/test01-785f9472-52a7-3dbf-8305-d482b1f7dc6f will contain the following info :
{
"AllocId": "590983f4-499a-380f-420e-e5be4d5f46d9",
"Ip": "192.168.127.62/24",
"Serial": "/dev/pts/3",
"Pid": "237216",
"Vnic": "veth05fb4547vm"
}
Don't specifying KernelImage and BootDisk it will default to rootfs.ext4 and vmlinux in the allocation directory.
job "example" {
datacenters = ["dc1"]
type = "service"
group "test" {
restart {
attempts = 0
mode = "fail"
}
task "test01" {
artifact {
source = "https://firecracker-kernels.s3-sa-east-1.amazonaws.com/vmlinux-5.4.0-rc5.tar.gz"
destination = "."
}
artifact {
source = "https://firecracker-rootfs.s3-sa-east-1.amazonaws.com/ubuntu16.04.rootfs.tar.gz"
destination = "."
}
driver = "firecracker-task-driver"
config {
Vcpus = 1
Mem = 128
Network = "default"
}
}
}
}
job "cni-network-configuration-example" {
datacenters = ["dc1"]
type = "service"
group "test" {
restart {
attempts = 0
mode = "fail"
}
task "test01" {
driver = "firecracker-task-driver"
config {
KernelImage = "/home/build/firecracker/hello-vmlinux.bin"
Firecracker = "/home/build/firecracker/firecracker"
Vcpus = 1
Mem = 128
BootDisk = "/home/build/firecracker/hello-rootfs.ext4"
Network = "fcnet"
}
}
}
}
job "neverwinter" {
datacenters = ["dc1"]
type = "service"
task "nwn-server" {
driver = "firecracker-task-driver"
config {
Vcpus = 1
KernelImage = "/home/cneira/Development/vmlinuxs/vmlinux"
BootDisk= "/home/cneira/Development/rootfs/ubuntu/18.04/nwnrootfs.ext4"
Disks = [ "/home/cneira/Development/disks/disk0.ext4:rw" ]
Mem = 1000
Network = "default"
}
}
}
The firecracker-task-driver exposes the serial console as this option is handy to troubleshoot network issues. Each microvm generates a state file on the /tmp/ directory, named using the job name + allocation id. For example:
-rw-r--r--. 1 root root 152 May 12 14:07 /tmp/test01-590983f4-499a-380f-420e-e5be4d5f46d9
The contents of the state file should be like the following:
{
"AllocId": "590983f4-499a-380f-420e-e5be4d5f46d9",
"Ip": "192.168.127.62/24",
"Serial": "/dev/pts/3",
"Pid": "237216",
"Vnic": "veth05fb4547vm"
}
Using the serial now we know which serial port is expose and it's a matter of connect to it. You could use SCREEN(1) to connect to the serial console.
$ sudo screen /dev/pts/3
Started Update UTMP about System Runlevel Changes.
CentOS Linux 7 (Core)
Kernel 4.14.225 on an x86_64
192 login:
It's also possible to support the project on Patreon
I work on this project on my free time and my country is not on the list available for github sponsors so any help for me continue working on this is appreciated.