bhyve-scripts

== bhyve-scripts

A collection of shell scripts to run bhyve VMs on my FreeBSD desktop and server machines.

== Environment

• Host Machines
  • FreeBSD 10.2-RELEASE
  • ZFS zvols as the virtual disks for bhyve VMs
  • Bridged tap devices for network access
• Guest Machines
  • Linux Guests: CentOS 6, 7, RHEL 7 and Ubuntu 14.04 LTS
  • BSD Guests: OpenBSD 5.8

== ZFS zvols

These +disk1+ are thin-provisioned zvols.

$ zfs list -r zroot/bhyve
NAME USED AVAIL REFER MOUNTPOINT
zroot/bhyve 107G 364G 144K none
zroot/bhyve/centos6-vertica 49.7G 364G 96K none
zroot/bhyve/centos6-vertica/disk1 49.7G 364G 41.2G -
zroot/bhyve/centos7-docker1 38.2G 364G 144K none
zroot/bhyve/centos7-docker1/disk1 38.2G 364G 32.3G -
zroot/bhyve/openbsd 969M 364G 96K none
zroot/bhyve/openbsd/disk1 969M 364G 969M -
zroot/bhyve/rhel7-ost-icehouse 7.55G 364G 144K none
zroot/bhyve/rhel7-ost-icehouse/disk1 7.55G 364G 8.27G -
zroot/bhyve/rhel7-ost-juno 5.20G 364G 144K none
zroot/bhyve/rhel7-ost-juno/disk1 5.20G 364G 4.76G -
zroot/bhyve/ubuntu-14.04 5.00G 364G 144K none
zroot/bhyve/ubuntu-14.04/disk1 5.00G 364G 5.00G -

$ sudo zfs create zroot/bhyve/centos7-docker1
$ sudo zfs create -V 48G -s zroot/bhyve/centos7-docker1/disk1

== Prerequisites

(TODO) Install grub-bhyve etc.

== Setup

The following script loads the kernel modules for bhyve and creates tap devices for the VMs.

$ sudo ./setup.sh

== Installing Linux -- Booting a VM from installer ISO Image

Edit +device.map+.

.bhyve-scripts/centos7/device.map

(hd0) /dev/zvol/zroot/bhyve/centos7-docker1/disk1
(cd0) /home/tatsuya/installers/centos/CentOS-7.0-1406-x86_64-DVD.iso

Run +grub-bhyve+ as root.

$ cd centos7 $ sudo grub-bhyve -m device.map -r cd0 -M 4096M centos7-docker1 grup> ls (hd0) (hd0,msdos2) (hd0,msdos1) (cd0) (cd0,msdos2) (host) (lvm/centos-root) (lvm/centos-swap) grub> ls (cd0)/ CentOS_BuildTag EFI/ EULA GPL images/ isolinux/ LiveOS/ Packages/ repodata/ RPM-GPG-KEY-CentOS-Testing-7 RPM-GPG-KEY-CentOS-7 TRANS.TBL grup> ls (cd0)/isolinux boot.cat boot.msg grub.conf initrd.img isolinux.bin isolinux.cfg memtest splash.png TRANS.TBL upgrade.img vesamenu.c32 vmlinuz

grub> linux (cd0)/isolinux/vmlinuz
grub> initrd (cd0)/isolinux/initrd.img
grub> boot

NOTE:

For CentOS, it's recommended to perform VNC-based graphical installation rather than console-based installation because graphical one gives you more customization options (e.g. custom disk layout) To do that, add the following options to +linux+ command.

grub> linux (cd0)/isolinux/vmlinuz vnc headless ip=dhcp ksdevice=eth0 lang=en_US keymap=us

Once the installer started, it will give tell you VNC server address and port.

For example,

Please manually connect your vnc client to 10.0.0.117:1 to begin the install.

Run +bhyve+ as root. Note that the VM has the installer DVD ISO attached.

$ sudo bhyve -c 4 -m 4096M -H -P -A -l com1,stdio
-s 0:0,hostbridge -s 1:0,lpc
-s 2:0,virtio-net,tap1
-s 3,ahci-cd,/home/tatsuya/installers/centos/CentOS-7.0-1406-x86_64-DVD.iso
-s 4,virtio-blk,/dev/zvol/zroot/bhyve/centos7-docker1/disk1
centos7-docker1

When installation is completed, press the "Restart" button on the installer screen. This will shutdown the VM and drop you into a shell prompt (instead of restarting). Proceed to next chapter to boot the VM from the disk image.

== Booting a Linux VM from Disk Image

$ cd centos7

Edit +grub.in+. You might want to take a look at grub menu for kernel parameters.

Run the VM.

$ sudo ./run.sh

== Installing OpenBSD -- Booting a VM from installer ISO Image

Edit +device.map+.

.bhyve-scripts/openbsd/device.map

(hd0) /dev/zvol/zroot/bhyve/openbsd/disk1
(cd0) /home/tatsuya/installers/openbsd/openbsd-5.7/install57.iso

Run +grub-bhyve+ as root. Ensure to boot with the OpenBSD install kernel (+bsd.rd+).

$ cd openbsd $ sudo grub-bhyve -m device.map -r cd0 -M 256M openbsd grup> ls (hd0) (cd0) (host) grub> ls (cd0)/ 5.7/ etc/ TRANS.TBL grup> ls (cd0)/5.7 amd64/ TRANS.TBL grub> ls (cd0)/5.7/amd64/ base57.tgz boot.catalog bsd bsd.mp bsd.rd cdboot cdbr comp57.tgz game57.tgz INSTALL.amd64 man57.tgz SHA256 TRANS.TBL xbase57.tgz xfont57.tgz xserv57.tgz xshare57.tgz

grub> kopenbsd -h com0 (cd0)/5.7/amd64/bsd.rd
grub> boot

NOTE: https://forums.freebsd.org/threads/howto-bhyve-using-openbsd-as-main-firewall-in-freebsd.50470/#post-282880[Example]

Run +bhyve+ as root. Note that the VM has the installer ISO attached. Also +-W+ option seems required. It forces virtio to use single-vector MSI.

$ sudo bhyve -W -m 256M -H -P -A -l com1,stdio
-s 0:0,hostbridge
-s 1:0,lpc
-s 2:0,virtio-net,tap4
-s 3,ahci-cd,/home/tatsuya/installers/openbsd/openbsd-5.7/install57.iso
-s 4,virtio-blk,/dev/zvol/zroot/bhyve/openbsd/disk1
openbsd

=== IMPORTANT: Adjust the System Clock

Login to the VM and ensure the system clock is correct. By some reason it's usually advance 3 days, so ntpd won't be able to adjust the clock. Run +ntpdate+ to fix it.2

[docker1] /home/tatsuya% sudo systemctl stop ntpd.service
[docker1] /home/tatsuya% sudo ntpdate -b pool.ntp.org
16 Mar 13:46:27 ntpdate[2046]: step time server 202.112.29.82 offset -277199.180289 sec
[docker1] /home/tatsuya% sudo systemctl start ntpd.service

NOTE: This might be fixed in 10.2-RELEASE (https://www.quernus.co.uk/2015/07/27/openbsd-as-freebsd-router/#comment-2163062052[Reference])

== When Linux Kernel in a VM is Upgraded

Edit +grub.in+ so that it will boot from the latest Linux Kernel.

.bhyve-scripts/centos7/grub.in

linux (hd0,msdos1)/vmlinuz-3.10.0-123.20.1.el7.x86_64 root=/dev/mapper/centos-root
initrd (hd0,msdos1)/initramfs-3.10.0-123.20.1.el7.x86_64.img
boot

== License

bhyve-script is open-sourced under the MIT license. See the link:LICENSE[LICENSE] file for details.