Network migration for bhyve guests using vm-bhyve

gistfile1.md

Please note this is not live migration, as that would require support in bhyve itself, which is probably still a fair way off. This powers the guest off on one host and starts it on the other, the basic process is as follows:

• Receiver is put into receive mode
• Sender takes a snapshot of running guest and sends it in full
• Sender shuts down guest
• Sender takes a second snapshot and sends incremental stream
• Receiver powers up guest (see note 1)

Note that I haven't been able to actually try this with 2 hosts. So far I've just tried sending a guest back to localhost. This is very rough and there's no guarantee it will work correctly, although it shouldn't break anything.

Requirements

• both ends must have vm-bhyve configured and functional. Any virtual switches used by the guest should be configured with the same name at both ends, and if using UEFI, the firmware needs to be installed on both hosts.
• vm-bhyve must be the latest version from GitHub as of 3/8/2016 (build 102019 or higher in vm version)
• both ends must be using ZFS storage. The sending side just needs to be sending from a ZFS datastore. The receiver can currently only receive to the default datastore, so that must be on ZFS. (see note 2)
• socat is required. I don't know what installed this on my system, and unfortunately didn't realise it's not part of base. This can be installed fairly quickly and easily from ports or pkg.

Process

On the receiving side,

# vm recv guestname

This should output that it is waiting for the snapshot and give a port number. Then on the sending side:

# vm send guestname hostname:port

Note 1

In build 102021 or newer you need to specify the -s option on the receiving end to start the guest automatically. The receiving end can't tell if the guest was running originally or not, and after creating the option to specify whether the guest should start or not, it made most sense to not start by default.

# vm recv -s guestname

Note 2

In build 102021 on newer it's possible to specify a receiving datasore using the -d option. If not specified, the guest will be created on the default datastore.

# vm recv -d mydatastore guestname

As with all code that uses getopt, which vm-bhyve uses for parsing all cmd line options, any options can be grouped:

# vm recv -sd mydatastore guestname

Found bug
This is vm list on sending host

NAME            DATASTORE       LOADER      CPU    MEMORY    VNC                  AUTOSTART    STATE
mail02          default         bhyveload   4      16G       -                    No           Stopped
testing_centos7 default         grub        4      16G       -                    No           Stopped
webhost         default         grub        8      16G       -                    Yes [1]      Running (1570)

vm recv host:

root@vm02:~ # vm recv testing_centos7
/usr/local/sbin/vm: ERROR: invalid virtual machine name - 'testing_centos7'

Name too long?

Humm, changed to cento7, now get this
Sending centos7 to vm02

* stage 1: taking snapshot - 20160808151925
* stage 1: sending zroot/vm/centos7@20160808151925
! error detected while sending snapshot

More digging... If you don't have socat on the recv side, that is the error.

With socat on recv side, linux moved just fine. and ran first try!

Any reason that this isn't at all documented in the man pages? It works fine, or so far it has. Did a "man vm" and was surprised that it wasn't there. If you just forgot to add it in, this is a reminder, but if you left it out on purpose, that's fine too.

This isn't working for me. Machine A:

# hostname
nh19.XXX.XXX
# socat -V
socat by Gerhard Rieger and contributors - see www.dest-unreach.org
socat version 1.7.3.4 on Jun 12 2020 02:39:53
   running on FreeBSD version FreeBSD 12.1-RELEASE-p6 GENERIC, release 12.1-RELEASE-p6, machine amd64
...
# vm version
vm-bhyve: Bhyve virtual machine management v1.4.2 (rev. 104002)
# vm recv dev-unh                                                                    
Recieving guest into /zroot/vm/dev-unh
  * stage 1: waiting for snapshot on port 12000
  ! error detected while recieving snapshot

Machine B:

# socat -V
socat by Gerhard Rieger and contributors - see www.dest-unreach.org
socat version 1.7.3.4 on Jun 12 2020 02:39:53
   running on FreeBSD version FreeBSD 12.1-RELEASE-p3 GENERIC, release 12.1-RELEASE-p3, machine amd64
...
# vm version
vm-bhyve: Bhyve virtual machine management v1.4.2 (rev. 104002)
# vm list | grep dev-unh
dev-unh    default    grub       4    8G      -    No         Stopped
# vm send dev-unh nh19.XXX.XXX:12000
Sending dev-unh to nh19.XXX.XXX
  * stage 1: taking snapshot - 20200623131242
  * stage 1: sending zroot/vm/dev-unh@20200623131242
  ! error detected while sending snapshot
# zfs list zroot/vm/dev-unh@20200623132119
NAME                              USED  AVAIL  REFER  MOUNTPOINT
zroot/vm/dev-unh@20200623132119      0      -  4.96G  -

Any other information I can supply?

debug
Set to a value other than [empty]/no/off/false/0 to run vm-bhyve in debug mode.
In this mode, all output from the bhyve process is written to
$vm_dir/{guest}/bhyve.log. This is useful if the guest is crashing or
exiting abnormally as the log will contain any output from bhyve.

Default: no

debug=""

Heads up, if you were using this, March 12, 2021 it all changed, for the better of course. Still not really documented, but see "churchers/vm-bhyve@0d5905b" for details