Network migration for bhyve guests using vm-bhyve

gistfile1.md

Please note this is not live migration, as that would require support in bhyve itself, which is probably still a fair way off. This powers the guest off on one host and starts it on the other, the basic process is as follows:

• Receiver is put into receive mode
• Sender takes a snapshot of running guest and sends it in full
• Sender shuts down guest
• Sender takes a second snapshot and sends incremental stream
• Receiver powers up guest (see note 1)

Note that I haven't been able to actually try this with 2 hosts. So far I've just tried sending a guest back to localhost. This is very rough and there's no guarantee it will work correctly, although it shouldn't break anything.

Requirements

• both ends must have vm-bhyve configured and functional. Any virtual switches used by the guest should be configured with the same name at both ends, and if using UEFI, the firmware needs to be installed on both hosts.
• vm-bhyve must be the latest version from GitHub as of 3/8/2016 (build 102019 or higher in vm version)
• both ends must be using ZFS storage. The sending side just needs to be sending from a ZFS datastore. The receiver can currently only receive to the default datastore, so that must be on ZFS. (see note 2)
• socat is required. I don't know what installed this on my system, and unfortunately didn't realise it's not part of base. This can be installed fairly quickly and easily from ports or pkg.

Process

On the receiving side,

# vm recv guestname

This should output that it is waiting for the snapshot and give a port number. Then on the sending side:

# vm send guestname hostname:port

Note 1

In build 102021 or newer you need to specify the -s option on the receiving end to start the guest automatically. The receiving end can't tell if the guest was running originally or not, and after creating the option to specify whether the guest should start or not, it made most sense to not start by default.

# vm recv -s guestname

Note 2

In build 102021 on newer it's possible to specify a receiving datasore using the -d option. If not specified, the guest will be created on the default datastore.

# vm recv -d mydatastore guestname

As with all code that uses getopt, which vm-bhyve uses for parsing all cmd line options, any options can be grouped:

# vm recv -sd mydatastore guestname

Heads up, if you were using this, March 12, 2021 it all changed, for the better of course. Still not really documented, but see "churchers/vm-bhyve@0d5905b" for details