I have a beefy Linux box that I use for development. One of the things this box does is run KVM instances that run a Triton headnode and several compute nodes.
I have a single gigabit link to the outside world. All networks are able to reach the outside world via NAT.
[root@beast mgerdts]# virsh net-list
Name State Autostart Persistent
--------------------------------------------------
coaladmin1 active yes yes
coalexternal1 active yes yes
default active yes yes
I obtained the configuration for each of the coal networks with virst net-dumpxml <name>.
<network connections='4'>
<name>coaladmin1</name>
<uuid>85816e6e-3b51-4375-a317-b19c28fb6f50</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr1' stp='on' delay='0'/>
<mac address='52:54:00:b9:d7:86'/>
<domain name='coaladmin1'/>
<ip address='10.99.99.1' netmask='255.255.255.0'>
</ip>
</network><network connections='4'>
<name>coalexternal1</name>
<uuid>db5f922d-f782-44ee-a380-5cf5e22c1d74</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr2' stp='on' delay='0'/>
<mac address='52:54:00:0d:9a:e8'/>
<domain name='coalexternal1'/>
<ip address='10.88.88.1' netmask='255.255.255.0'>
</ip>
</network>I'm using ZFS for storage, putting an entire Triton data center under one dataset. The idea is that if I'm experimenting, it becomes easy to snapshot an entire datacenter with one zfs snapshot -r. What I show below is not the result of careful planning, just what I did on the fly.
[root@beast mgerdts]# virsh vol-list data
Name Path
-------------------------------------------------------------
vms/coal-1/boot-usb /dev/zvol/data/vms/coal-1/boot-usb
vms/coal-1/cn1-disk0 /dev/zvol/data/vms/coal-1/cn1-disk0
vms/coal-1/cn2-disk0 /dev/zvol/data/vms/coal-1/cn2-disk0
vms/coal-1/cn3-disk0 /dev/zvol/data/vms/coal-1/cn3-disk0
vms/coal-1/disk0 /dev/zvol/data/vms/coal-1/disk0
vms/fedora30 /dev/zvol/data/vms/fedora30
[root@beast mgerdts]# virsh vol-dumpxml --pool data vms/coal-1/cn1-disk0
<volume type='block'>
<name>vms/coal-1/cn1-disk0</name>
<key>data/vms/coal-1/cn1-disk0</key>
<source>
</source>
<capacity unit='bytes'>21474836480</capacity>
<allocation unit='bytes'>0</allocation>
<target>
<path>/dev/zvol/data/vms/coal-1/cn1-disk0</path>
</target>
</volume>
To ensure that I don't accidentally blow away my headnode disks, I created a snapshot of each of then and then used zfs hold to add a hold.
In order for the Triton setup program to recognize the USB image, I had to configure it (using the virtmanager gui, surely there's another way) to be a USB drive with the removable box ticked. Even though the USB image is on a pool made up of decent SSDs (mirrored), the performance of this virtual USB mimics the perfomance of a cheap thumb drive (single digit MB/s).
After the HN is set up, you can switch the drive over to a non-removable SATA drive. In virtmanager, I had to halt the VM, remove the USB disk, then add it back as a SATA disk to get past an error saying that "removable" (which was unchecked) is only applicable to USB. The zvol was not destroyed, but that may have been a result of the aforementioned zfs hold. You have been warned.
After making this change, you will probably need to update the boot order so that the newly added SATA disk is first in the boot order.
Note: When I did this procedure, I had not yet switched the boot disk from USB to SATA.
The 4 GiB USB image has room for about 3 platform images. After bumping against this limit a couple times, I decided to grow this disk. The procedure went roughly like:
root@beast# zfs snapshot data/vms/coal-1/boot-usb@before-grow
root@beast# zfs set volsize=16g data/vms/coal-1/boot-usb
Halt the HN VM, then boot it. A reboot within the guest is not sufficient to get it to see the new size.
Save a copy of the usbkey contents. Be sure to use . rather than * so that dot files are not missed.
[root@headnode (coal-1) ~]# sdc-usbkey mount
/mnt/usbkey
[root@headnode (coal-1) ~]# cd /mnt/usbkey
[root@headnode (coal-1) /mnt/usbkey]# tar cvzf /var/tmp/mg-usbkey.tar.gz .
Compressing '/var/tmp/mg-usbkey.tar.gz' with '/usr/bin/gzip'...
...
[root@headnode (coal-1) /mnt/usbkey]# cd
[root@headnode (coal-1) ~]# sdc-usbkey unmount
Now we are ready to grow the partition and put a new file system on the partition.
[root@headnode (coal-1) ~]# format -e
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <QEMU-QEMU HARDDISK-2.5+-16.00GB> reserved
/pci@0,0/pci1af4,1100@6,7/storage@4/disk@0,0
1. c2t1d0 <QEMU-HARDDISK-2.5+-60.00GB>
/pci@0,0/pci1af4,1100@9/disk@1,0
Specify disk (enter its number): 0
selecting c1t0d0: reserved
[disk formatted]
Note: capacity in disk label is smaller than the real disk capacity.
Select <partition> <expand> to adjust the label capacity.
...
format> pa
PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
7 - change `7' partition
8 - change '8' partition
9 - change `9' partition
expand - expand label to use whole disk
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
!<cmd> - execute <cmd>, then return
quit
partition> expand
The expanded capacity is added to the unallocated space.
partition> pr
Volume: reserved
Current partition table (original):
Total disk sectors available: 33537981 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 system wm 2048 256.00MB 526335
1 boot wm 526336 1.00MB 528383
2 root wm 528384 3.46GB 7794687
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
7 unassigned wm 0 0 0
8 reserved wm 33538015 8.00MB 33554398
partition> 2
Part Tag Flag First Sector Size Last Sector
2 root wm 528384 3.46GB 7794687
Enter partition id tag[root]:
Enter partition permission flags[wm]:
Enter new starting Sector[528384]:
Enter partition size[7266304b, 7794687e, 3548mb, 3gb, 0tb]: 33538014e
partition> pr
Volume: reserved
Current partition table (unnamed):
Total disk sectors available: 33537981 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 system wm 2048 256.00MB 526335
1 boot wm 526336 1.00MB 528383
2 root wm 528384 15.74GB 33538014
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
7 unassigned wm 0 0 0
8 reserved wm 33538015 8.00MB 33554398
partition> la
[0] SMI Label
[1] EFI Label
Specify Label type[1]:
Ready to label disk, continue? y
partition> ^D
The values I used for size were taken from the final pr output above.
[root@headnode (coal-1) ~]# mkfs -F pcfs -o b=SMARTSOBOOT,nofdisk,size=$((33538014-528384+1)),fat=32 /dev/rdsk/c1t0d0s2
Construct a new FAT file system on /dev/rdsk/c1t0d0s2: (y/n)? y
Now the content needs to be restored to the usb disk.
[root@headnode (coal-1) ~]# mount -F pcfs /dev/dsk/c1t0d0s2 /mnt/usbkey
[root@headnode (coal-1) ~]# cd /mnt/usbkey/
[root@headnode (coal-1) /mnt/usbkey]# tar xvzf /var/tmp/mg-usbkey.tar.gz
Decompressing '/var/tmp/mg-usbkey.tar.gz' with '/usr/bin/gzcat'...
...
[root@headnode (coal-1) /mnt/usbkey]# cd
[root@headnode (coal-1) ~]# df -h /mnt/usbkey
Filesystem Size Used Available Capacity Mounted on
/dev/dsk/c1t0d0s2 15.7G 3.36G 12.3G 22% /mnt/usbkey
[root@headnode (coal-1) ~]# umount /mnt/usbkey
I expect at this point that the boot loader configuration on the usbdisk is busted, so I updated it by assigning a newer PI to the HN.
[root@headnode (coal-1) ~]# sdcadm platform list
VERSION CURRENT_PLATFORM BOOT_PLATFORM LATEST DEFAULT
20191017T032813Z 0 1 true false
20191017T000357Z 0 1 false true
20191014T204909Z 3 1 false false
20191009T235451Z 1 1 false false
20151126T062538Z 0 0 false false
[root@headnode (coal-1) ~]# sdcadm platform assign 20191017T032813Z headnode
updating headnode 55657228-6967-0142-a965-6b7be4e6e9b6 to 20191017T032813Z
Setting boot params for 55657228-6967-0142-a965-6b7be4e6e9b6
Updating booter cache for servers
Done updating booter caches
Verifying boot_platform updates
Cleaned up deprecated 'latest' symlink
[root@headnode (coal-1) ~]# reboot