smartos-live src publish/testing workflow

workflow.md

Workflow

This guide was designed to be the second phase of the development workflow. While the first guide focuses on a quick iterative sandbox where changes can be previewed immediately, this guide focuses on putting those changes into a live image and testing the final implementation.

Goals

The goals of this workflow are to allow the following:

1. Development occurs directly on the developers workstation, in his or her native editor and toolchain.
2. Incremental builds can generate an official boot image.
3. A SmartOS node can be booted into the newly created boot image.
4. While changes are not immediately reflected, code changes can be previewed in under 10 minutes on a clean, isolated node.

Overview

End Result

After the initial setup, the workflow will flow between three different contexts: the desktop workstation, a SmartOS build zone, and an isolated SmartOS node where changes are previewed on boot.

Desktop

Your desktop is where the development occurs. This workflow allows edits to be made directly on the source that can be committed and pushed directly. The main advantage here is that a native editor/environment can be used and ultimately there is no source that needs to be synced back onto the repository before the code can be pushed.

The desktop will need to act as an nfs server, and export the smartos-live/src directory. This directory will later be mounted inside of the SmartOS build zone.

VMWare

Vmware will be leveraged to create 2 networks: An external NAT or Bridged interface that will provide external access to the internet and a custom private network that will be used to PXE boot the test node into the preview image.

Vmware allows custom networks by default, but we will need to shutoff the dhcp server for this network and adjust some ip ranges and netmasks.

This setup can work on virtual box, however the VirtualBox performance sucks compared to VMWare and ultimately you can't test kvm brands on VirtualBox so you're on your own there.

SmartOS build zone

A dedicated build zone is the easiest way to create an environment where the code can be built. Due to the way the project is setup, a full build is required once. After that point, incremental builds will be sufficient to generate a bootable image.

For the build, we'll actually follow the default setup instructions from the wiki with one slight exception: we'll mount in the smartos-live project with the nfs export from the desktop, and then rsync the code into our build directory. This allows changes on the desktop workstation to be included in the build.

The build zone will double as a PXE boot server as well, serving the incremental build output boot images directly.

Setup

VMware

Following this guide we will need to adjust the custom vmware network so we can use it as the admin network and provide dhcp/tftp services. Using sudo, modify /Library/Preferences/VMware\ Fusion/networking in an editor of your choice and ensure the following values are set:

answer VNET_1_DHCP no
answer VNET_1_HOSTONLY_NETMASK 255.255.255.0
answer VNET_1_HOSTONLY_SUBNET 192.168.200.0

And now reload the vmware networking configuration with the following commands:

sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --configure
sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --stop
sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-cli --start

Workstation

The workstation needs to export the smartos-live directory via an nfs share. I use a mac, so the steps go like this:

$ echo '"/Users/tylerflint/Work/clones/smartos-live" -network 172.16.89.0 -mask 255.255.255.0 -mapall=501:20' >> /etc/exports
$ sudo nfsd update

NOTE: The location of your local code path will differ, and the vmware network may be different as well.

Build Zone Hypervisor

The hypervisor setup that will host the build zone is a nearly identical setup to this guide. The only difference is the network setup. For this workflow we will have an external bridged or NATed interface that will provide this machine with internet access, and an admin interface on the custom vmware network that will allow us to provide dhcp/tftp services to boot a test node with our custom builds. To to this, you'll need to shut down the vm before it boots as vmware won't let you modify configuration while it's running.

We will need to create 2 network adapters, which should look like this. The primary "external" interface should be configured like this and the secondary like this.

It is also a good idea to give your build hypervisor plenty or RAM and CPU cores. I prefer 8G RAM and 6 cores for my macbook.

Once your vmware vm has been configured you can start it up. Run through the setup as outlined in the wiki. Once you reach the last question asking you to review the final configuration, answer "Y". You'll need to adjust that config file to look like this:

#
# This file was auto-generated and must be source-able by bash.
#

coal=true
# admin_nic is the nic admin_ip will be connected to for headnode zones.
admin_nic=YOUR_MAC_HERE
admin_ip=192.168.200.2
admin_netmask=255.255.255.0
admin_network=
admin_gateway=

external_nic=YOUR_MAC_HERE
external0_ip=dhcp
external0_netmask=
external0_network=
external0_gateway=dhcp

headnode_default_gateway=172.16.89.2

dns_resolvers=8.8.8.8,8.8.4.4
dns_domain=

ntp_hosts=0.smartos.pool.ntp.org
compute_node_ntp_hosts=dhcp

Your mac addresses can be found in the networking configuration for each nic, here's an example.

Build Zone

1. From the global zone, create a build zone

2. Import the recommended build image

```bash
$ imgadm import a1d74530-4212-11e3-8a71-a7247697c8f2
```

2. Create the build zone

```bash
$ vmadm create <<-END
{
  "brand": "joyent",
  "max_physical_memory": 8192,
  "tmpfs": 8192,
  "fs_allowed": "ufs,pcfs,tmpfs",
  "image_uuid": "a1d74530-4212-11e3-8a71-a7247697c8f2",
  "quota": 15,
  "alias": "build",
  "hostname": "build",
  "resolvers": ["8.8.8.8", "8.8.4.4"],
  "nics": [
    {
      "nic_tag": "external",
      "ip": "dhcp",
      "default": true
    },  
    {
      "nic_tag": "admin",
      "ip": "192.168.200.3",
      "netmask": "255.255.255.0",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_restricted_traffic": true
    }
  ],
  "internal_metadata": {
    "root_pw": "password",
    "admin_pw": "password"
  }
}
END
```

2. Configure the build zone

3. zlogin into the build zone

```bash
$ zlogin $(vmadm list | grep build | awk '{ print $1 }')
```

2. install git

```bash
$ pkgin in git
```

3. mount the source directory from the workstation

```bash
$ /usr/sbin/mount -Fnfs -o rw,intr 172.16.89.1:/Users/tylerflint/Work/clones/smartos-live /root/smartos-live
```

4. create a build directory

```bash
$ mkdir -p /root/build
```

5. create helper script to sync code from your workstation into the build dir

```bash
cat > /root/sync.sh <<-END
#!/usr/bin/env bash

rsync \\
  -a \\
  -v \\
  /root/smartos-live/ \\
  /root/build
  
END

chmod +x /root/sync.sh
```

6. create a helper script to incrementally build image

```bash
cat > /root/build.sh <<-END
#!/usr/bin/env bash
if [[ -d /root/build/output ]]; then
  rm -rf /root/build/output
fi

if [[ -f /root/build/0-livesrc-stamp ]]; then
  rm -f /root/build/0-livesrc-stamp
fi

export MAX_JOBS=8

(cd /root/build && gmake live)

END

chmod +x /root/build.sh
```

7. create a helper script to publish the recent build

```bash
cat > /root/publish.sh <<-END
#!/usr/bin/env bash

if [[ -d /var/db/pxe/platform ]]; then
  rm -rf /var/db/pxe/platform
fi

mkdir -p /var/db/pxe/platform

if [[ -f /root/build/output/platform-latest.tgz ]]; then
  rm -f /root/build/output/platform-latest.tgz
fi

cp /root/build/output/platform-*.tgz /root/build/output/platform-latest.tgz

tar -xf /root/build/output/platform-latest.tgz -C /var/db/pxe/platform --strip-components=1

END

chmod +x /root/publish.sh
```

8. create pxe boot directory

```bash
mkdir -p /var/db/pxe/boot/grub
```

9. install pxegrub

```bash
wget --no-check-certificate -O /var/db/pxe/pxegrub http://aszeszo.googlepages.com/pxegrub
```

10. generate grub boot menu

```bash
cat > /var/db/pxe/boot/grub/menu.lst <<-EOF
default=0
timeout=5
min_mem64 1024

title Live 64-bit (text)
   kernel /platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA',smartos=true
   module /platform/i86pc/amd64/boot_archive

title Live 64-bit (noinstall)
   kernel /platform/i86pc/kernel/amd64/unix -B console=text,root_shadow='$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA',standalone=true,noimport=true
   module /platform/i86pc/amd64/boot_archive

title Live 64-bit (text) +kmdb
   kernel /platform/i86pc/kernel/amd64/unix -kd -B console=text,root_shadow='$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA',smartos=true
   module /platform/i86pc/amd64/boot_archive

title Live 64-bit (ttya) +kmdb
   kernel /platform/i86pc/kernel/amd64/unix -kd -B console=ttya,ttya-mode="115200,8,n,1,-",root_shadow='$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA',smartos=true
   module /platform/i86pc/amd64/boot_archive

title Live 64-bit (ttya)
   kernel /platform/i86pc/kernel/amd64/unix -B console=ttya,ttya-mode="115200,8,n,1,-",root_shadow='$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA',smartos=true
   module /platform/i86pc/amd64/boot_archive
EOF
```

11. install dnsmasq

```bash
pkgin -y in dnsmasq
```

12. configure dnsmasq

```bash
cat > /opt/local/etc/dnsmasq.conf <<-EOF
  interface=net1
  dhcp-option=3,192.168.200.2
  dhcp-range=192.168.200.100,192.168.200.150,1h
  dhcp-boot=pxegrub,192.168.200.3,192.168.200.3
  enable-tftp
  tftp-root=/var/db/pxe/
  dhcp-leasefile=/tmp/dnsmasq.leases
  log-facility=/var/log/dnsmasq.log
  log-dhcp
EOF
```

13. start dnsmasq

```bash
svcadm enable -s dnsmasq
```

3. Build the initial image

4. create configure script

```bash
cp /root/build/sample.configure.smartos /root/build/configure.smartos
```

2. configure

```bash
(cd /root/build && ./configure)
```

3. build

```bash
export MAX_JOBS=8
(cd /root/build && gmake live)
```

4. publish the first image

```bash
/root/publish.sh
```

Build Zone Hypervisor

The hypervisor setup that will test the image will be nearly identical to the setup of the build hypervisor with the only exeptions being the /usbkey/config file and the vmware boot order. It's easiest to configure the test node with the Joyent provided vmware image as we did above, and then subsequent boots will load the custom image instead.

To ensure the node will pxe boot, you'll need to set the host-only-adapter as the boot device, like this.

During the setup, once you reach the last question asking you to review the final configuration, answer "Y". You'll need to adjust that config file to look like this:

#
# This file was auto-generated and must be source-able by bash.
#

#coal=true
# admin_nic is the nic admin_ip will be connected to for headnode zones.
admin_nic=YOUR_MAC_HERE
admin_ip=dhcp
admin_netmask=
admin_network=
admin_gateway=dhcp

external_nic=YOUR_MAC_HERE
external0_ip=dhcp
external0_netmask=
external0_network=
external0_gateway=dhcp

headnode_default_gateway=172.16.89.2

dns_resolvers=8.8.8.8,8.8.4.4
dns_domain=

ntp_hosts=0.smartos.pool.ntp.org
compute_node_ntp_hosts=dhcp

Your mac addresses can be found in the networking configuration for each nic, here's an example.

And that's it, once it reboots it will boot into your custom build image.

Iterate!

Workstation

Modify existing files, add new files, remove old files, etc.

Build

./sync.sh && ./build.sh && ./publish.sh

Test!

On the test node, simply reboot:

reboot

Summary

The initial setup takes a bit of time, but after the initial setup and build, the iterative builds and are quite quick. I've found that it takes aproximately 30 minutes for the initial setup, and about 3 hours for the initial build. I usually kickoff an initial build before I go to bed.

In my experience, a source code change from my workstation can be on the test node within 10 minutes.

This is a work in progress. Any suggestions would be appreciated.