Installing and Configuring SmartOS on a budget server (with a /29)

smartos-on-a-budget.sh

# Licensed under CC BY 3.0 http://creativecommons.org/licenses/by/3.0/
# Derived works must attribute https://gist.github.com/4492300 at the beginning, and the date.

##################################################################
Installing and Configuring SmartOS on a budget server (with a /29)
##################################################################

# if you find this gist useful, please star it

# please be aware that budget hosting companies usually cut corners somewhere,
# you have been warned!

# thanks to: jamesog, linuxprofessor, ryancnelson for help with routing

############### 
1. Installation
###############

# log in to the Linux "Recovery System".

# download the latest SmartOS USB image
wget https://download.joyent.com/pub/iso/latest-USB.img.bz2
bunzip2 latest-USB.img.bz2

# note: SmartOS requires a USB key on your server
# find out its device name using:
fdisk -l
# on my system it is /dev/sdd

# write the image to the USB key (/dev/sdd)
# IMPORTANT /dev/sdd WILL BE ERASED
dd if=latest-USB.img of=/dev/sdd  bs=1024

# now make the USB drive bootable
fdisk /dev/sdd
# Command (m for help): a
# Partition number (1-4): 1
# Command (m for help): w

reboot

# now request a remote console (e.g. LANA, RAC) from the budget hosting company, and log in via the
# Java applet (usually).

# use the robot to request a reboot of the server, and press DEL when the POST shows to enter the BIOS.

# your BIOS is set to boot from the network - do not change that, you need it to gain access to the 
# recovery system in the future. Instead, change the order of the fallback local boot options. These
# are labelled "Hard Drive BBS Priorities" on my motherboard - you want to set this to boot from the
# USB key.

# (OPTIONAL) [if you want your zpool to only use some (but not all) of the available drives, then make
# a note of the order in which they are displayed on the POST screen, so that you know which drives are
# which during SmartOS install - the will be in the same order]

# save settings and exit the BIOS

# machine will try network boot and fail, then try a local boot from the USB key. You should see the
# SmartOS GRUB screen now. Let it boot the SmartOS installer.

# follow the SmartOS install wizard, using 'dhcp' as the IP address. Reboot, and you're finished with
# the LANA.

###

# MESSED UP? If it all goes wrong, you can boot SmartOS with the (noinstall) option, using the image's
# default root password. Then list disks with 'format' and delete them with 'fdisk /dev/rdsk/c0t0d0p0'
# - note the p0 at the end. Root password available here https://download.joyent.com/pub/iso/

#################################
2. Basic Configuration (OPTIONAL)
#################################

# (OPTIONAL) set a hostname: http://wiki.smartos.org/display/DOC/Setting+a+static+hostname+at+boot+time

# (OPTIONAL) upload a root SSH key: http://www.perkin.org.uk/posts/smartos-global-zone-tweaks.html

###########################
3. Configuring a /29 subnet
###########################

# Many budget hosting companies will give you a /29 (or indeed a /28) subnet which is *statically routed*
# to your server's main IP. In the subnet x.x.x.200/29 the first address (x.x.x.200) is used to identify
# the network, and the last address (x.x.x.207) is used for broadcast, leaving six usable IP addresses
# (but we have to use one for the gateway, so we only get FIVE usable IPs).

# The budget hosting company will route the subnet traffic to the main IP of your server, and expect you 
# to provde your own gateway for the subnet. Threfore we have to set up a vnic in the global zone to act as
# a router for the subnet. This uses up the the first available ip of our /29.

dladm create-vnic -l rge0 vnic0                             # rge0 = physical nic (from ifconfig)
ifconfig vnic0 plumb x.x.x.201 netmask 255.255.255.248 up   # x.x.x.201 = first usable ip
svcadm enable route                                         # turn on ipv4 routing

# check that you can now ping x.x.x.201 from the internet

# now we can launch zones using the five remaining ips, for example x.x.x.202
# the gateway is set to x.x.x.201 which is the router we just set up in the global zone

cat > /tmp/zonedef << EOF
{
  "brand": "joyent",
  "autoboot": true,
  "dataset_uuid": "fdea06b0-3f24-11e2-ac50-0b645575ce9d",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "x.x.x.202",
      "netmask": "255.255.255.248",
      "gateway": "x.x.x.201"
    }
  ]
}
EOF

vmadm create -f /tmp/zonedef

# check that you can now ping x.x.x.202 from the internet

# and that the zone can reach the internet:
zlogin <Zone UUID>
ping google.com
# google.com is alive
exit

### Persistance ###

# if everything works, then we need to persist the configuration of the global zone so that it
# survives a reboot:

# first create an SMF service to run a script on boot
# you don't need to customise any of this XML
mkdir -p /opt/custom/smf

cat >> /opt/custom/smf/subnet-routing-setup.xml << EOF
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='site/subnet-routing-setup' type='service' version='1'>
    <create_default_instance enabled='true'/>
    <single_instance/>
    <dependency name='network' grouping='require_all' restart_on='error' type='service'>
      <service_fmri value='svc:/milestone/network:default'/>
    </dependency>
    <dependency name='filesystem' grouping='require_all' restart_on='error' type='service'>
      <service_fmri value='svc:/system/filesystem/local'/>
    </dependency>
    <exec_method name='start' type='method' exec='/opt/custom/scripts/subnet-routing-setup' timeout_seconds='60'>
      <method_context>
        <method_credential user='root' group='staff'/>
        <method_environment>
          <envvar name='PATH' value='/usr/bin:/usr/sbin:/bin'/>
        </method_environment>
      </method_context>
    </exec_method>
    <exec_method name='stop' type='method' exec=':true' timeout_seconds='0'/>
    <property_group name='startd' type='framework'>
      <propval name='duration' type='astring' value='transient'/>
    </property_group>
  </service>
</service_bundle>
EOF

# then create the actual script
# you need to customise this as shown previously
mkdir -p /opt/custom/scripts/

cat >> /opt/custom/scripts/subnet-routing-setup << EOF
#!/bin/sh
. /lib/svc/share/smf_include.sh

dladm create-vnic -l rge0 vnic0                               # <-- customise
ifconfig vnic0 plumb x.x.x.201 netmask 255.255.255.248 up     # <-- customise
svcadm enable route

exit $SMF_EXIT_OK
EOF

chmod +x /opt/custom/scripts/subnet-routing-setup

# now, reboot and check that everything works
# you can use ifconfig, svcs, and routeadm to debug things.
reboot

This is great. But do you have any idea how to make this work when you get individual IPs from Hetzner rather than a /29 subnet ?)

I have 3 IPs available each with a dedicated MAC address. I am having problems getting networking / routing working with my zones, can't contact the internet from them, but the GZ and the other zones can ping each other.

I really would love to do this, but when booting the image from the usb key It says: "Divide Error" and dumps.
I tried several BIOS options with no luck. Did you experience something similar before?

On PX91 machines you should disable X2APIC option in BIOS. Otherwise, SmartOS will crash. See (TritonDataCenter/smartos-live#675) for details.

On SmartOS 20170216T015949Z I had to run svcadm enable ipv4-forwarding before running svcadm enable route to be able to ping the internet from vnic0 in the global zone (and from inside the non-global zone), and to be able to ping the non-global zone IP from the internet.

Hetzner is now blocking MAC addresses not routed to the main IP so this is no longer valid. Adding either an etherstub or creating a vlan is the only option at this moment or SmartOS will no longer be accommodated by Hetzner.

This is great. But do you have any idea how to make this work when you get individual IPs from Hetzner rather than a /29 subnet ?)

@johnknott did you figure out how to solve this?

@johnknott did you figure out how to solve this?

@wyan Unfortunately, I moved onto other things. This comment has reminded me how much I loved SmartOS though. Funnily enough, just this last week I have set up a Proxmox instance - on a Hetzner /29 Subnet. 7 years on and my life appears to have stagnated. At least I got networking working properly this time ...