Created: 26.02.2016
Let us never forget that most modern hardware is basically a cheap or sometimes expensive coaster without software.
For development the new NUC6i5SYK is seemingly great. 32GB of RAM, 2 cores and 2 "hyper threads". This NUC also features an NVMe port running at x4 over PCIe 3.0. At 10 VM's the NUC should be able to support about 150 MB/s (theoretically) of disk traffic per VM, this is usually the thing that will prevent a VM server from performing well. For a Micro Services type architecture this is an ideal environment for development.
The primary issue with Linux on the Skylake NUC is that none of the major distributions support the Skylake Processor and peripherals out of the box fully. There are two pieces of hardware that lack good support under most of the shipping kernels, the i915 graphics and most importantly the v219 ethernet adapter. The ethernet support is basically a non-starter.
These seemed like the major issues... but there were some others.
According to the Inquirer the target kernel with substantial Skylake support is reported to be 4.3.
So the search starts here - where can we find a distro that will support the hardware?
An educated guess said that Xen Server 6.5 would not work - indeed, it failed to recognize the network card. Citrix's web site shows that they don't support the i219 so I did not spend additional time beyond the "Do you have additional drivers?" screen.
Ubuntu was the next stop. 15.10 server and desktop both have the 4.2 kernel. It seemed "close enough". Indeed it is very close, but at first glance the Ubuntu installer is broken leaving an unbootable system in UEFI and Legacy modes (more on this later).
Fedora 23 and CentOS 7 were also given a chance, these did not work either with a series of timeout messages and finally giving up.
After exhausting the seemingly "easy" options I turned to Arch Linux which is known for up-to-date distributions and packages. I thought surely I would be successful. Arch was not chosen initially because it is a bit more complex. Arch is nice and flexible but not "simple". Fortunately, Arch has some really great documentation, that later helped me resolve the issues with Ubuntu.
Arch installed... except the system wouldn't boot after it was installed. Meh.
To make a long story short, the version of GRUB supplied with Arch [does not have a patch applied] (https://bugs.archlinux.org/task/47447) that is [required] (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785149) to boot from NVMe. It turns out that almost all of the booting issues encountered were related to NVMe support, except for UEFI support under Ubuntu which was due to the partition layout. For Arch this is an easy fix, download GRUB2, trunk. Compile and install as a first order of business. Ubuntu's fix is a bit different.
After installing Arch I found that the current Xen package is not up to date. Commentary on how to make it work is scattered about. Ultimately I attempted to compile Xen on Arch system using the multilib-build method but to no avail. Unfortunately this precluded the use of Arch for this endeavor.
After having installed Arch, I applied a few of the techniques to Ubuntu and arrived at a working system.
Taking these experience together told me that Linux would run on the NUC. The "catch" was that I'd either need a working build environment(!) for Xen or preferably prebuilt packages. Enter Ubuntu - they do have a working package for Xen, but they are missing the xapi packages to use Xen Center.
To successfully install Xen and Ubuntu you will need:
- 1 NUC6i5SKY
- 1 erasable USB 3.0 drive, SD Card or USB 2.0 drive in order of monetary value of time.
- 1 Desktop or Laptop
- 1 Network connection between NUC and PC/Laptop
- A working internet connection
Be aware that a standard install will not work without the additional actions a.k.a "Special Sauce"
Download Ubuntu Server 15.10 - AMD64
Recommendations
Windows - Universal USB Installer
Linux - dd
There are two methods for installing an OS on the NUC. The first is a normal USB drive. The second is the SD card slot. If a USB 3.0 drive is handy then I recommend this as the best option. If an SD card reader and writer is readily available then it can be a fast way of booting and installing since it is substantially faster than writing a USB 2.0 device. Either way, dd or the Universal Installer will work with any of these media types.
One word of caution, if an SD card is used the installer will overwrite the boot sector of the SD card. So be sure to write protect it before booting.
The NUC carries the Intel "Visual BIOS" this is an upgrade to the traditional AMI model. There are some important settings that need to be modified before installing. The general outline out of the box is:
- Download the latest BIOS Version 28 write it to a FAT32 formatted USB drive or SD card
- Connect monitor
- Connect keyboard
- Connect mouse (the visual BIOS is much easier with the mouse)
- Insert the installation media before turning the machine on
- Power on
- Press F2 repeatedly
- Select the BIOS update and update, it'll take some time
- Insert the bootable media
- Reboot
- Press F2 repeatedly
- Disable UEFI Boot
- Enable Legacy Boot
- Be sure to re-enable the M.2 Drive (hidden in some menus)
- Reorder the boot priority such that the install media is first
- Hit F10 to save and reboot
At this point you should see Ubuntu 15.10 GRUB loader. It should have a graphic logo, if it is just a plain console then it is probably in UEFI mode.
For this installation only Basic Server and OpenSSH were enabled. All other options are at the user's discretion. It is recommended to use a static IP during network configuration. After rebooting the system the console will no longer work and further actions will be required via SSH. Don't worry about the GRUB errors as the installer completes.
At the conclusion of the installation DO NOT REBOOT
-
Select "Go Back" on the [!!] Finish the installation dialog
-
Under [!] Ubuntu installer main menu select "Execute a shell"
The primary issue is that the installer does not properly identify the NVMe device as the GRUB install location. As a result GRUB must be installed manually on the NVMe drive.
The first non-standard task is to fix up the boot loader with note that:
The IP address of the new system will be required after rebooting
chroot /target /bin/bash
grub-install /dev/nvme0n1
grub-mkconfig -o /boot/grub/grub.cfg
vi /boot/grub/grub.cfg
exit
reboot
These commands change the currently running system root to /target. Target is the installer mounted location for the NVMe drive and where all of the new system files have been written. When the context is changed all commands are running under the context of the the newly installed system. We then execute grub-install on the device, not a partition. The device is denoted by "nvme0n1" (partitions have a pX nomenclature e.g. nvme0n1p1). grub-install will start at block zero on the device and write the boot loader to its proper location. Finally, we generate a new grub configuration. The generated configuration should be inspected for correctness. Exiting the chroot context is required before rebooting.
At this point the system is operational - except that it may not appear this way. Some of the GRUB options, in conjunction with the partially working display drivers cause the physical console to be inaccessible. A quick note on Kernel versions. Please note that some of the kernels tested did not work, this was the first working version.
Login to the new system using the user/pass
ssh [install ip]
Modify the GRUB defaults, ensure
sudo vi /etc/default/grub
has these these modifications
GRUB_HIDDEN_TIMEOUT=5
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=5
:q and setup the install location
sudo mkdir -p /tmp/linux-4.3.6; \
cd /tmp/linux-4.3.6; \
install the new Kernel
sudo wget \
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.6-wily/linux-headers-4.3.6-040306-generic_4.3.6-040306.201602191831_amd64.deb \
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.6-wily/linux-headers-4.3.6-040306_4.3.6-040306.201602191831_all.deb \
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.6-wily/linux-image-4.3.6-040306-generic_4.3.6-040306.201602191831_amd64.deb
sudo dpkg -i linux-*.deb
verify the GRUB settings contain the new Kernel (The new default should be 4.3.6)
vi /boot/grub/grub.cfg
reboot
sudo reboot
The console should be fixed after this change, next install Xen
ssh [host ip]
sudo apt install xen-hypervisior xen-tools
Verify the Xen boot settings
vi /boot/grub/grub.cfg
make any modifications and reboot
sudo reboot
The default boot will change to Xen and the boot sequence will be a bit different at this point the NUC should be running with a Dom 0 and the xl commands can be used to start provisioning VM's.
Keywords: NUC6i5SYK, skylake, ubuntu, xen