nixos-infect

#! /usr/bin/env bash

# nixos-infect is so named because there's a good chance the system will get
# sick if anything goes wrong, and possibly die, requiring reprovisioning.
# Use with caution.
#
# WARNING NB This script wipes out the targeted host's root filesystem when it
# runs to completion. Any errors halt execution. set -x is used to help debug,
# as often a failed run leaves the system in an inconsistent state, requiring a
# rebuild (in DigitalOcean panel: Droplet Settings -> "Destroy" -> "Rebuild
# from original").
#
# TO USE:
#  - Add any custom config you want (see notes below)
#  - Deploy a Debian 8.3 x64 droplet (enable ipv6; add your ssh key)
#  - cat customConfig.optional nixos-infect | ssh root@targethost bash
#
# This was last tested with the DigitalOcean Debian 8.3 x64 and Ubuntu 15.10
# x64 images. Different versions and archs (namely i386) should work as well,
# but then, there's not much point in selecting something different if you
# intend to wipe out the fs, as this script does. Some Ubuntu droplets have gpt
# partition tables but only a single ext4 root partition. It's way too much
# effort to try to get nixos to install grub using blocklists, so just avoid
# improperly configured images like those.
#
# You may need to make minor modifications to use in other templates, but
# basically all that will ever need tweaking are already inlined in this file:
#  /etc/nixos/{,hardware-}configuration.nix : rudimentary mostly static config
#  /etc/nixos/networking.nix, networking settings determined at runtime
#    tweak if no ipv6, different number of adapters, etc.
#
# Motivation for this script: nixos-assimilate should supplant this script
# entirely, if it's ever completed. nixos-in-place was quite broken when I
# tried it, and also took a pretty janky approach that was substantially more
# complex than this (although it supported more platforms): it didn't install
# to root (/nixos instead), left dregs of the old filesystem (almost always
# unnecessary since starting from a fresh deployment), and most importantly,
# simply didn't work for me! (old system was being because grub wasnt properly
# reinstalled)

set -ex

nixos_channel=nixos-unstable
#nixos_channel=nixos-16.03

makeConf() {
  # NB <<"EOF" quotes / $ ` in heredocs, <<EOF does not
  mkdir -p /etc/nixos
  local IFS=$'\n'; keys=($(grep -vE '^[[:space:]]*(#|$)' /root/.ssh/authorized_keys))
  cat > /etc/nixos/configuration.nix << EOF
{ ... }: {
  imports = [
    ./hardware-configuration.nix
    ./networking.nix # generated at runtime by nixos-infect
  ];

  boot.cleanTmpDir = true;
  networking.hostName = "$(hostname)";
  networking.firewall.allowPing = true;
  services.openssh.enable = true;
  users.users.root.openssh.authorizedKeys.keys = [$(for key in ${keys[@]}; do echo -n "
    \"$key\""; done)
  ];
}
EOF
  # (nixos-generate-config will add qemu-user and bind-mounts, so avoid)
  cat > /etc/nixos/hardware-configuration.nix << EOF
{ ... }: {
  imports = [ <nixpkgs/nixos/modules/profiles/qemu-guest.nix> ];
  boot.loader.grub.devices = [ "/dev/vda" ];
  fileSystems."/" = { device = "/dev/vda1"; fsType = "ext4"; };
}
EOF

  local IFS=$'\n'
  ip4s=($(ip address show dev eth0 | grep 'inet '          | sed -r 's|.*inet ([0-9.]+)/([0-9]+).*|{ address="\1"; prefixLength=\2; }|'))
  ip6s=($(ip address show dev eth0 | grep 'inet6 .*global' | sed -r 's|.*inet6 ([0-9a-f:]+)/([0-9]+).*|{ address="\1"; prefixLength=\2; }|'))
  gateway=($(ip route show dev eth0 | grep default | sed -r 's|default via ([0-9.]+).*|\1|'))
  gateway6=($(ip -6 route show dev eth0 | grep default | sed -r 's|default via ([0-9a-f:]+).*|\1|'))
  ether0=($(ip address show dev eth0 | grep link/ether | sed -r 's|.*link/ether ([0-9a-f:]+) .*|\1|'))
  ether1=($(ip address show dev eth1 | grep link/ether | sed -r 's|.*link/ether ([0-9a-f:]+) .*|\1|'))
  nameservers=($(grep ^nameserver /etc/resolv.conf | cut -f2 -d' '))

  cat > /etc/nixos/networking.nix << EOF
{ ... }: {
  # This file was populated at runtime with the networking
  # details gathered from the active system.
  networking = {
    nameservers = [$(for a in ${nameservers[@]}; do echo -n "
      \"$a\""; done)
    ];
    defaultGateway = "${gateway}";
    defaultGateway6 = "${gateway6}";
    interfaces = {
      eth0 = {
        ip4 = [$(for a in ${ip4s[@]}; do echo -n "
          $a"; done)
        ];
        ip6 = [$(for a in ${ip6s[@]}; do echo -n "
          $a"; done)
        ];
      };
      # eth1 is for private networking or something?
      eth1.useDHCP = false;
    };
  };
  services.udev.extraRules = ''
    KERNEL=="eth*", ATTR{address}=="${ether0}", NAME="eth0"
    KERNEL=="eth*", ATTR{address}=="${ether1}", NAME="eth1"
  '';
}
EOF
  #! /usr/bin/env bash
  # NB put your semi-sensitive (not posted to github) configuration in a separate
  # file and include it via this customConfig() function. e.g.:
  #  customConfig() {
  #    cat > /etc/nixos/custom.nix << EOF
  #    { config, lib, pkgs, ... }: {
  #    }
  #    EOF
  #  }
  #
  # then you can add the files in configuration.nix's imports above and run something like:
  #   cat customConfig nixos-infect | root@targethost bash
  if [[ `type -t customConfig` == "function" ]]; then customConfig; fi
}

makeSwap() {
  swapFile=`mktemp`
  dd if=/dev/zero of=$swapFile bs=1M count=$((1*1024))
  chmod 0600 $swapFile
  mkswap $swapFile
  swapon $swapFile
}

makeConf
makeSwap # smallest (512MB) droplet needs extra memory!

apt-get install -y curl rsync sudo

groupadd -r nixbld
seq 1 10 | xargs -I{} useradd -c "Nix build user {}" -d /var/empty -g nixbld -G nixbld -M -N -r -s `which nologin` nixbld{}

curl https://nixos.org/nix/install | sh

source ~/.nix-profile/etc/profile.d/nix.sh

nix-channel --add https://nixos.org/channels/${nixos_channel} nixos
nix-channel --update

newRootImg=`mktemp`
newRootMount=`mktemp -d`
oldRootMount=`mktemp -d`

export NIXOS_CONFIG=/etc/nixos/configuration.nix
nix-env -i                                                      \
  -f /nix/var/nix/profiles/per-user/root/channels/nixpkgs/nixos \
  -A config.system.build.nixos-install

# XXX GOTCHA NB bindmount causes /bin/bash permission BUG on many
# versions (nix 1.10-1.11, nixpkgs 15-16), so we must use loopback image instead.
# See: https://github.com/NixOS/nixpkgs/issues/10230
dd if=/dev/zero of=$newRootImg bs=1M count=2047 # XXX 2048+ will cause mkfs.ext4 to fail on x86
mkfs.ext4 -F $newRootImg
mount $newRootImg $newRootMount

rsync -Ra /./etc/nixos $newRootMount
nixos-install --root $newRootMount

mount -B / $oldRootMount

# Everything up to this point is revertible; this is the truly destructive step.
rsync -a --delete --exclude=$(dirname $newRootMount) $newRootMount/ $oldRootMount

# Restore access to commands
/nix/var/nix/profiles/system/activate # (this destroys resolv.conf)
for a in ${nameservers[@]}; do echo "nameserver $a" >> /etc/resolv.conf; done
source /nix/var/nix/profiles/system/etc/profile


# grub/initrd was probably installed incorrectly (using false root device), so we need a final rebuild
# TODO see aszlig's comment in issue about not even having to call rebuild, just nix-build system or something; without ever having to use nixos-install either? and separate ext4fs?
#   man nixos-rebuild mentions this!!: nixos-rebuid build == nix-build /path/to/nixpkgs/nixos -A system
nixos-rebuild boot --install-grub

#swapoff $swapFile && rm -f $swapFile || true
sync
echo "You may now Ctrl-C or otherwise terminate this process."
reboot -f

XXX GOTCHA NB bindmount causes /bin/bash permission BUG on many

versions (nix 1.10-1.11, nixpkgs 15-16), so we must use loopback image instead.

Do you have a link to the respective issue? Was this resolved?

Thank you for this, BTW 😄

To "infect" an HVM EC2 instance, I had to tweak the config just slightly:

{ config, lib, pkgs, modulesPath, ... }:

{
  imports = [
    "${modulesPath}/virtualisation/amazon-image.nix"
  ];

  # This option configures grub and the root FS for an HVM instance.
  # This option is marked internal, and is usually set by something like nixops.
  # We're not using nixops to create the instance, so we need to set it ourselves.
  ec2.hvm = true;

  # ... other stuff ...
}

With that, I let DHCP take care of network configuration, and I let the amazon-image.nix configure grub and the root FS, so I don't need ./networking.nix or ./hardware-configuration.nix.

I somehow missed that mention of "EOF" in the bash manpage, thanks.

I didn't file the bug because I've seen many different incarnations of it and it seems hard to track down. Sometimes it's "unshare" not working, other times it's some weird part of the runtime not loading properly (manifested as /bin/bash permission error). I just accepted nixos-install only supported installation to actual mount points, although I agree that it really should support any type of directory/mount. I think nixos-install gotten a little more tolerant over time, but it still breaks rather easily when run from non-NixOS linuxes. I'll revisit trying to install to an unmounted dir and file a bug if I hit something again.

Glad you found it helpful. Out of curiosity, what use case do you have where this is better than using the nixops AMIs for EC2?

Oops, I totally forgot that I did actually comment on this issue I found as it was near enough to what I was seeing:

NixOS/nixpkgs#10230

The "unshare" problem is something I saw on non-NixOS invocations of nixos-install; it's probably something to do with the particular, actual OS.

Moved to https://github.com/elitak/nixos-infect for issue tracking.