nixos-nas-drives-setup.md

NixOS NAS drives setup

My guide for setup of NixOS with LVM on multiple device LUKS on mdadm:

1. mdadm for RAID 1:

   • for system and data physical volumes,
   • you can use RAID5 or other levels, based on needs,

2. LUKS for device encryption:

   • separate LUKS for each disk array,
   • password reuse, unlock all at boot with same password,

3. LVM for / and /storage/data partitions:

   • can be grown to more disks in future without need of array rebuild,

Notes:

• this guide assumes there's just 2 HDDs without any SSD for /,
• this guide is preparation for real build, and was only tested in VirtualBox yet.

1st phase - initial dual HDD install

In the first phase, setup humble NAS with just 2 HDDs.

1.1 Disks partitioning

Setup partitions for the first drive:

gdisk /dev/sda

• o - create new empty GPT
• n - add partition, 500M, type EF00 EFI
• n - add partition, remaining space, type 8300 Linux
• w - write partition table and exit

Setup second drive:

gdisk /dev/sdb

• o - create new empty GPT
• n - add partition, 500M, type 8300 Linux, won't be used
• n - add partition, remaining space, type 8300 Linux
• w - write partition table and exit

1.2 Setup RAID 1 with mdadm

Next, setup RAID 1:

mdadm --create /dev/md1 --level=mirror --raid-devices=2 /dev/sda2 /dev/sdb2

1.3 Setup LUKS disk encryption

Setup encryption on top of RAID array:

cryptsetup luksFormat /dev/md1
cryptsetup luksOpen /dev/md1 enc-md1

1.4 Setup LVM

Setup LVM on top of LUKS encrypted RAID array:

# create physical volume
pvcreate /dev/mapper/enc-md1

# create volume group
vgcreate vgnas /dev/mapper/enc-md1

# create logical volumes
lvcreate -L 1G -n swap vgnas
lvcreate -L 6G -n root vgnas
lvcreate -l '100%FREE' -n data vgnas

1.5 Create and mount filesystems

Create filesystems for EFI, system, data and swap:

mkfs.fat /dev/sda1
mkswap /dev/vgnas/swap
mkfs.ext4 /dev/vgnas/root
mkfs.ext4 /dev/vgnas/data

Next, mount filesystems to desired structure:

mount /dev/vgnas/root /mnt

mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

mkdir -p /mnt/storage/data
mount /dev/vgnas/data /mnt/storage/data

swapon /dev/vgnas/swap

Now, you should see something like this:

# lsblk

NAME               MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0                7:0    0  1.9G  1 loop  
sda                  8:0    0    8G  0 disk  
├─sda1               8:1    0  500M  0 part  /mnt/boot
└─sda2               8:2    0  7.5G  0 part  
  └─md1              9:1    0  7.5G  0 raid1 
    └─enc-md1      254:0    0  7.5G  0 crypt 
      ├─vgnas-swap 254:1    0    1G  0 lvm   [SWAP]
      ├─vgnas-root 254:2    0    6G  0 lvm   /mnt
      └─vgnas-data 254:3    0  500M  0 lvm   /mnt/storage/data
sdb                  8:16   0    8G  0 disk  
├─sdb1               8:17   0  500M  0 part  
└─sdb2               8:18   0  7.5G  0 part  
  └─md1              9:1    0  7.5G  0 raid1 
    └─enc-md1      254:0    0  7.5G  0 crypt 
      ├─vgnas-swap 254:1    0    1G  0 lvm   [SWAP]
      ├─vgnas-root 254:2    0    6G  0 lvm   /mnt
      └─vgnas-data 254:3    0  500M  0 lvm   /mnt/storage/data
sr0                 11:0    1    2G  0 rom   /iso

1.6 Configure and install NixOS

Generate config:

nixos-generate-config --root /mnt

Ensure initrd is setup to open LUKS:

• get UUID of /dev/md1 to be opened at boot using:

  lsblk -f /dev/md1
  

• check configuration is present in:

  cat /mnt/etc/nixos/hardware-configuration.nix
  

• add configuration to vim /mnt/etc/nixos/configuration.nix - IMPORTANT to boot properly:

    boot.initrd.luks.devices = {
      "enc-md1".device = "/dev/disk/by-uuid/c09db7a6-33ea-4967-a8f1-7fda68e6cb95";
    };
  

Now, all is setup for NixOS installation:

• you might want to disable X and GNOME installation in case of low-sized root volume

nixos-install

2nd phase - adding more HDDs

In the second phase, NAS is upgraded with 2 more HDDs, which are presenting themselves as /dev/sdc and /dev/sdd.

2.1 Disks partitioning

Setup drives with partitions to extend LVM.

fdisk /dev/sdc
fdisk /dev/sdd

For both, do the same:

• g - create new empty GPT
• n - add partition, number 2, use all space, type 8300 Linux
• w - write partition table and exit

2.2 Setup RAID 1 with mdadm

Next, setup RAID 1:

mdadm --create /dev/md2 --level=mirror --raid-devices=2 /dev/sdc2 /dev/sdd2

2.3 Setup LUKS disk encryption

Setup encryption on top of RAID array:

cryptsetup luksFormat /dev/md2
cryptsetup luksOpen /dev/md2 enc-md2

• enter same passphrase to unlock all drives at the same time at boot.

Ensure initrd configuration is correct:

• get UUID of /dev/md1 to be opened at boot using:

  lsblk -f /dev/md1
  

• add configuration to vim /mnt/etc/nixos/configuration.nix - IMPORTANT to boot properly:

    boot.initrd.luks.devices = {
      "enc-md1".device = "/dev/disk/by-uuid/c09db7a6-33ea-4967-a8f1-7fda68e6cb95";
      "enc-md2".device = "/dev/disk/by-uuid/62381011-590d-4e40-aa22-12fdcfc98b71";
    };
  

• rebuild initrd:

  nixos-rebuild boot
  

• if you have forgotten to do the above, as I did, then you can boot live media and update it afterwards with some more effort.

2.4 Setup LVM

After initrd configuration was updated, you can setup new physical volume and extend volume group:

# create physical volume
pvcreate /dev/mapper/enc-md2

# extend volume group
vgextend vgnas /dev/mapper/enc-md2

# extend logical volumes
lvextend -L '+3G' /dev/vgnas/root
lvextend -L '+3G' /dev/vgnas/data

2.5 Resize filesystems

Resize filesystems to use extended size of logical volumes:

resize2fs /dev/vgnas/root
resize2fs /dev/vgnas/data

3rd phase - replacing HDDs with higher capacity

In the far future, once there won't be enough physical space to add new HDDs, start replacing drives two by two.

1. add higher capacity drives to the system,
2. mirror some raid array (rebuild RAID1 on 4 disks),
3. remove old smaller-sized drives,
4. grow array capacity to new size,
5. grow physical volume size,
6. grow logical volumes

Actual steps to be added,...