Clinical NGS bioinformatics server at reasonable cost and TAT

dx-ngs.md

Purpose

A proof-of-concept high-performance server for primary and secondary NGS analyses with reasonable cost and TAT.

Hardware and OS

Acquired a Dell Precision 5820 tower workstation in mid 2018 with the following specs. Minimally, you want fast single-thread performance, at least 64GB RAM preferably ECC, and very speedy disks. A GPU with at least 16GB VRAM allows you to run Nvidia's Parabricks v4.4 or at least 12GB VRAM for Parabricks v3.8.

• Intel Xeon W-2145 (supports ECC memory and AVX-512; decent single-thread performance)
• 208GB DDR4-2666 ECC Memory (ECC reduces odds of data corruption)
• NVIDIA RTX A2000 (12GB VRAM)
• 1x 960GB Intel Optane 905P PCIe SSD (nvme0n1)
• 4x 4TB Samsung 990 EVO PCIe NVMe SSD (nvme1n1, nvme2n1, nvme3n1, nvme4n1)
• 1x 16TB 7200rpm SATA 3.5" HDD (sda)

We'll choose to install Clear Linux for the latest kernel optimized for performance.

1. On a Windows PC, download the Clear Linux Server ISO image.
2. Boot into BIOS on the server (press F2 during Dell logo), and ensure that:
   • SATA Operation is set to AHCI (disables hardware RAID)
   • Secure Boot is disabled (does not work with Clear Linux)
   • AC Recovery is set to Last Power State
   • POST Behaviour is set to Continue on Warnings and Keyboard Error Detection is disabled
   • Virtualization, VT for Direct I/O, and Trusted Execution are disabled (incompatible with NVIDIA drivers)
3. Follow these instructions to create a bootable USB stick.
4. Plug in the USB stick and boot into it (press F12 during Dell logo).
5. After the live server image boots, follow the instructions to login as root and start clr-installer.
6. Use the smaller SSD (nvme0n1) as the installation media and add yourself as a sudo user.
7. Click "Advanced Options" to set the hostname, make sure other options are set to your preference, and select "Install".
8. After booting into the installed OS, login and install some basic bundles:

   sudo swupd bundle-add -y sysadmin-remote storage-utils git vim

9. Follow instructions here to install NVIDIA drivers.
10. (Optional) Install a minimal desktop environment and reboot:

    sudo swupd bundle-add -y os-utils-gui lightdm
    sudo systemctl enable --now lightdm

11. Create a unix group for your team (dx below), make it your primary group, and logout:

    sudo groupadd dx
    sudo usermod -g dx $USER
    sudo groupdel $USER
    exit

12. Log back in and type id to confirm that your gid is now dx and you are still in group wheel.
13. Create a level 0 RAID using the four larger SSDs, mount it as /hot, and make it writable for your team:

    sudo mdadm --create --verbose --level=0 --raid-devices=4 /dev/md0 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
    sudo mkfs.xfs -f /dev/md0
    sudo mkdir /hot
    sudo blkid -o export /dev/md0 | grep ^UUID | xargs -I{} echo "{} /hot xfs defaults,noatime 0 2" | sudo tee -a /etc/fstab
    sudo systemctl daemon-reload
    sudo mount /hot
    sudo chown root:dx /hot
    sudo chmod 775 /hot

14. Format and mount the large HDD as /bkp and use Samba/NetBIOS to share it over the network:

    sudo mkfs.xfs -f /dev/sda
    sudo mkdir /bkp
    sudo blkid -o export /dev/sda | grep ^UUID | xargs -I{} echo "{} /bkp xfs defaults 0 2" | sudo tee -a /etc/fstab
    sudo systemctl daemon-reload
    sudo mount /bkp
    sudo chown root:dx /bkp
    sudo chmod 775 /bkp
    sudo mkdir -p /etc/samba
    echo -e "[bkp]\n  path=/bkp\n  writeable=yes" | sudo tee -a /etc/samba/smb.conf
    sudo systemctl enable --now smb
    sudo systemctl enable --now nmb

15. Set a samba password for your username using sudo smbpasswd -a $USER