Test an NP-series VM on Azure with Dragen's pay-as-you-go (PAYG) license
Sign up for an Azure subscription if you don't already have one.
Visit Quotas in Azure Portal, login if needed, and increase the NP series quota to 40, so we can operate up to 4 NP10 VMs at a time, or 2 NP20 VMs. Based on demand for these SKUs in your region, you may also need to submit a service request and justify your use-case to a person before that quota gets approved.
A proof-of-concept high-performance server for primary and secondary NGS analyses with reasonable cost and TAT.
Acquired a Dell Precision 5820 tower workstation in mid 2018 with the following specs. Minimally, you want fast single-thread performance, at least 64GB RAM preferably ECC, and very speedy disks. A GPU with at least 16GB VRAM allows you to run Nvidia's Parabricks v4.4 or at least 12GB VRAM for Parabricks v3.8.
Ensembl's VEP (Variant Effect Predictor) is popular for how it picks a single effect per gene as detailed here, its CLIA-compliant HGVS variant format, and Sequence Ontology nomenclature for variant effects.
The official instructions to install VEP have never worked well from the United States because of the flaky network connection to their FTP servers in the UK. So, we will instead use conda to install VEP and its dependencies and then manually download VEP caches and reference genomes using rsync.
If you don't already have conda, download and install it into $HOME/miniconda3:
curl -sL https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda.sh
bash miniconda.sh -bup $HOME/minic
| # Fetch the WGS gnomAD 3.1.2 per-chrom VCFs (the large size is mostly due to INFO fields): | |
| mkdir gnomad | |
| gsutil -m cp gs://gcp-public-data--gnomad/release/3.1.2/vcf/genomes/gnomad.genomes.v3.1.2.sites.chr*.vcf.bgz gnomad | |
| gsutil -m cp gs://gcp-public-data--gnomad/release/3.1.2/vcf/genomes/gnomad.genomes.v3.1.2.sites.chr*.vcf.bgz.tbi gnomad | |
| # Shortlist INFO fields we want to keep when merging these into a single VCF of reduced file size: | |
| bcftools view -h gnomad/gnomad.genomes.v3.1.2.sites.chr21.vcf.bgz | grep ^##INFO | cut -f3- -d= | grep -Ev "controls|non_cancer|non_neuro|non_topmed|non_v2|vep" | sort | less -S | |
| cadd_phred | |
| cadd_raw_score |
| #!/usr/bin/env python | |
| # Prereqs: Run "az login" to get a refresh token at "~/.azure/msal_token_cache.json" which expires only if unused for 90 days | |
| # Depends: pip install azure-identity azure-storage-blob | |
| # Sources: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/samples/blob_samples_containers.py | |
| STORAGE_ACCOUNT_URL = "https://blahdiblahdiblah.blob.core.windows.net" | |
| CONTAINER_NAME = "mdlhot" | |
| # Use the MSAL refresh token to get a temporary access token for use with blob storage libraries |
Ensembl's VEP (Variant Effect Predictor) is popular for how it picks a single effect per gene as detailed here, its CLIA-compliant HGVS variant format, and Sequence Ontology nomenclature for variant effects.
Instead of the official instructions, we will use mamba (conda, but faster) to install VEP and its dependencies. If you don't already have mamba, use these steps to download and install it into $HOME/mambaforge, then run a script that adds it to your $PATH:
curl -L https://github.com/conda-forge/miniforge/releases/download/4.12.0-0/Mambaforge-Linux-x86_64.sh -o /tmp/mambaforge.sh
sh /tmp/mambaforge.sh -bfp $HOME/mambaforge && rm -f mambaforge.sh
. $HOME/mambaforge
| # Prepare a conda environment with tools we will need: | |
| mamba create -n ref; conda activate ref | |
| mamba install -y -c bioconda htslib==1.13 bcftools==1.13 samtools==1.13 picard-slim==2.26.2 bwa-mem2==2.2.1 bwa==0.7.17 gsutil==4.68 | |
| # Fetch the alignment-ready human reference FASTA and index: | |
| gsutil -m cp gs://genomics-public-data/references/GRCh38_Verily/GRCh38_Verily_v1.genome.fa{,.fai} . | |
| # Index the reference FASTA for use with various tools: | |
| picard CreateSequenceDictionary -R GRCh38_Verily_v1.genome.fa | |
| bwa-mem2 index GRCh38_Verily_v1.genome.fa |
This guide will show you how to install conda and then use it to install nextflow and singularity for executing popular bioinformatics workflows. Unfortunately, singularity is not available on Windows or macOS. So, this guide will only target Linux environments. If you have to use Windows 10, then try WSL2. If you have to use macOS, then try a Virtual Machine.
Download the Miniconda3 installer for Linux environments:
curl -L https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda.shInstall into a folder named miniconda3 under your home directory, and delete the installer:
bash miniconda.sh -bup $HOME/miniconda3 && rm -f miniconda.sh
| # GOAL: Create a small test dataset for CI/CD of cancer bioinformatics tools/pipelines | |
| # Prerequisites: A clean Linux environment with a working internet connection | |
| # Download and install mambaforge into a folder under your home directory: | |
| curl -L https://github.com/conda-forge/miniforge/releases/download/4.14.0-0/Mambaforge-Linux-x86_64.sh -o /tmp/mambaforge.sh | |
| sh /tmp/mambaforge.sh -bfp $HOME/mambaforge && rm -f /tmp/mambaforge.sh | |
| # Add the following to your ~/.bashrc file to activate the base environment whenever you login: | |
| if [ -f "$HOME/mambaforge/etc/profile.d/conda.sh" ]; then | |
| . $HOME/mambaforge/etc/profile.d/conda.sh |
On a Linux VM or Workstation with docker installed, fetch the GRCh38 FASTA, its index, and a pair of FASTQs:
wget -P /hot/ref https://storage.googleapis.com/genomics-public-data/references/GRCh38_Verily/GRCh38_Verily_v1.genome.fa{,.fai}
wget -P /hot/reads/test https://storage.googleapis.com/data.cyri.ac/test_L001_R{1,2}_001.fastq.gzIf on a Slurm cluster, here is an example of wrapping a docker run command in an sbatch request:
sbatch --chdir=/hot --output=ref/std.out --error=ref/std.err --nodes=1 --ntasks-per-node=1 --cpus-per-task=8 --mem=30G --time=4:00:00 --wrap="docker run --help"