| #!/bin/bash | |
| set -uo pipefail | |
| error() { | |
| echo "Error: $1" >&2 | |
| exit 1 | |
| } | |
| if (( $# != 3 )); then |
Reproduce the double free or corruption (fasttop) error from Dragen 4.3.6 on their CentOS 7.9 image on Azure NP10 VMs.
This error was found to be correlated to Dragen's code for streaming FASTQs from Azure blob storage. As a workaround, we now use azcopy to copy FASTQs onto VM local storage before alignment, which significantly reduces runtime too.
Build an Azure stack to operate NP-series VMs on Azure with Dragen's pay-as-you-go (PAYG) license
Microsoft.Network, Microsoft.Storage, and Microsoft.Compute.Standard NPS Family vCPUs in your preferred region. These VMs are only available in some regions per the FAQs here. Based on demand for these SKUs in your region, you may also need to submit a service request and justify youA proof-of-concept high-performance server for primary and secondary NGS analyses with reasonable cost and TAT.
Acquired a Dell Precision 5820 tower workstation in mid 2018 with the following specs. Minimally, you want fast single-thread performance, at least 64GB RAM preferably ECC, and very speedy disks. A GPU with at least 16GB VRAM allows you to run Nvidia's Parabricks v4.4 or at least 12GB VRAM for Parabricks v3.8.
Ensembl's VEP (Variant Effect Predictor) is popular for how it picks a single effect per gene as detailed here, its CLIA-compliant HGVS variant format, and Sequence Ontology nomenclature for variant effects.
The official instructions to install VEP have never worked well from the United States because of the flaky network connection to their FTP servers in the UK. So, we will instead use conda to install VEP and its dependencies and then manually download VEP caches and reference genomes using rsync.
If you don't already have conda, download and install it into $HOME/miniconda3:
curl -sL https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda.sh
bash miniconda.sh -bup $HOME/minic
| # Fetch the WGS gnomAD 3.1.2 per-chrom VCFs (the large size is mostly due to INFO fields): | |
| mkdir gnomad | |
| gsutil -m cp gs://gcp-public-data--gnomad/release/3.1.2/vcf/genomes/gnomad.genomes.v3.1.2.sites.chr*.vcf.bgz gnomad | |
| gsutil -m cp gs://gcp-public-data--gnomad/release/3.1.2/vcf/genomes/gnomad.genomes.v3.1.2.sites.chr*.vcf.bgz.tbi gnomad | |
| # Shortlist INFO fields we want to keep when merging these into a single VCF of reduced file size: | |
| bcftools view -h gnomad/gnomad.genomes.v3.1.2.sites.chr21.vcf.bgz | grep ^##INFO | cut -f3- -d= | grep -Ev "controls|non_cancer|non_neuro|non_topmed|non_v2|vep" | sort | less -S | |
| cadd_phred | |
| cadd_raw_score |
| #!/usr/bin/env python | |
| # Prereqs: Run "az login" to get a refresh token at "~/.azure/msal_token_cache.json" which expires only if unused for 90 days | |
| # Depends: pip install azure-identity azure-storage-blob | |
| # Sources: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/samples/blob_samples_containers.py | |
| STORAGE_ACCOUNT_URL = "https://blahdiblahdiblah.blob.core.windows.net" | |
| CONTAINER_NAME = "mdlhot" | |
| # Use the MSAL refresh token to get a temporary access token for use with blob storage libraries |
Ensembl's VEP (Variant Effect Predictor) is popular for how it picks a single effect per gene as detailed here, its CLIA-compliant HGVS variant format, and Sequence Ontology nomenclature for variant effects.
Instead of the official instructions, we will use mamba (conda, but faster) to install VEP and its dependencies. If you don't already have mamba, use these steps to download and install it into $HOME/mambaforge, then run a script that adds it to your $PATH:
curl -L https://github.com/conda-forge/miniforge/releases/download/4.12.0-0/Mambaforge-Linux-x86_64.sh -o /tmp/mambaforge.sh
sh /tmp/mambaforge.sh -bfp $HOME/mambaforge && rm -f mambaforge.sh
. $HOME/mambaforge
| # Prepare a conda environment with tools we will need: | |
| mamba create -n ref; conda activate ref | |
| mamba install -y -c bioconda htslib==1.13 bcftools==1.13 samtools==1.13 picard-slim==2.26.2 bwa-mem2==2.2.1 bwa==0.7.17 gsutil==4.68 | |
| # Fetch the alignment-ready human reference FASTA and index: | |
| gsutil -m cp gs://genomics-public-data/references/GRCh38_Verily/GRCh38_Verily_v1.genome.fa{,.fai} . | |
| # Index the reference FASTA for use with various tools: | |
| picard CreateSequenceDictionary -R GRCh38_Verily_v1.genome.fa | |
| bwa-mem2 index GRCh38_Verily_v1.genome.fa |
This guide will show you how to install conda and then use it to install nextflow and singularity for executing popular bioinformatics workflows. Unfortunately, singularity is not available on Windows or macOS. So, this guide will only target Linux environments. If you have to use Windows 10, then try WSL2. If you have to use macOS, then try a Virtual Machine.
Download the Miniconda3 installer for Linux environments:
curl -L https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda.shInstall into a folder named miniconda3 under your home directory, and delete the installer:
bash miniconda.sh -bup $HOME/miniconda3 && rm -f miniconda.sh