Reproduce sporadic "double free" errors by Dragen on Azure NP10 VMs

Purpose

Reproduce the double free or corruption (fasttop) error from Dragen 4.3.6 on their CentOS 7.9 image on Azure NP10 VMs.

Update (Mar 2025)

This error was found to be correlated to Dragen's code for streaming FASTQs from Azure blob storage. As a workaround, we now use azcopy to copy FASTQs onto VM local storage before alignment, which significantly reduces runtime too.

Prerequisites

Sign up for an Azure subscription if you don't already have one.
Visit Quotas in Azure Portal, login if needed, and increase Standard NPS Family vCPUs in your preferred region. These VMs are only available in some regions per the FAQs here. Based on demand for these SKUs in your region, you may also need to submit a service request and justify your use-case to a person before that quota gets approved. Also note that a quota of 40 vCPUs lets you run 4 NP10 VMs at a time, 2 NP20 VMs, or 1 NP40 VM.
Visit this page, login if needed, and ensure that Status is set to Enable for the Azure subscription you intend to use. This allows programmatic deployment of the Dragen 4.3.6 Pay-As-You-Go (PAYG) VMs that we intend to use.
Generate an SSH key using the Ed25519 algorithm and store it as ~/.ssh/id_ed25519. We'll use this to SSH into VMs.
Install Azure CLI, then run az login --use-device-code and follow the instructions to link your subscription. Also install azcopy.
All commands below ran on an Azure D2pls_v6 ARM64 VM running Ubuntu 24.04 with these dotfiles, but you should be fine with Bash in any Linux environment.
Create an Azure resource group with a private VNet, subnet, network security group, and an Azure storage account with a container in which we can store FASTQs. Edit these env variables as appropriate.
```
RES_GROUP="use-mypenngenome-net-rg"
VNET="use-mypenngenome-vnet"
SUBNET="use-mypenngenome-sub1"
NS_GROUP="use-mypenngenome-nsg"
STOR_ACCT="mypenngenome"
CONTAINER="fastqs"
```

Data

Download the bs (BaseSpace CLI) binary and move it under /usr/local/bin so we don't have to modify our PATH. If using x86_64, then download from this link instead.

curl -LO https://launch.basespace.illumina.com/CLI/latest/arm-linux/bs
chmod +x bs
sudo mv bs /usr/local/bin/

Visit basespace.illumina.com and login. Create a free account if needed. Then visit basespace.illumina.com/s/htWXpgEKrRu6 and click "Accept" to add the demo WGS data to your account. Use "bs auth" to login before using the "bs" commands below.

Download the TruSeq PCR-free WGS FASTQs made by a NovaSeq X a GIAB sample (HG004).

mkdir ajtrio-mom
bs list dataset --terse --project-name "TruSeq-PCRfree-HG001-HG007-10B-2-v13" --is-type "common.fastq" --filter-term "TSPF-HG004-10B-2-v13-Rep1" | xargs -L1 bs contents dataset --terse --extension fastq.gz --id | xargs -L1 bs download file --no-metadata -o ajtrio-mom --id

Create an SAS token with upload permissions to our storage account and an expiry of 2 hours.

KEY=$(az storage account keys list --resource-group $RES_GROUP --account-name $STOR_ACCT --query [0].value --output tsv)
END=$(date -u -d "2 hours" '+%Y-%m-%dT%H:%MZ')
az storage container policy create --account-name $STOR_ACCT --container-name $CONTAINER --account-key $KEY --name upload --permissions cw --expiry $END
SAS=$(az storage container generate-sas --account-name $STOR_ACCT --name $CONTAINER --account-key $KEY --policy-name upload --https-only -o tsv)

Upload the FASTQs to our blob storage account and then create a fastq_list.csv using blob URIs.

azcopy cp "ajtrio-mom/*.fastq.gz" "https://$STOR_ACCT.blob.core.windows.net/$CONTAINER/ajtrio-mom?$SAS" --metadata="SM=ajtrio-mom" --content-type="text/fastq" --content-encoding="gzip"
echo "RGID,RGSM,RGLB,Lane,Read1File,Read2File" > ajtrio-mom_fastq_list.csv
az storage blob list --account-name $STOR_ACCT --account-key $KEY -c $CONTAINER --prefix ajtrio-mom/ --query [].name --output tsv | sed "s/^/https:\/\/${STOR_ACCT}.blob.core.windows.net\/${CONTAINER}\//" | paste -sd ',\n' | awk '{print "22TGVWLT3."NR",ajtrio-mom,UnknownLibrary,"NR","$0}' >> ajtrio-mom_fastq_list.csv

Test

Start a Dragen 4.3.6 VM on the private subnet and NSG created earlier.

SUBNET_ID=$(az network vnet subnet show --resource-group $RES_GROUP --vnet-name $VNET --name $SUBNET --query id --output tsv)
NSG_ID=$(az network nsg show --resource-group $RES_GROUP --name $NS_GROUP --query id --output tsv)
VM_IP=$(az vm create --resource-group $RES_GROUP --subnet $SUBNET_ID --nsg $NSG_ID --name dgn01 --size Standard_NP10s --ephemeral-os-disk true --ephemeral-os-disk-placement ResourceDisk --nic-delete-option delete --public-ip-address "" --enable-agent false --image illuminainc1586452220102:dragen-vm-payg:dragen-4-3-6-payg:latest --plan-publisher illuminainc1586452220102 --plan-product dragen-vm-payg --plan-name dragen-4-3-6-payg --accept-term --admin-username $USER --ssh-key-values ~/.ssh/id_ed25519.pub --query privateIpAddress --output tsv)

Copy the fastq_list.csv into the VM and then SSH into it.

SSH_OPTIONS="-q -i ~/.ssh/id_ed25519.pub -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=QUIET -o ServerAliveInterval=120 -o ServerAliveCountMax=30"
scp $SSH_OPTIONS ajtrio-mom_fastq_list.csv $VM_IP:~/
LANG='en_US.UTF-8' ssh $SSH_OPTIONS $VM_IP

Create some folders in the large local disk and download and untar the hg38 reference data.

sudo chown -R $USER:$GROUPS /mnt
mkdir -p /mnt/{ref,tmp,out}
wget -P /mnt/ref https://webdata.illumina.com/downloads/software/dragen/resource-files/hg38-alt_masked.cnv.graph.hla.rna-10-r4.0-1.tar.gz
tar -zxf /mnt/ref/hg38-alt_masked.cnv.graph.hla.rna-10-r4.0-1.tar.gz -C /mnt/ref

Set bash variables STOR_ACCT and KEY to your storage account name and account key respectively. And then pass those into Dragen so that it can stream the FASTQs.

AZ_ACCOUNT_NAME=$STOR_ACCT AZ_ACCOUNT_KEY=$KEY dragen --intermediate-results-dir /mnt/tmp --ref-dir /mnt/ref --enable-map-align true --enable-map-align-output true --output-format CRAM --enable-duplicate-marking true --generate-sa-tags true --enable-sort true --soft-read-trimmers polyg,quality --trim-min-quality 2 --qc-coverage-ignore-overlaps true --fastq-list ajtrio-mom_fastq_list.csv --fastq-list-sample-id ajtrio-mom --output-directory /mnt/out --output-file-prefix ajtrio-mom

Roughly 3 minutes into the runtime of the command above, there will be a 4 in 5 chance of a double free or corruption (fasttop) error.

Delete the VM to save money.

az vm delete --yes --resource-group $RES_GROUP --name dgn01

ckandoth/dgn_np10_errors.md

Purpose

Update (Mar 2025)

Prerequisites

Data

Test