Downloading genome annotation files from NCBI ftp with given FTP URL list

URL list

$ head choose_ftp.txt
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/015/405/GCA_000015405.1_ASM1540v1
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/620/625/GCA_000620625.1_ASM62062v1
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/972/925/GCA_000972925.1_ASM97292v1
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/001/021/385/GCA_001021385.1_ASM102138v1
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/328/565/GCA_000328565.1_ASM32856v1

Target files

genome: *_genomic.fna.gz.
protein: *_protein.faa.gz, may be missing.
gff: *_genomic.gff.gz.
genbank: *_genomic.gbff.gz.

Downloading using wget and rush or GNU parallel in parallel

# rush
$ cat choose_ftp.txt | rush -j 6 -c --verbose \
    'mkdir -p {%}; cd {%}; \
    for s in _genomic.fna.gz _protein.faa.gz _genomic.gff.gz _genomic.gbff.gz; do \
    wget -c -q {}/{%}$s; done'

# parallel
$ cat choose_ftp.txt | parallel -j 1 --joblog log --verbose \
    'mkdir -p {/}; cd {/}; for s in _genomic.fna.gz _protein.faa.gz _genomic.gff.gz _genomic.gbff.gz; do wget -c -q {}/{/}$s; done'

shenwei356/Downloading genome annotation files from NCBI ftp with given FTP URL list.md