How to query and download from archive.org using command line

Search for all binaries from a specific URL

$ wget http://web.archive.org/cdx/search/cdx?url=dlm3.download.intuit.com/SBD/QuickBooks/* -q -O - | grep octet-stream

com,intuit,download,dlm3)/sbd/quickbooks/2012/us_r9/quickbookspremier2012.exe 20210224173151 http://dlm3.download.intuit.com/SBD/QuickBooks/2012/US_R9/QuickBooksPremier2012.exe application/octet-stream 200 HJYXHXCU33SITK476OBWSAMJSY7USDIX 483202952
com,intuit,download,dlm3)/sbd/quickbooks/2012/us_r9/quickbookspremier2012.exe 20210224173151 http://dlm3.download.intuit.com/SBD/QuickBooks/2012/US_R9/QuickBooksPremier2012.exe application/octet-stream 200 HJYXHXCU33SITK476OBWSAMJSY7USDIX 483202952

Download artifact from specific URL

Syntax:

https://web.archive.org/web/<snapshot-date>if_/<original-site-URL>

wget https://web.archive.org/web/20210307215842if_/https://dlm3.download.intuit.com/SBD/QuickBooks/2012/US_R1/QuickBooksPro2012.exe

Extract download links from a web page and check for snapshot

http://www.tucows.com/preview/391885 leads to: http://www.tucows.com/download.html?software_id=391885&t=2

Snapshots of download/mirror link page (mirror list changed between years):

https://web.archive.org/web/20080204163204/http://www.tucows.com/download.html?software_id=391885&t=2

https://web.archive.org/web/20150915004243/http://www.tucows.com/download.html?software_id=391885&t=2

One-liner to extract the original mirror links from web page and use archive.org CDX/C API to query for any existing snapshots:

$ wget -qO- "https://web.archive.org/web/20080204163204/http://www.tucows.com/download.html?software_id=391885&t=2" | \
   sed -n "s/.*dl('\([^\"]*\)',.*/\1/p" | \
   sed -e 's/.*http:\/\/\(.*\)/\1/' | \
   xargs -I {} -exec sh -c 'echo "Checking: {}" ; \
   wget http://web.archive.org/cdx/search/cdx?url="{}" -q -O - | \
   grep application/ ; sleep 10' \;

Checking: tucows.mts.net/files5/HallmarkTrialSetup.exe
Checking: tucows.netnitco.net/files5/HallmarkTrialSetup.exe
Checking: quantum.tucows.com/files5/HallmarkTrialSetup.exe
Checking: theplanet.tucows.com/files5/HallmarkTrialSetup.exe
Checking: tds.tucows.com/files5/HallmarkTrialSetup.exe
Checking: chariot.tucows.com/files5/HallmarkTrialSetup.exe
Checking: filearena.tucows.com/files5/HallmarkTrialSetup.exe
Checking: overflow.tucows.com/files5/HallmarkTrialSetup.exe
...
Checking: dataphone.tucows.com/files5/HallmarkTrialSetup.exe
Checking: port80.tucows.com/files5/HallmarkTrialSetup.exe
Checking: tucows.archive.sunet.se/files5/HallmarkTrialSetup.exe
Checking: genotec.tucows.com/files5/HallmarkTrialSetup.exe
Checking: ows.tucows.com/files5/HallmarkTrialSetup.exe
Checking: benson.tucows.com/files5/HallmarkTrialSetup.exe
Checking: ankara.tucows.com/files5/HallmarkTrialSetup.exe
Checking: tucows.ulak.net.tr/files5/HallmarkTrialSetup.exe
Checking: tr-net.tucows.com/files5/HallmarkTrialSetup.exe
Checking: adanet.tucows.com/files5/HallmarkTrialSetup.exe
Checking: tucows.alaska.net/files5/HallmarkTrialSetup.exe
Checking: aspect1.tucows.com/files5/HallmarkTrialSetup.exe
Checking: tucows.nyi.net/files5/HallmarkTrialSetup.exe
Checking: epix.tucows.com/files5/HallmarkTrialSetup.exe
Checking: tucows.whc.net/files5/HallmarkTrialSetup.exe
Checking: tucows.rmplc.co.uk/files5/HallmarkTrialSetup.exe
Checking: ukms.tucows.com/files5/HallmarkTrialSetup.exe
Checking: blueyonder.tucows.com/files5/HallmarkTrialSetup.exe
Checking: zol.tucows.com/files5/HallmarkTrialSetup.exe

Tidbit: Tucows on archive.org

Links

rmi1974/how_to_search_and_download_from_archive_org.md

Select an option