Skip to content

Instantly share code, notes, and snippets.

@paulmwatson

paulmwatson/cdxt_commoncrawl_example_domain.sh

Created May 25, 2021 10:46
Show Gist options
  • Save paulmwatson/7a1b30312f3643b0ba4ab15833735835 to your computer and use it in GitHub Desktop.
Save paulmwatson/7a1b30312f3643b0ba4ab15833735835 to your computer and use it in GitHub Desktop.
Download ZIP
Use CDXT Toolkit to search CommonCrawl for certain domains/URLs
Raw
cdxt_commoncrawl_example_domain.sh
cdxt --cc --limit 100 --filter '=status:200' iter 'github.com/*'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment