Skip to content

Instantly share code, notes, and snippets.

@jasonm23
Created June 5, 2010 01:08
Show Gist options
  • Save jasonm23/426168 to your computer and use it in GitHub Desktop.
Save jasonm23/426168 to your computer and use it in GitHub Desktop.
#!/bin/bash
# read a list of urls and find code resource related keywords.
# url list via stdin.
while read a; do
echo @@@$a
curl -s $a | grep -E 'svn|swc|zip|git|download|source'
echo @@@END
done
# Cygwin variety... (we're pulling from the adrian parr CSV hence read a b c (and the sed / tr to cleanup the input))
cat /dev/clipboard | sed 's/",/ @/g' | tr -d '"' | while IFS=@ read a b c;
do
echo "@@@${a}::curl ${c}";
curl -s $c | grep -E 'svn|git|download|source|zip|swc|tar'; echo "@@@END@@@";
done > scrape.txt
# grab svn url's for libspark.org
pbpaste | grep -o '/svn/as3/.*/"' | tr -d '"' | cut -d '/' -f-4 | sort | uniq | pbcopy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment