Skip to content

Instantly share code, notes, and snippets.

@carlosmcevilly
Created December 27, 2012 00:25
Show Gist options
  • Save carlosmcevilly/4384324 to your computer and use it in GitHub Desktop.
Save carlosmcevilly/4384324 to your computer and use it in GitHub Desktop.
fetch a URL using wget and sending Firefox as the user agent, with an optional referer URL
#!/bin/bash
export url="$1"
export referer="$2"
export uagent='Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11';
if [[ "$referer" == '' ]]; then
wget --user-agent="$uagent" "$url"
else
wget --user-agent="$uagent" --referer "$referer" "$url"
fi
@shekkbuilder
Copy link

I thought this was referring to using firefox cookies with wget. I was looking for some examples on reading the moz_cookies sqlite table and found this.


# Example script to get content using Firefox cookies.
# by Jean-Sebastien Morisset (http://surniaulula.com/)

cookie_file="`echo $HOME/Library/Application\ Support/Firefox/Profiles/*.default/cookies.sqlite`"
user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:15.0) Gecko/20100101 Firefox/15.0.1"

echo ".mode tabs
select host, case when host glob '.*' then 'TRUE' else 'FALSE' end,
path, case when isSecure then 'TRUE' else 'FALSE' end, 
expiry, name, value from moz_cookies;" | \
    sqlite3 "$cookie_file" | \
        wget --load-cookies=/dev/stdin --user-agent="$user_agent" --output-document=/dev/stdout "$1" 2>/dev/null | \
            sed -n -e "/>/G" -e "s/^.*href=['\"]\([^\"]*\)['\"].*$/\1/p" | \
                while read line; do echo -e "$1\t$line"; done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment