get all URLs of all network requests made on a web page from the Chrome Dev Tools (from an exported HAR)

HOWTO.md

Setup

Using Homebrew on Mac OS X:

brew install jq

Add these aliases to your profile (e.g., ~/.zshrc, ~/.bashrc, ~/.profile, etc.):

alias hurlp='pbpaste | jq ".log.entries" | tee >(jq --raw-output "[.[] | .request.url] | sort | unique | .[]")'
alias hurld='pbpaste | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" | harurls | tee >(xargs -n 1 curl -O $1)'

Usage

1. Open the Chrome Developer Tools (Command+Option+i)
2. Click the Network tab
3. Reload the page (Command+r)
4. In the Network tab, hover over any area in the table that is not a link (e.g., the Status column)
5. Right click (Control+click) to open the context menu and click Copy All as HAR
6. Open your favourite command-line interface (e.g., Terminal)
7. Enter hurlp to print the URLs or hurld to download the files

Copy and paste HAR to a text file.

Let's assume you call this file requests.json

Then run the following on your terminal:

cat requests.json | jq '.log.entries|.[].request.url'

This assumes you have jq installed on your system

Thanks smartexpert, I followed your instructions and I have jq installed. It returned a lot of data all in green, but then where are my results now? I paste the clipboard and still have your cat entry there. Thanks again.

@Sifasol - you probably need to append something like | pbcopy to the end of hurlp to place the results back on the clipboard; the commands above just display the results (for hurlp) or download their contents as individual files (for hurld).

For a more universal solution (that works on Linux, et. al.), try:

1. Use the Network tab of Chrome/Edge/etc. Developer Tools as described above, except use the 'Save all as HAR with content' option to save it as a .har file.

2. Run one of the following from the directory where you saved that file, replacing requests.json with the filename you used when saving that .har file (note: these were tested in the bash shell):
   a. View the results (same results as 'hurlp' above):

   • cat requests.json | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" | less

   b. Save the results to a text file:

   • cat requests.json | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" >> sorted-urls-from-requests_$(date +%Y-%m-%d_%H:%M).txt

what's harurls ?