get all URLs of all network requests made on a web page from the Chrome Dev Tools (from an exported HAR)

HOWTO.md

Setup

Using Homebrew on Mac OS X:

brew install jq

Add these aliases to your profile (e.g., ~/.zshrc, ~/.bashrc, ~/.profile, etc.):

alias hurlp='pbpaste | jq ".log.entries" | tee >(jq --raw-output "[.[] | .request.url] | sort | unique | .[]")'
alias hurld='pbpaste | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" | harurls | tee >(xargs -n 1 curl -O $1)'

Usage

1. Open the Chrome Developer Tools (Command+Option+i)
2. Click the Network tab
3. Reload the page (Command+r)
4. In the Network tab, hover over any area in the table that is not a link (e.g., the Status column)
5. Right click (Control+click) to open the context menu and click Copy All as HAR
6. Open your favourite command-line interface (e.g., Terminal)
7. Enter hurlp to print the URLs or hurld to download the files

Here is my solution:
....
4. Paste results to anywhere you want

The result contains all urls from source page. How to get only urls from filtered url list in network tab?

Anyway my HAR is too large to be processed.

thanks, @z333d for the solution.

Anyway my HAR is too large to be processed.

+1
Same problem for me @z333d Could you please provide any solution?

Copy and paste HAR to a text file.

Let's assume you call this file requests.json

Then run the following on your terminal:

cat requests.json | jq '.log.entries|.[].request.url'

This assumes you have jq installed on your system

Thanks smartexpert, I followed your instructions and I have jq installed. It returned a lot of data all in green, but then where are my results now? I paste the clipboard and still have your cat entry there. Thanks again.

@Sifasol - you probably need to append something like | pbcopy to the end of hurlp to place the results back on the clipboard; the commands above just display the results (for hurlp) or download their contents as individual files (for hurld).

For a more universal solution (that works on Linux, et. al.), try:

1. Use the Network tab of Chrome/Edge/etc. Developer Tools as described above, except use the 'Save all as HAR with content' option to save it as a .har file.

2. Run one of the following from the directory where you saved that file, replacing requests.json with the filename you used when saving that .har file (note: these were tested in the bash shell):
   a. View the results (same results as 'hurlp' above):

   • cat requests.json | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" | less

   b. Save the results to a text file:

   • cat requests.json | jq ".log.entries" | jq --raw-output "[.[] | .request.url] | sort | unique | .[]" >> sorted-urls-from-requests_$(date +%Y-%m-%d_%H:%M).txt

what's harurls ?