Load all urls from a sitemap.xml file

load_all_sitemap_urls.sh

#!/bin/bash
# This script crawls all urls in a /sitemap.xml file and loads them, effectively priming the cache
# Usage: ./warm_cache.sh www.example.com

time wget --quiet https://$1/sitemap.xml --output-document - | \
    egrep -o "https?://[^<]+" | \
    grep $1 | \
    grep -v "jpg" | \
    xargs -i -d '\n' curl --output /dev/null --silent --write-out '%{http_code} %{time_total}ms %{url_effective} \n' {}

While caching shouldn't solely be relied upon for site performance, there are certain instances where priming the cache is useful. In my case, a client regularly imports products with hundreds of images each (for 360° views) and when the PDP is loaded for the first time, it takes 1-2 minutes for Magento to generate the product images. In my case, the php bin/magento catalog:images:resize command does not generate the images necessary since the 360° images are powered by a third-party extension and don't tie into that command.

Example output of this script:

200 0.263ms https://example.com/example-url-1.html
200 0.263ms https://example.com/example-url-2.html
200 113.260ms https://example.com/example-url-3.html
400 0.257ms https://example.com/example-url-broken-url.html