One-liner for crawling domain list of sub-domains << extract urls with parameters << filter the result to only 4 urls for each endpoint >> filter urls to delete unwanted extensions.

One liner for gathering urls.

cat sub-domains.txt | hakrawler | grep 'http' | cut -d ' ' -f 2  > crawling.txt && gau -subs http://domain.com >> crawling.txt && waybackurls http://domain.com >> crawling.txt && cat crawling.txt | grep "?" | unfurl --unique format %s://%d%p > base.txt ; cat base.txt | parallel -j 4 grep {} -m5 crawling.txt | tee final1.txt; cat final1.txt | egrep -iv ".(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico|pdf|svg|txt|js)" > final.txt && rm -rf base.txt final1.txt

I think it is more useful as below.
domain=testfire.net && cat sub-domains.txt | hakrawler | grep 'http' | cut -d ' ' -f 2 > crawling.txt && gau -subs $domain >> crawling.txt && waybackurls $domain >> crawling.txt && cat crawling.txt | grep "?" | unfurl --unique format %s://%d%p > base.txt ; cat base.txt | parallel -j 4 grep {} -m5 crawling.txt | tee final1.txt; cat final1.txt | egrep -iv ".(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico|pdf|svg|txt|js)" > final.txt && rm -rf base.txt final1.txt

nice

Getting error like that
unknown shorthand flag: 's' in -subs

edit: --subs