Created
October 15, 2019 06:03
-
-
Save harisec/bff0ac7ad2e90da7e82297d945c63ef1 to your computer and use it in GitHub Desktop.
filter links
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| An idea for a tool. | |
| So, going through Wayback archive we get thousands of links like the ones below. | |
| It's the same path and different values for GET parameters. | |
| A very useful tool would filter these using this algo: | |
| - for each link, parse the link with an URL parser: | |
| - create a hash from: hostname + path + (parameter names (without values) sorted). | |
| - save the hash in an array | |
| - for the next links, compute the hash | |
| - if the hash is in the array, ignore the link | |
| - if not pass it through | |
| The links: | |
| http://answers.yahoo.com/question/?qid=20110416092517AAbELYv | |
| http://answers.yahoo.com/question/?qid=20110420134930AAu4umN | |
| http://answers.yahoo.com/question/?qid=20110421192556AAhgq4D | |
| http://answers.yahoo.com/question/?qid=20110423171909AAg40TX | |
| http://answers.yahoo.com/question/?qid=20110427120319AAIOx5N | |
| http://answers.yahoo.com/question/?qid=20110427151942AAtZbxo | |
| http://answers.yahoo.com/question/?qid=20110429151912AAlrIHT | |
| http://answers.yahoo.com/question/?qid=20110510150840AAf6A4r | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420805&b=0&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420809&b=1020&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420809&b=1330&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420812&b=1330&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420815&b=1541&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420817&b=1751&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420820&b=1550&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420825&b=1291&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420828&b=40&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420831&b=540&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420833&b=430&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420834&b=1700&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420836&b=1211&keyword=&byto=title | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420841&b=1180&keyword=&byto=title | |
| From these links only 2 should pass: | |
| http://answers.yahoo.com/question/?qid=20110416092517AAbELYv | |
| http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420805&b=0&keyword=&byto=title |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
i made it with easier way by renaming every parameter value to "FUZZ" with sed then sort unique the output gives me non repeatable URLs which i could inject any payload on it