filter links

linkfilter.txt

An idea for a tool. 
So, going through Wayback archive we get thousands of links like the ones below.
It's the same path and different values for GET parameters.

A very useful tool would filter these using this algo:
- for each link, parse the link with an URL parser:
- create a hash from: hostname + path + (parameter names (without values) sorted). 
- save the hash in an array
- for the next links, compute the hash
- if the hash is in the array, ignore the link
- if not pass it through

The links:
http://answers.yahoo.com/question/?qid=20110416092517AAbELYv
http://answers.yahoo.com/question/?qid=20110420134930AAu4umN
http://answers.yahoo.com/question/?qid=20110421192556AAhgq4D
http://answers.yahoo.com/question/?qid=20110423171909AAg40TX
http://answers.yahoo.com/question/?qid=20110427120319AAIOx5N
http://answers.yahoo.com/question/?qid=20110427151942AAtZbxo
http://answers.yahoo.com/question/?qid=20110429151912AAlrIHT
http://answers.yahoo.com/question/?qid=20110510150840AAf6A4r
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420805&b=0&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420809&b=1020&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420809&b=1330&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420812&b=1330&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420815&b=1541&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420817&b=1751&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420820&b=1550&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420825&b=1291&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420828&b=40&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420831&b=540&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420833&b=430&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420834&b=1700&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420836&b=1211&keyword=&byto=title
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420841&b=1180&keyword=&byto=title

From these links only 2 should pass:
http://answers.yahoo.com/question/?qid=20110416092517AAbELYv
http://kr.maple.auto.yahoo.com/magazine/view.html?bbs_code=1&cate_code=14&no=11420805&b=0&keyword=&byto=title

Use wsort.py. :)

i made it with easier way by renaming every parameter value to "FUZZ" with sed then sort unique the output gives me non repeatable URLs which i could inject any payload on it