The following bash one-liners are useful commands for manipulating wordlists (or any text file). For instance, if you need to remove all blank lines from a file, a one-liner will do the trick. Similarly, if you need to remove duplicate passwords (or text), you can do that too. If you have multiple wordlists, you can also combine them into one large file.
- Remove duplicates
awk '!(count[$0]++)' old.txt > new.txt
- Sort wordlist by length
awk '{print length, $0}' old.txt | sort -n | cut -d " " -f2- > new.txt
- Sort by alphabetical order
sort old.txt | uniq > new.txt
- Merge multiple text files into one
cat file1.txt file2.txt > combined.txt
- Remove all blank lines
egrep -v "^[[:space:]]*$" old.txt > new.txt