Skip to content

Instantly share code, notes, and snippets.

@atomicstack

atomicstack/dupe-remover.sh

Created March 5, 2022 22:25
Show Gist options
  • Save atomicstack/93bb6f48dc5b0dbccaf14a840d16ddbe to your computer and use it in GitHub Desktop.
Save atomicstack/93bb6f48dc5b0dbccaf14a840d16ddbe to your computer and use it in GitHub Desktop.
Download ZIP
removes files with duplicate SHA-256 hashes. tweaked enhancement of https://gist.github.com/atomicstack/2c5501f6210b6a303b626f447d50d487
Raw
dupe-remover.sh
find . -type f -print0 | xargs -0 sha256sum | pv -l -s $( find . -type f | wc -l ) | sponge | perl -naE 'my $hash = shift @F; $seen{$hash}++ or next; $ENV{UNLINK_FILES} and unlink "@F"; my $result = $! ? "failed" : ""; say qq{unlinked @F $result}'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment