Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save objectoriented/542408 to your computer and use it in GitHub Desktop.
Save objectoriented/542408 to your computer and use it in GitHub Desktop.
Ensure complete backup or copy of an Amazon S3 bucket (compare file lists on S3 vs your filesystem)
#!/bin/sh
#
# Diff file lists to find missing files on local copy
#
s3cmd ls -r s3://<<..S3 BUCKET..>>/|awk '{ print $4 }' | awk '{sub(/^s3\:\/\/<<..S3 BUCKET..>>\//, "")};1' | sort > /tmp/s3-file-list.txt.sorted
find . -type f | awk '{sub(/\.\//, "")};1' | sort > /tmp/my-file-list.txt.sorted
diff /tmp/s3-file-list.txt.sorted /tmp/my-file-list.txt.sorted
#
# Compare file counts
#
find . -type f|wc
s3cmd ls -r s3://<<..S3 BUCKET..>>/|wc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment