Backup mongodb database directly to S3

mongodb-backup.sh

# Use environment variable as password (could also be a password stored in a file, read with cat)
# DO NOT use --password argument in mongodump command or authentication will fail
# DO NOT add a file name after --archive argument, or mongodump output won't be sent to stdout
# Use S3 infrequent access storage class to reduce cost
# Use aes-256 encryption of your file on S3 (or your own KMS keys)

# Benefits of this method :
# 1 > You won't see the password sent to mongodump using in ps command (it's safe in a multi-user environment)
# 2 > You don't have to store the backup locally before sending it to S3
# 3 > Everything is done in one line, fits a cronjob easily
# 4 > You can do the "reverse" command and use mongorestore from stdin and awscli to read from S3 !

(echo $BACKUP_PASSWORD) \
  | mongodump --host localhost --port 27017 -u backupUser --authenticationDatabase users --archive --gzip --db mydatabase \
  | aws s3 cp - s3://mybucket/mydatabase/mydatabase-backup-file.agz --storage-class STANDARD_IA --sse --profile s3uploader
  

So this worked for me for the restore process:
But it will need to be run from the Primary DB instance only (i tried on a secondary instance but it will not work with error Failed: EOF)

#Download backup from S3
aws s3 cp s3://BucketName/$DBNAME.agz /tmp

(echo $BACKUP_PASSWORD) | mongorestore --host $HOST --port 27017 --authenticationDatabase admin -u admin --gzip --archive=/tmp/$DBNAME.agz --nsInclude "$DBNAME.*"

Also, it will need to be run twice if the database does not exist before. Don't know if there could be a walk around so it just runs once.

Thank you so much for the help.

Does not work:

Failed: archive writer: error writing data for collection `blah.blah` to disk: short write / write /dev/stdout: broken pipe

same error as @talaikis anyoone could help

Hi @mehdibenfeguir, could you give more information ??

Script? Full error output? Environment?

Maybe is some kind of error with the Mongodump or mongoDB versions.. Any additional info would be great to help you!

Regards :)