Skip to content

Instantly share code, notes, and snippets.

@joelrebel
Last active December 21, 2015 07:29
Show Gist options
  • Select an option

  • Save joelrebel/6271290 to your computer and use it in GitHub Desktop.

Select an option

Save joelrebel/6271290 to your computer and use it in GitHub Desktop.
Git -> s3 script
#!/usr/bin/python
#Script to sync git repo with s3, only changed files are uploaded
#Depends on the s3cmd utility, runs git diff ONLY with the previous repo HEAD.
#Include script into the git repo root directory, ensure it is executable
#
#To add a convienient alias defined the below in .git/config
#[alias]
#push-s3 = "!$(pwd)/s3-sync.py"
#push-both = "!git push && git push-s3"
#
#To use with alias,
#git push-s3 OR git push-both
import subprocess
git="/usr/bin/git diff HEAD~1 HEAD --name-status"
s3cmd="/usr/local/bin/s3cmd"
acl='--acl-public'
s3bucket='s3://some_bucket/' #trailing slash matters.
#File status flags:
#M modified - File has been modified
#C copy-edit - File has been copied and modified
#R rename-edit - File has been renamed and modified
#A added - File has been added
#D deleted - File has been deleted
#U unmerged - File has conflicts after a merge
git_action = { 'M': 'sync', 'A': 'sync', 'D': 'del' } #as of now we just care about added, modified, deleted files
def s3cmd_run(cmd):
cmd = subprocess.Popen(cmd, stdout=subprocess.PIPE)
out = cmd.communicate()[0]
print out
# s3cmd sync /tmp/testfile s3://bucket/
# s3cmd del s3://bucket/testfile
c = subprocess.Popen(git.split(), stdout=subprocess.PIPE)
s = c.communicate()[0]
files = s.split('\n')
files = filter(None, files)
for f in files:
#print f
(action, filename) = f.split()
s3bucket_path = s3bucket + filename
if action in git_action:
if action == 'M':
print "adding modified file " + filename
s3cmd_run([s3cmd, acl, git_action['M'], filename, s3bucket_path])
if action == 'A':
print "adding new file " + filename
s3cmd_run([s3cmd, acl, git_action['A'], filename, s3bucket_path])
#This action is too risky, a filename like '*' will wipe out all files in the bucket.
# if action == 'D':
# target = s3bucket+filename
# print "removing remote file " + filename
# s3cmd_run([s3cmd, git_action['D'], target])
else:
print "Unknown action -> " + action + " on file " + filename
print "This script needs to be updated for this action"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment