gsutil commands

gistfile1.sh

The gsutil rsync command makes the contents under dst_url the same as the contents under src_url, by copying any missing files/objects, and (if the -d option is specified) deleting any extra files/objects. For example, to make gs://mybucket/data match the contents of the local directory “data” you could do:

gsutil rsync -d data gs://mybucket/data

To recurse into directories use the -r option:

gsutil rsync -d -r data gs://mybucket/data

To copy only new/changed files without deleting extra files from gs://mybucket/data leave off the -d option:

gsutil rsync -r data gs://mybucket/data

If you have a large number of objects to synchronize you might want to use the gsutil -m option, to perform parallel (multi-threaded/multi-processing) synchronization:

gsutil -m rsync -d -r data gs://mybucket/data

The -m option typically will provide a large performance boost if either the source or destination (or both) is a cloud URL. If both source and destination are file URLs the -m option will typically thrash the disk and slow synchronization down.

To make the local directory “data” the same as the contents of gs://mybucket/data:

gsutil rsync -d -r gs://mybucket/data data

To make the contents of gs://mybucket2 the same as gs://mybucket1:

gsutil rsync -d -r gs://mybucket1 gs://mybucket2

You can also mirror data across local directories:

gsutil rsync -d -r dir1 dir2

To mirror your content across clouds:

gsutil rsync -d -r gs://my-gs-bucket s3://my-s3-bucket