Skip to content

Instantly share code, notes, and snippets.

@catichenor
Created February 20, 2018 21:50
Show Gist options
  • Save catichenor/205d2c2f7d488db2d8f6bda68539757f to your computer and use it in GitHub Desktop.
Save catichenor/205d2c2f7d488db2d8f6bda68539757f to your computer and use it in GitHub Desktop.
Copy a large list of sequential files to GCS
for i in `seq -f '%06g' 1 10`; do echo "/path/to/file.${i}.png"; done | xargs > ./filelist.txt
# ^Generate a space-delimited list of filenames and output it to `filelist.txt`.
# Contents of `filelist.txt`:
# /path/to/file.000001.png /path/to/file.000002.png /path/to/file.000003.png /path/to/file.000004.png /path/to/file.000005.png /path/to/file.000006.png /path/to/file.000007.png /path/to/file.000008.png /path/to/file.000009.png /path/to/file.000010.png
gsutil cp -m `cat ./filelist.txt` gs://png-bucket/path/to/receptacle/
# ^Read the list of files as an argument to gsutil and use multithreaded copy to copy to the bucket.
@catichenor
Copy link
Author

catichenor commented Feb 21, 2018

Maybe can be written as:

gsutil cp -m $(for i in `seq -f '%06g' 1 10`; do echo "/path/to/file.${i}.png"; done | xargs) gs://png-bucket/path/to/receptacle/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment