Skip to content

Instantly share code, notes, and snippets.

@RooseveltAdvisors

RooseveltAdvisors/hadoop copy from s3

Created April 11, 2017 15:43
Show Gist options
  • Save RooseveltAdvisors/b906fc56da60530a76a7a9e82ad9fd78 to your computer and use it in GitHub Desktop.
Save RooseveltAdvisors/b906fc56da60530a76a7a9e82ad9fd78 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
hadoop copy from s3
hadoop distcp -Dmapreduce.map.memory.mb=4096 -Dfs.s3a.awsAccessKeyId=XXX -Dfs.s3a.awsSecretAccessKey=XXXX -m 250 hdfs:///data/* s3a://api-v3-data-sources/output/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment