Skip to content

Instantly share code, notes, and snippets.

@manboubird

manboubird/hql_parallel.txt

Last active December 22, 2015 15:39
Show Gist options
  • Save manboubird/6493938 to your computer and use it in GitHub Desktop.
Save manboubird/6493938 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
hql_parallel.txt
# http://www.gnu.org/software/parallel/parallel_tutorial.html#controling_the_output
# issue hive queries
cat hqls.hql | parallel --line-buffer --tagstring "[ {#} - {} ]" --results parallel_std_logs --eta --progress --joblog parallel_job_logs -j0 --halt 1 --load 100% --noswap "hive -e {} > result_{#}_{}.txt"
# dry run
cat hqls.hql | parallel --dry-run --line-buffer --tagstring "[ {#} - {} ]" --results parallel_std_logs --eta --progress --joblog parallel_job_logs -j0 --halt 1 --load 100% --noswap "hive -e {} > result_{#}_{}.txt"
# transfer jar and submit job
/usr/local/bin/parallel -S hadoop@localhost --wd ... --transfer "/usr/local/hadoop-mr1/bin/hadoop jar {} scalding.marketing.JobRunner scalding.marketing.UniqueUser --hdfs --output /tmp/output --dt 2000-02-08" ::: ./target/scalding-marketing-0.0.1.jar
# with cat
/usr/local/bin/parallel -S hadoop@localhost --wd ... --transfer "/usr/local/hadoop-mr1/bin/hadoop jar {} scalding.marketing.JobRunner scalding.marketing.UniqueUser --hdfs --output /tmp/output --dt 2000-02-08; /usr/local/hadoop-mr1/bin/hadoop fs -cat '/tmp/output/part-*'" ::: ./target/scalding-marketing-0.0.1.jar
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment