Skip to content

Instantly share code, notes, and snippets.

@holdenk
Created April 10, 2018 18:00
Show Gist options
  • Save holdenk/8119ddc7bd90407e0b184df7e958933c to your computer and use it in GitHub Desktop.
Save holdenk/8119ddc7bd90407e0b184df7e958933c to your computer and use it in GitHub Desktop.
Run Beam Go Demo
#!/bin/bash
export TESTBUCKET=YOURTESTBUCKETGOESHEREEH
set -x
echo "Starting job server"
java -cp runners/flink/build/libs/flink-deploy.jar org.apache.beam.runners.flink.FlinkJobServerDriver --job-host localhost:3000 &> logs &
echo $! &> jobserver.pid
echo "Making the output bucket world writeable, because fuck credentials are hard"
gsutil acl ch -u AllUsers:W gs://$TESTBUCKET
pushd ./sdks/go/examples/build/bin/linux_amd64/
./wordcount --input gs://apache-beam-samples/shakespeare/kinglear.txt --output gs://$TESTBUCKET/output.txt --runner=flink --endpoint=localhost:3000 --worker_binary ./wordcount
popd
gsutil acl ch -d AllUsers gs://$TESTBUCKET
echo "Stopping job server"
kill `cat jobserver.pid`
echo "Reading word count"
gsutil cat gs://$TESTBUCKET/output.txt
set +x
sudo service docker restart
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment