Skip to content

Instantly share code, notes, and snippets.

@GEOFBOT

GEOFBOT/EMR_flinkerations.sh

Last active January 6, 2017 22:24
Show Gist options
  • Save GEOFBOT/2572c6eb6863cbfa0b21643a9261bbe8 to your computer and use it in GitHub Desktop.
Save GEOFBOT/2572c6eb6863cbfa0b21643a9261bbe8 to your computer and use it in GitHub Desktop.
Download ZIP
Bootstrap script to update Flink on Amazon EMR to use my build with Python bulk iterations
Raw
EMR_flinkerations.sh
#!/bin/bash
# Runs after installation of included Flink
set -e
cd ~
sudo rm /usr/lib/flink/lib/flink-* # make sure we don't have two versions of jars
sudo tar -xzf flinkerations-emr.tgz -C /usr/lib/
rm flinkerations-emr.tgz
# Copy over EMRFS jars to Flink lib path
sudo cp /usr/share/aws/emr/s3-dist-cp/lib/*.jar /usr/lib/flink/lib/
# Copy Java R1DL jar
aws s3 cp s3://r1dl/script/flink-r1dl-1.0-SNAPSHOT.jar /home/hadoop/
Raw
EMR_flinkerations_pre.sh
#!/bin/bash
# Gets what we need to set up Flink
set -e
cd ~
wget -S -T 10 -t 5 https://github.com/GEOFBOT/flink/releases/download/iterations-working/flinkerations-emr.tgz
aws s3 cp s3://r1dl/script/EMR_flinkerations.sh /tmp/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment