README.md

Deploy Cumulus Config

This bash script is intended to demonstrate how to programmatically deploy config (collections, providers, rules) to cumulus via a proxy through the bastion host

Environment

Most of the ENV is in cumulus' app/.env file. However, there are some non-standard configs assumed:

AWS Env:

AWSENV="--profile=$AWSPROFILE --region=$AWS_DEFAULT_REGION"

URS U:P

AUTH=$(printf "$EARTHDATA_USERNAME:$EARTHDATA_PASSWORD" | base64)

Need to set EARTHDATA_USERNAME and EARTHDATA_PASSWORD or add them to your app/.env

Maturity level:

MATURITY=dev

This is whatever value you're using for your API Gateway stage: https://${API}.execute-api.${AWS_DEFAULT_REGION}.amazonaws.com/${MATURITY}

Stack Id:

STACKNAME=sirc-ingest-${MATURITY}

This should be whatever you call yuor stack in cloudformation

Config Files

This is just a directory structure of your config in native json format:

config/
├── collections
│   ├── SIRCGRD.json
│   ├── SIRCGRDMETADATA.json
│   ├── SIRCSLC.json
│   └── SIRCSLCMETADATA.json
├── providers
│   ├── ASFDAAC.json
│   └── WOS2.json
└── rules
    ├── MonitorSircQueueOneTime.json
    ├── SircAllInOneOneTime.json
    └── SircSns.json

where any json file just the value you want to send to cumulus:

config/providers/ASFDAAC.json:

{ 
 "id": "ASFDAAC",
 "protocol": "s3",
 "host": "s3.amazonaws.com"
}

Caveats:

URS:

Currently, the script only supports URS auth. This will need to be adapted for LaunchPad

Injecting Dynamic values into config:

Environment variables, in the format of $SOME_ENV in JSON files will be extrapolated.

Broken tunnels:

If the script failes, it can leave the SSH tunnel open. if you run it again, you may see this error:

bind: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 8001
Could not request local forwarding.

You can either ignore it, or kill your orphaned tunnel.

Known Hosts

Bastions are rotated ~ monthly. This Will cause know hosts files complain after a rotation. Just delete the offending line from ~/.ssh/known_hosts

I probably forgot some other config bit that is required:

I make no promises

deploy-cumulus-config.sh

# export vars for extrapolation
set -a

# until this is Jenkinized
source ../app/.env

# Check ENV
if [ -z "$MATURITY" ]; then echo "No MATURITY Provided"; exit 1; fi
if [ -z "$AWSPROFILE" ]; then echo "No AWSPROFILE Provided"; exit 1; fi
if [ -z "$AWS_DEFAULT_REGION" ]; then echo "No AWS_DEFAULT_REGION Provided"; exit 1; fi

# Grab account id... 
AWSENV="--profile=$AWSPROFILE --region=$AWS_DEFAULT_REGION"
ACCOUNTID=$(aws $AWSENV sts get-caller-identity --query "Account" --output=text)

# Stack Setup
STACKNAME=sirc-ingest-${MATURITY}
API=$(aws apigateway get-rest-apis --query "items[?name=='${STACKNAME}-backend'].id" --output=text $AWSENV)
CUMULUS_BASEURL=$(echo "https://${API}.execute-api.${AWS_DEFAULT_REGION}.amazonaws.com/${MATURITY}")
echo "API is $CUMULUS_BASEURL"
if [ ${#API} -le 4 ]; then echo "Could not figure out API AWS for $STACKNAME using profile $AWSPROFILE" ; exit 1; fi

# Grab the landing SNS:
SNSTOPIC=$(aws $AWSENV cloudformation describe-stacks --stack-name $STACKNAME --query "Stacks[0].Outputs[?OutputKey=='LandingTopic'].OutputValue" --output=text)
echo SNSTOPIC is $SNSTOPIC

# set up SSH tunnel through bastion to allow us to talk to private API Gateway
export BASTION=$(aws ec2 $AWSENV describe-instances --filters "Name=tag:Name,Values=NGAP SSH Bastion" --query "Reservations[0].Instances[0].NetworkInterfaces[0].Association.PublicDnsName" --output=text)
if [ ${#BASTION} -le 20 ]; then echo "Could not determine Bastion host"; exit 1; fi
ssh -fN -D localhost:8001 ec2-user@$BASTION
if [ ! $? -eq 0 ]; then echo "Could not establish proxy SSH connection"; exit 1; fi
SSH=$(pgrep -f 'ssh -fN -D localhost:8001')
export PROXY='--proxy socks5h://localhost:8001'

# Get the token URL
ORIGIN=$(dirname $CUMULUS_BASEURL)
LOGIN_URL="$CUMULUS_BASEURL/token"
BACKEND_URL="$CUMULUS_BASEURL/v1"

# create a base64 hash of your login credentials
AUTH=$(printf "$EARTHDATA_USERNAME:$EARTHDATA_PASSWORD" | base64)

# Request the Earthdata url with client id and redirect uri to use with Cumulus
echo ">>> Attempting auth @ ${LOGIN_URL}"
AUTHORIZE_URL=$(curl $PROXY -s -i ${LOGIN_URL} | grep location | sed -e "s/^location: //");
if [ -z "$AUTHORIZE_URL" ]; then echo "Could not contact Auth API; CHECK VPN!"; exit 1; fi

# Request an authorization grant code
TOKEN_URL=$(curl $PROXY -s -i -X POST \
  -F "credentials=${AUTH}" \
  -H "Origin: ${ORIGIN}" \
  ${AUTHORIZE_URL%$'\r'} | grep Location | sed -e "s/^Location: //")

# Validate there is a `code` redirect
if [[ ! $TOKEN_URL =~ "code=" ]]; then echo "Could not get token redirect, check URS App Redirect URI for $CUMULUS_BASEURL/token"; exit 1; fi

# Request the token through the CUMULUS API url that's returned from Earthdata
TOKEN=$(curl $PROXY -s ${TOKEN_URL%$'\r'} | sed 's/.*\"token\"\:\"\(.*\)\".*/\1/')
if [ ${#TOKEN} -le 10 ]; then echo "Could not get TOKEN for API Access" ; exit 1; fi
TH="--header 'Authorization: Bearer $TOKEN'"
echo ">>> Bearer token was: ${TOKEN:0:10}..."

for TYPE in providers collections rules; do 
    T_PATH=../config/$TYPE/
    echo ">>> Processing $TYPE from $T_PATH"
    for OBJECT in `ls -1 $T_PATH`; do 
        echo ">>> checking out ${TYPE}: $OBJECT"

        # Figure out the unique id         
        if [[ $TYPE =~ .*providers.* ]]; then
            ID=$(cat ${T_PATH}$OBJECT | grep '"id"' | sed 's/.*\"id\"\:[^\"]*\"\([^\"]*\)\".*/\1/')
        else
            ID=$(cat ${T_PATH}$OBJECT | grep '"name"' | sed 's/.*\"name\"\:[^\"]*\"\([^\"]*\)\".*/\1/' | head -1)
        fi

        if [[ $TYPE =~ .*collection.* ]]; then
            # need to know the Version for collections
            VERSION=$(cat ${T_PATH}$OBJECT | grep '"version"' | sed 's/.*\"version\"\:[^\"]*\"\([^\"]*\)\".*/\1/')
            ID="${ID}/${VERSION}"
        fi

        URL=$BACKEND_URL/$TYPE/$ID

        # Check if this Object exists
        echo "Checking if object exist.... "
        CREATED=$(eval $( echo curl $PROXY -s $TH ${URL} ) )

        echo " >>> Created value was $CREATED"

        if [[ $CREATED =~ .*createdAt.* ]]; then
            # Do an update
            URL=$BACKEND_URL/$TYPE/$ID
            echo "ID IS >>>>$ID<<<<<"
            echo ">>> Updating $TYPE ID:${ID} @ $BACKEND_URL/$TYPE/$ID"
            TH2='--header "Content-Type: application/json"'
            # echo curl $PROXY -s --request PUT $TH $TH2 $URL -d @${T_PATH}$OBJECT

            # extrapolate ENV Vars inside the json
            perl -pe 's/\$([_A-Z]+)/$ENV{$1}/g' < ${T_PATH}$OBJECT > /tmp/$OBJECT
            echo ">>> Wrote filtered object /tmp/$OBJECT"
            echo "RUNNING UPDATE>>> curl $PROXY -s --request PUT $TH $TH2 $URL -d @/tmp/$OBJECT"
            UPDATEDOBJECT=$(eval $( echo curl $PROXY -s --request PUT $TH $TH2 $URL -d @/tmp/$OBJECT ) )

            if [[ $UPDATEDOBJECT =~ .*createdAt.* ]]; then
                echo ">>> $ID Successfully Updated: $UPDATEDOBJECT"
            else
                echo ">>> Failed to update $TYPE $ID: $UPDATEDOBJECT"
                exit 1;
            fi
        else
            # Do a Put
            echo ">>> Creating $TYPE $ID"
            URL=$BACKEND_URL/$TYPE
            TH2='--header "Content-Type: application/json"'
            # echo curl $PROXY -s --request POST $TH $TH2 $URL -d @${T_PATH}$OBJECT

            # extrapolate ENV Vars inside the json
            perl -pe 's/\$([_A-Z]+)/$ENV{$1}/g' < ${T_PATH}$OBJECT > /tmp/$OBJECT
            echo ">>> Wrote filtered object /tmp/$OBJECT"
            echo "RUNNING CREATE>>> curl $PROXY -s --request POST $TH $TH2 $URL -d @/tmp/$OBJECT"
            NEWOBJECT=$(eval $( echo curl $PROXY -s --request POST $TH $TH2 $URL -d @/tmp/$OBJECT ) )

            if [[ $NEWOBJECT =~ .*(createdAt|Record saved).* ]]; then
                echo ">>> $ID Successfully Created: $NEWOBJECT"
            else
                echo ">>> Failed to create $TYPE $ID: $NEWOBJECT"
                echo curl $PROXY -s --request POST $TH $TH2 $URL -d @${T_PATH}$OBJECT
                exit 1;
            fi
        fi
    done
done

kill -9 $SSH