vaquarkhan

## Python 2.7 
## GlueLastRunDuration.py 
## Version 1
## by Lydon Carter October 2018

## USE
# Script to get a specific AWS Glue Job and tell you the duration of
# the last run.
# Notes:
# -- The script will use the location you setup for your Glue Context in the "Needed stuff"

vaquarkhan

# AWS Version 4 signing example
# taken from:  
#    http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
#    https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
#    https://www.javaquery.com/2016/01/aws-version-4-signing-process-complete.html
# Lambda API (InvokeAsync)
# http://docs.aws.amazon.com/lambda/latest/dg/API_InvokeAsync.html

# See: http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
# This version makes a POST request and passes request parameters

vaquarkhan

'use strict'

// IMPORTANT: Replace environment variables with your current values
const aws = require('aws-sdk')
aws.config.region = process.env.REGION

const sts = new aws.STS({apiVersion: '2011-06-15'})

module.exports.handler = (event, context, callback) => {
    console.log('User email', event.queryStringParameters.email)

vaquarkhan

# Modified by adding names from https://github.com/richardasaurus/nginx-access-log-parser/blob/master/main.py
pat = (r''
        '(?P<ip>\d+.\d+.\d+.\d+)\s-\s-\s' #IP address
        '\[(?P<time>.+)\]\s' #datetime
        '"(?P<method>GET|POST)\s(?P<uri>.+)\s(?P<ver>\w+/.+)"\s(?P<status>\d+)\s' #requested file
        '(?P<content_length>\d+)\s"(?P<referrer>.+)"\s' #referrer
        '"(?P<user_agent>.+)"' #user agent
      )
r = re.match(pat, logline)
r.groupdict()

vaquarkhan

Databricks Delta Lake - A Friendly Intro

This article introduces Databricks Delta Lake. A revolutionary storage layer that brings reliability and improve performance of data lakes using Apache Spark.

First, we'll go through the dry parts which explain what Apache Spark and data lakes are and it explains the issues faced with data lakes. Then it talks about Delta lake and how it solved these issues with a practical, easy-to-apply tutorial.

Introduction to Apache Spark

If you don't know what Spark is, Apache Spark is a large-scale data processing and unified analytics engine for big data and machine learning. It was originally developed at UC Berkeley in 2009. Apache Spark is 100% open source, hosted at the vendor-independent Apache Software Foundation.

vaquarkhan

import json
import boto3

"""
API gateway URL example. You have to allow your quicksight domin setting to be accessed from amazonaws.com including subdomains.
https://xxxxxxxx.execute-api.ap-northeast-1.amazonaws.com/xxx-gateway-stage-xxxx/your-lambda-func-name?dashboard_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx
"""

def lambda_handler(event, context):
    dashboard_id=event["queryStringParameters"]['dashboard_id']

vaquarkhan

AWSLambdaBasicExecutionPolicy
---
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",

vaquarkhan

# Import the schema, datafile and io submodules
# from avro (easy_install avro)
from avro import schema, datafile, io
 
OUTFILE_NAME = 'sample.avro'
 
SCHEMA_STR = """{
    "type": "record",
    "name": "sampleAvro",
    "namespace": "AVRO",

vaquarkhan

#!/usr/bin/env python

import getpass
import json
import requests
import sys
import urllib

import boto3

vaquarkhan

[
  {
    "Name": "Ship Flink runtime to cluster",
    "Path": "s3://<your_bucket>/flink/ship_flink_runtime.sh"
  },
  {
    "Name": "Ship application to cluster",
    "Path": "s3://<your_bucket>/flink/ship_app.sh"
  }
]