vaquarkhan

'use strict'

// IMPORTANT: Replace environment variables with your current values
const aws = require('aws-sdk')
aws.config.region = process.env.REGION

const sts = new aws.STS({apiVersion: '2011-06-15'})

module.exports.handler = (event, context, callback) => {
    console.log('User email', event.queryStringParameters.email)

vaquarkhan

# Modified by adding names from https://github.com/richardasaurus/nginx-access-log-parser/blob/master/main.py
pat = (r''
        '(?P<ip>\d+.\d+.\d+.\d+)\s-\s-\s' #IP address
        '\[(?P<time>.+)\]\s' #datetime
        '"(?P<method>GET|POST)\s(?P<uri>.+)\s(?P<ver>\w+/.+)"\s(?P<status>\d+)\s' #requested file
        '(?P<content_length>\d+)\s"(?P<referrer>.+)"\s' #referrer
        '"(?P<user_agent>.+)"' #user agent
      )
r = re.match(pat, logline)
r.groupdict()

vaquarkhan

Databricks Delta Lake - A Friendly Intro

This article introduces Databricks Delta Lake. A revolutionary storage layer that brings reliability and improve performance of data lakes using Apache Spark.

First, we'll go through the dry parts which explain what Apache Spark and data lakes are and it explains the issues faced with data lakes. Then it talks about Delta lake and how it solved these issues with a practical, easy-to-apply tutorial.

Introduction to Apache Spark

If you don't know what Spark is, Apache Spark is a large-scale data processing and unified analytics engine for big data and machine learning. It was originally developed at UC Berkeley in 2009. Apache Spark is 100% open source, hosted at the vendor-independent Apache Software Foundation.

vaquarkhan

import json
import boto3

"""
API gateway URL example. You have to allow your quicksight domin setting to be accessed from amazonaws.com including subdomains.
https://xxxxxxxx.execute-api.ap-northeast-1.amazonaws.com/xxx-gateway-stage-xxxx/your-lambda-func-name?dashboard_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxx
"""

def lambda_handler(event, context):
    dashboard_id=event["queryStringParameters"]['dashboard_id']

vaquarkhan

AWSLambdaBasicExecutionPolicy
---
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",

vaquarkhan

# Import the schema, datafile and io submodules
# from avro (easy_install avro)
from avro import schema, datafile, io
 
OUTFILE_NAME = 'sample.avro'
 
SCHEMA_STR = """{
    "type": "record",
    "name": "sampleAvro",
    "namespace": "AVRO",

vaquarkhan

#!/usr/bin/env python

import getpass
import json
import requests
import sys
import urllib

import boto3

vaquarkhan

[
  {
    "Name": "Ship Flink runtime to cluster",
    "Path": "s3://<your_bucket>/flink/ship_flink_runtime.sh"
  },
  {
    "Name": "Ship application to cluster",
    "Path": "s3://<your_bucket>/flink/ship_app.sh"
  }
]

vaquarkhan

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0

vaquarkhan

# First I created a working directory
~$ mkdir s3-kafka
  
# Then I installed the dependency I needed in that directory with pip
~$ cd s3-kafka
~$ pip install kafka-python -t $(pwd)
  
# Then I put my code into a file called s3-kafka.py
~$ vi s3-kafka.py