Graham Duncan duncangh

Preface

This article walks you through an example of deploying a Python 3.6 application that uses Pandas and AWS S3 on AWS Lambda using Boto3 in Python in 2018. No shell, no bash, no web console, everything is automated in Python. The previous article of a Hello World example can be found here.

Again, the reason to use Python Boto3 to interact with AWS is that,

I'm more familiar with Python than Bash, which means a Python script can be more flexible and powerful than Bash for me.
I'm not a fun of the AWS web console. It might be easier to do certain things, but it is definitely not automated.

Introduction

Real-time Grid Component with Laravel, Vue.js, Vuex & Socket.io (Google Docs-like Functionality)

Motivation

The exercise of writing this tutorial -- as well as recording it as a screencast -- has helped me better understand the concepts behind a couple of my favorite open source tools. Both the tutorial and screencast will be of personal use in the future as references. If they are of help to others, that will be great too.

I love Google Docs' real-time, multi-user interactive capability, and I've have always been a fan of spreadsheets. I wanted to see if I could replicate that type of functionality. What I've done is taken the basic Vue.js Grid Component example and altered it a bit so that when a user clicks on a cell, that cell becomes highlighted or "active", not just in the user's browser but in any browser instance cur

Autoscroll Twitter Timeline

Want to quickly find your regrettable 10 year old tweets without having to learn how to use the twitter API or scroll manually through the infinite ether of the bad takes you've tweeted throughout the ages?

Then this is for you, ya lazy human. Just paste in the console and watch those hot takes go by.

function autoScrolling() {
   window.scrollTo(0,document.body.scrollHeight);
}

	from __future__ import division
	import string
	import math

	tokenize = lambda doc: doc.lower().split(" ")

	document_0 = "China has a strong economy that is growing at a rapid pace. However politically it differs greatly from the US Economy."
	document_1 = "At last, China seems serious about confronting an endemic problem: domestic violence and corruption."
	document_2 = "Japan's prime minister, Shinzo Abe, is working towards healing the economic turmoil in his own country for his view on the future of his people."
	document_3 = "Vladimir Putin is working hard to fix the economy in Russia as the Ruble has tumbled."

	// To load this script, open a Google Sheet (yeah, weird I know), then select "Tools->Script Editor"
	// From there, past this content into a script. You can set up triggers to run this script every day at a certain time
	// by selecting Resources -> Triggers.
	// I recommend you set the trigger to every 5-10 minutes. This will let the batches complete. If the trigger is too infrequent, it wont have time to finish.

	// https://developers.google.com/apps-script/reference/gmail/

	// For large inboxes (the ones you want to analyze) gmail will rate limit you.
	// They recommend adding a sleep to call less, but then your exec time will be too long.
	// To solve this, we run in batches. This is the batch size. Decrease this if exec time is too long.

	import token
	import tokenize
	from six.moves import cStringIO as StringIO
	import json
	from pandas.io.json import json_normalize
	import requests
	import pandas as pd


	def __fix_lazy_json(in_text):

	import boto3
	import pandas as pd
	from io import BytesIO

	bucket, filename = "bucket_name", "filename.csv"

	s3 = boto3.resource('s3')
	obj = s3.Object(bucket, filename)
	with BytesIO(obj.get()['Body'].read()) as bio:
	df = pd.read_csv(bio)

	def s3_to_pandas(client, bucket, key, header=None):

	# get key using boto3 client
	obj = client.get_object(Bucket=bucket, Key=key)
	gz = gzip.GzipFile(fileobj=obj['Body'])

	# load stream directly to DF
	return pd.read_csv(gz, header=header, dtype=str)

	def s3_to_pandas_with_processing(client, bucket, key, header=None):

	# I'm a one trick pony, but I can do ec2 conns too
	import boto3

	ec2 = boto3.client('ec2')
	instances : list = ec2.describe_instances()['Reservations']

	def _make_string_from_instance(instance):
	key_name = instance['KeyName']
	public_dns_name = instance['NetworkInterfaces'][0]['Association']['PublicDnsName']