Skip to content

Instantly share code, notes, and snippets.

View maatthc's full-sized avatar
🍄

MaaT maatthc

🍄
  • MaaT Tech
  • Melbourne
  • 18:12 (UTC -12:00)
View GitHub Profile
@maatthc
maatthc / 90-notify_ip_to_slack.sh
Last active May 14, 2019 04:09
Shell script to post to Slack every time the Raspberry Pi changes its IP Address via DHCP. Should be in the folder /lib/dhcpcd/dhcpcd-hooks/
#!/bin/bash
if [ "$interface" = wlan0 ]; then
if [ "$if_up" = true ]; then
IP=`ip add show wlan0|grep global |awk '{print $2}'`
PREVIOUS_IP=`cat /tmp/notify_output`
if [ "$PREVIOUS_IP" != "$IP" ]; then
echo 'Notifying ip: '$IP' to Slack Channel..'
echo $IP > /tmp/notify_output
curl -X POST -H 'Content-type: application/json' --data '{"text":"RaspBerry TV IP is now: '${IP}'","icon_emoji":":tv:", "username":"RaspBerry"}' https://hooks.slack.com/services/XXXX
fi
@maatthc
maatthc / process_csv_using_stream.js
Created May 14, 2019 04:13
Avoid the "JavaScript heap out of memory" : Example of how to process huge CSV files using Streams in Nodejs.
#!/usr/bin/env node
const fs = require('fs')
const path = require('path')
const moment = require('moment')
const csv = require('fast-csv')
const highland = require('highland')
const writeableStream = fs.createWriteStream(path.join(__dirname, 'input.csv'))
const csvStream = csv.fromPath(path.join(__dirname, 'output.csv'), { headers: true, trim: true, objectMode: true })
const outputStream = csv.createWriteStream({ headers: true })
@maatthc
maatthc / find_latest_file_in_s3_bucket.py
Last active May 22, 2019 07:41
Find the latest file in a AWS S3 Bucket using Boto3
import boto3
def find_latest_file_in_folder(bucket, path):
s3 = boto3.client('s3')
s3_keys = s3.list_objects_v2(Bucket=bucket, Prefix=path)['Contents']
last_added = max(s3_keys, key=lambda x: x['LastModified'])
return "%s/%s" % (bucket,last_added['Key'])
@maatthc
maatthc / Glue_Spark_job_example.yml
Last active February 10, 2022 18:02
Cloud Formation example for Glue Spark Job with metrics and scheduler
AWSTemplateFormatVersion: '2010-09-09'
Description: Cloud Formation example for Glue Spark Job with metrics and scheduler
Parameters:
ArtifactBucket:
Description: A global deployable artefact bucket
Type: String
Default: artefacts
ServiceName:
Description: Service Name that owns the stack when created
const BATCH_SIZE = 10
const buildParamforSqsBatch = (products, queueUrl) => {
const entries = products.map(product => toOutputMessage(product))
return {
Entries: entries,
QueueUrl: queueUrl
}
}
const emitUpdateEvent = (params) => {
@maatthc
maatthc / promise_timeout.js
Last active August 14, 2019 03:54
Javascript Promises Timeout
//
// Javascript Promises Timeout
//
// A way to implement timeouts in jS promises using :
// - setTimeout():
// sets up a function to be called after a specified delay in milliseconds.
// - Promise.race():
// Creates a Promise that is resolved or rejected when any of the provided Promises are resolved or rejected.
//
// Run 'seq 10 | xargs -Iz node promise_timeout.js' on the command line to see the results flapping between 'Done' and 'TimeOut'
@maatthc
maatthc / aws-websockets-api-gw-mock.yaml
Last active June 19, 2025 01:03
Basic WebSocket mock in AWS ApiGateway using CloudFormation in AWS
AWSTemplateFormatVersion: '2010-09-09'
Description: |
AWS CloudFormation template for Mock WebSocket API Gateway. When deploying this stack please remember to check the option:
- I acknowledge that AWS CloudFormation might create IAM resources.
This template can help you to solve issues like:
- CloudWatch Logs role ARN must be set in account settings to enable logging
- Execution failed due to configuration error: statusCode should be an integer which defined in request template
- This custom domain name cannot map to WEBSOCKET protocol APIs
- Error during WebSocket handshake: Unexpected response code: 500
@maatthc
maatthc / async_tasks.py
Created February 8, 2020 11:16
Running asynchronous functions (coroutines) with Python 3 - similarly with JavaScript Promise.all
import asyncio
import random
loop = asyncio.get_event_loop()
async def execute_task():
time_out = random.uniform(0.5, 5)
print("Starting {:1.2f}..".format(time_out))
await asyncio.sleep(time_out)
print("Stoping {:1.2f}..".format(time_out))
AWSTemplateFormatVersion: "2010-09-09"
Description: CloudFormation for creating Python Glue Jobs and schedule it's execution
Parameters:
ServiceName:
Type: String
Description: Name of the service
ArtifactBucket:
Type: String
Description: A bucket to get artefacts from
@maatthc
maatthc / budget.yml
Created April 2, 2020 05:26
Cloud Formation template for Budget, using tags
AWSTemplateFormatVersion: "2010-09-09"
Description: Budget alerts for various AWS services
Parameters:
Environment:
Type: String
Description: Deployment environment for the budget checks
GlueDevEndpointDPUHours:
Type: Number
Description: Number of Glue dev endpoint DPU hours we budget for in the period