Skip to content

Instantly share code, notes, and snippets.

View rgpower's full-sized avatar

Rick Power rgpower

5 followers · 0 following
  • Halifax, NS
View GitHub Profile
@rgpower
rgpower / Semaphore.pm
Created February 15, 2021 18:10
AnyEvent Semaphore for Perl
package AnyEvent::Semaphore;
use common::sense;
use AnyEvent;
use Try::Tiny;
use Carp 'croak';
use Class::Accessor::Lite (
ro => [ qw(limit waiting) ],
);
@rgpower
rgpower / Firebase.pm
Created February 15, 2021 18:07
Firebase API for Perl w/JWT
package AnyEvent::Firebase;
use common::sense;
use AnyEvent;
use AnyEvent::HTTP 'http_request';
use Carp 'croak';
use Coro qw(rouse_cb rouse_wait);
use Coro::Semaphore;
use Crypt::JWT 'encode_jwt';
use JSON::XS;
@rgpower
rgpower / backfill.py
Created January 14, 2021 15:42
process n items using m threads using asyncio and a queue in python or use it to backfill failed lambda executions
from dotenv import load_dotenv
load_dotenv()
import asyncio
import datetime
import functools
import json
import logging
import os
@rgpower
rgpower / invoke_lambda_many_times.sh
Last active January 5, 2021 22:09
invoke lambda many times with different payloads, from a sed-based template
# invoke (synchronously) a lambda with a bunch of different uid payloads
cat << EOF > user.lst
abc123
def345
... more...uids...
EOF
cat << EOF > sns.json
{
"Records": [
@rgpower
rgpower / parse_iso8601.py
Last active June 13, 2021 18:42
parse iso8601 '2019-11-26T18:26:19Z' format string in python
"""
parse iso8601 '2019-11-26T18:26:19Z' format string in python
"""
def from_iso8601(datetime_iso: str):
return datetime.datetime.strptime(datetime_iso, "%Y-%m-%dT%H:%M:%S%z")
@rgpower
rgpower / lambda_fn.py
Last active January 4, 2021 19:20
Include RequestId in AWS Lambda CloudWatch logs
"""
Use this to get output that includes the Lambda context RequestId in your CloudWatch logs
This allows you to differentiate the logs of concurrent invocations.
To use in your lambda:
import logging
logging.getLogger().setLevel(logging.INFO)
def handler(event, context):
@rgpower
rgpower / Browserstack.java
Created December 15, 2020 19:39
A Junit5 Extension that automagically sets BrowserStack sessions to pass/fail for Browser Automation Tests
/**
* Implements the Browserstack API for setting test sessions to pass/fail status as desribed in
* https://www.browserstack.com/docs/automate/selenium/view-test-results/mark-tests-as-pass-fail
*
**/
package utils.browserstack;
import com.google.gson.Gson;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
@rgpower
rgpower / GitUtils.java
Created December 15, 2020 19:06
Use JGit to grab latest commit from a git working directory, and detect local changes
package utils.git;
import org.eclipse.jgit.api.Git;
import org.eclipse.jgit.api.errors.GitAPIException;
import org.eclipse.jgit.lib.Repository;
import org.eclipse.jgit.lib.RepositoryBuilder;
import org.eclipse.jgit.revwalk.RevCommit;
import java.io.File;
import java.io.IOException;
@rgpower
rgpower / lambda_s3_to_bq_import.js
Created October 18, 2020 01:07
Firehose S3 to BigQuery Import, with NDSON per-row processing
"use strict";
const GCP_PROJECT = process.env["GCP_PROJECT"];
const BQ_DATASET = process.env["BQ_DATASET"];
const BQ_TABLE = process.env["BQ_TABLE"];
const S3_BUCKET = process.env["S3_BUCKET"];
const S3_PREFIX = process.env["S3_PREFIX"];
const GCP_CREDENTIALS = process.env["GCP_CREDENTIALS"];
@rgpower
rgpower / manually_run_scheduled_query.md
Created October 17, 2020 13:10
BigQuery: locate and manually run a scheduled query for a specific run datetime

BigQuery: locate and manually run a scheduled query for a specific run datetime

$ bq --project_id my_project_id ls --transfer_config --transfer_location=us | grep query-name

  projects/my_project_id/locations/us/transferConfigs/config_id   query-name   scheduled_query   SUCCEEDED  

$ bq mk --transfer_run --run-time="2020-09-30T23:30:00Z" "projects/my_project_id/locations/us/transferConfigs/config_id"