Skip to content

Instantly share code, notes, and snippets.

View gmr's full-sized avatar
:octocat:
🐘 🐇 🐍

Gavin M. Roy gmr

:octocat:
🐘 🐇 🐍
View GitHub Profile
@gmr
gmr / quorum-to-classic.py
Created July 18, 2020 19:40
Python CLI application that converts all quorum queues on a RabbitMQ cluster to classic queues with the intent of not loosing any messages or impacting production workloads (aside from disconnecting consumers on a queue)
#!/usr/bin/env python3
import argparse
import json
import logging
import sys
import time
import typing
from urllib import parse
import httpx
We can't make this file beautiful and searchable because it's too large.
loan_amount,city,state,business_type,industry,lender
495.62,KIHEI,HI,Sole Proprietorship,Reupholstery and Furniture Repair,HawaiiUSA FCU
495.00,HONOLULU,HI,Sole Proprietorship,"Photography Studios, Portrait",First Hawaiian Bank
488.75,KANEOHE,HI,Limited Liability Company(LLC),All Other Personal Services,First Hawaiian Bank
425.00,LIHUE,HI,Sole Proprietorship,Taxi Service,Kaua'i Government Employees' FCU
422.50,KAPOLEI,HI,Sole Proprietorship,Taxi Service,First Hawaiian Bank
415.00,HONOLULU,HI,Limited Liability Company(LLC),Janitorial Services,Bank of Hawaii
325.00,HONOLULU,HI,Limited Liability Company(LLC),All Other Transit and Ground Passenger Transportation,Bank of Hawaii
294.57,KAPOLEI,HI,Limited Liability Company(LLC),Offices of Physicians (except Mental Health Specialists),Bank of Hawaii
282.50,HONOLULU,HI,Limited Liability Partnership,Offices of All Other Miscellaneous Health Practitioners,First Hawaiian Bank
CREATE OR REPLACE VIEW redirect_list AS
WITH RECURSIVE recursive_url_relationships (job_id, source, source_url, target, target_url, redirect) AS
(SELECT a.job_id,
a.source,
b.value AS source_url,
a.target,
c.value AS target_url,
a.redirect
FROM url_relationships AS a
JOIN urls AS b ON b.id = a.source
{
"$schema": "http://json-schema.org/schema#",
"$id": "https://pglifecycle.readthedocs.io/en/stable/schemata/table.html",
"title": "Table",
"description": "Defines a table",
"type": "object",
"properties": {
"name": {
"title": "Name",
"description": "The table name",
@gmr
gmr / 1-example.py
Last active June 4, 2019 22:25
Example usage of pgdumplib to read data from a pg_dump file
import pgdumplib
dump = pgdumplib.load('build/data/dump.compressed')
print('# Header: {}'.format(dump.toc.header))
print('# Database: {}'.format(dump.toc.dbname))
print('# Archive Timestamp: {}'.format(dump.toc.timestamp))
print('# Server Version: {}'.format(dump.toc.server_version))
print('# Dump Version: {}'.format(dump.toc.dump_version))
import gzip
import requests
r = requests.get('http://data.dot.state.mn.us/iris_xml/incident.xml.gz')
with open('incident.xml', 'w') as handle:
handle.write(gzip.decompress(r.content).decode('utf-8'))
@gmr
gmr / s3-strip-object-tags.py
Created August 29, 2018 17:33
Small Python CLI application for removing any object tags from all objects in a bucket
#!/usr/bin/env python3
import argparse
import logging
import sys
import boto3
LOGGER = logging.getLogger(__name__)
@gmr
gmr / table_sizes.sql
Last active February 6, 2018 00:46
A PostgreSQL view that combines the relation size, the size of its indexes, the size of its toast table, and the size of the toast table indexes, combines them to give you a total table size.
CREATE OR REPLACE VIEW public.table_sizes AS
WITH tables AS (
SELECT a.oid, b.nspname, a.relname, a.reltoastrelid, pg_relation_size(a.oid) AS size
FROM pg_class AS a
JOIN pg_namespace AS b ON b.oid = a.relnamespace
WHERE a.relkind = 'r'
AND b.nspname NOT IN ('pg_catalog', 'information_schema')),
indexes AS (
SELECT i.oid, n.nspname, i.relname AS idxname, c.oid AS reloid, c.relname AS relname,
pg_relation_size(i.oid) AS size
@gmr
gmr / gist:3c28c9e3a71f5ef9fd83f1867fa6f11f
Created January 23, 2018 19:11
execute time per 10k chunk
Time: 3198.725 ms (00:03.199)
Time: 6887.960 ms (00:06.888)
Time: 10622.234 ms (00:10.622)
Time: 13110.130 ms (00:13.110)
Time: 21388.449 ms (00:21.388)
Time: 24127.707 ms (00:24.128)
Time: 28515.858 ms (00:28.516)
Time: 32666.766 ms (00:32.667)
@gmr
gmr / rabbitmq-autocluster_k8s_persistent.bash
Created January 10, 2018 20:24 — forked from pmint93/rabbitmq-autocluster_k8s_persistent.bash
Deploy rabbitmq-autocluster on k8s with persistent storage (EBS)
#!/bin/bash
set -eo pipefail
export KUBE_NAMESPACE=test
export REPLICA_COUNT=3
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1beta1
kind: StatefulSet