Skip to content

Instantly share code, notes, and snippets.

View SQLadmin's full-sized avatar

SqlAdmin SQLadmin

View GitHub Profile
@SQLadmin
SQLadmin / aws-athena-auto-partition-between-dates.py
Last active March 23, 2023 21:08
AWS Athena automatically add partitions for given two dates for cloudtrail logs via lambda / Python
# Lambda function / Python to create athena partitions for Cloudtrail log between any given days.
# If you run this in AWS Lambda then it can't able to ceate all the partitions.
# Because lambda can run any functions up to 5mins. But create partition query will take avg 6 secs.
# I did a benchmark on lambda, it created upto 3 months of partitions on 16 regions.
'''
-----------------------------------------------------------------
AWS Athena Create Partitions Automatically For Given Any TWO DAYS
-----------------------------------------------------------------
Version 1.0
@SQLadmin
SQLadmin / aws-athena-auto-partition-lambda.py
Last active May 11, 2021 12:16
AWS Athena create auto partition for CloudTrail logs on Daily Basis
# Lambda function to create partition for Cloudtrail log on daily basis.
# You need to schedule it in AWS Lambda.
'''
-------------------------------------------
AWS Athena Create Partitions Automatically
-------------------------------------------
Version 1.0
Author: SqlAdmin
@SQLadmin
SQLadmin / postgres-user-migration.txt
Last active November 21, 2021 17:23
Migrate Postgresql users to another server and RDS
-- Export the users
pg_dumpall -g > users.sql
-- Importing to another PostgresSQL
awk '/CREATE/' users.sql > migrate.sql
-- Import to RDS; It won't support super user and replication roles. But we can grant minimal superuser with rds_superuser
psql -h localserver -d postgres -t -c"select 'grant rds_superuser to '||rolname ||';' from pg_roles where rolsuper='t';" -P "footer=off" >> migrate.sql
sed -i -e's/NOSUPERUSER//g; s/SUPERUSER//g; s/NOREPLICATION//g; s//REPLICATION/g' migrate.sql
psql -h rds-endpoint -U adminuser -d postgres < migrate.sql
@SQLadmin
SQLadmin / multi-rds-lambda.py
Created January 14, 2018 13:20
Get processlist from any RDS(mysql) while a cloudwatch alarm triggers.
# Run any query on any RDS(MYSQL) while a cloudwatch alarm triggers.
# Version 1.0
# Author: SqlAdmin
# Twitter: https://twitter.com/SqlAdmin
# Blog: www.sqlgossip.com
#----------------------------------------------------------------------------------------------------------------
# Make a note:
# In this function we used to get the RDS instance Identifier from the SNS, It won't give the complete endpoint
# So in your account go and see the endpoint, it should like identifier+some-random-string+region+amazon.com
# The random string which is next to the Identifier is differ from other account.
@SQLadmin
SQLadmin / lambda.py
Last active October 2, 2018 03:58
lambda error
# my code
from __future__ import print_function
import json
import sys
import os
print('Loading function')