g-a-d’s gists

g-a-d / gist:033c57ec013b9a30e7062d3975159fb8

Created August 24, 2017 10:56

Adding parameters to a pyspark job in AWS Glue

	from awsglue.transforms import *
	from awsglue.utils import getResolvedOptions
	from pyspark.context import SparkContext
	from awsglue.context import GlueContext
	from awsglue.job import Job
	from pyspark.sql.types import *
	from awsglue.dynamicframe import DynamicFrame

	## @params: [JOB_NAME, CUSTOM1, CUSTOM2, CUSTOM3]
	args = getResolvedOptions(sys.argv, ['JOB_NAME', 'CUSTOM1', 'CUSTOM2', 'CUSTOM3'])

g-a-d / lambda_handler.py

Last active August 22, 2021 06:55

Process either S3 event notifications or SNS messages with python in AWS Lambda

	import urllib

	def lambda_handler(event, context):
	for record in event['Records']:
	try:
	if 'aws:sns' == record['EventSource'] and record['Sns']['Message']:
	record = json.loads(record['Sns']['Message'])['Records'][0]
	except KeyError:
	pass
	bucket = record['s3']['bucket']['name']

g-a-d / gist:4bc7f716bc57e42b64e1ef450be9bae8

Created November 6, 2017 15:41

Converting CloudFormation parameter files to CodePipeline Template Configuration files

	Using jq, we can convert between CloudFormation parameter files and CodePipeline template configuration files:

	$ jq '{ Parameters: [ .[] \| { (.ParameterKey): .ParameterValue } ] \| add } ' < cloudformation_parameter_file.json

	This is useful in the case of receiving 'Template configuration is not valid' when running a CloudFormation action.

	A CloudFormation parameter file has format:

	[
	{

g-a-d / gist:4e460c3a6b1a2eb1693eaef529cbe057

Created November 29, 2021 09:36

Dump email addresses from multiple github organizations

	export TOKEN="github_personal_access_token"

	for org in Org1 Org2 Org3 Org4 ; do \
	curl -G 'https://api.github.com/scim/v2/organizations/'"$org"'/Users?Count=1000' \
	-H "Authorization: token $TOKEN" -H "Accept: application/vnd.github.v3+json" \| \
	jq -r ".Resources[].emails[0] \| [\"$org\", .value] \| @csv"
	done >> emails.csv

g-a-d / get_api_token.py

Created April 22, 2022 16:02

Get Github API token for a Github app (python)

	def get_github_token(app_id, pem_file, organization_slug, repository, private_key_file):
	jwt = get_jwt_token(app_id, private_key_file)
	installation = requests.get(f'https://api.github.com/orgs/{organization_slug}/installation', headers={'Authorization': f'Bearer {jwt}'})
	access_tokens_url = installation.json()['access_tokens_url']
	# values for the permissions are documented here: https://docs.github.com/en/rest/overview/permissions-required-for-github-apps
	data = {'repository': repository, 'permission': {'contents': 'write'}}
	token_request = requests.post(access_tokens_url, headers={'Authorization': f'Bearer {jwt}'}, json=data)
	token = token_request.json()['token']
	return token

Guy Davies g-a-d