jamiekt’s gists

jamiekt / gist:d09e5c45afc4bc50aba39a0fcc6c860f

Created May 10, 2021 21:22

terraform debug output when attempting to deploy bigquery table with a policy tag

	2021/05/10 21:13:23 [WARN] Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for google_bigquery_table.table, but we are tolerating it because it is using the legacy plugin SDK.
	The following problems may be the cause of any confusing errors from downstream operations:
	- .schema: planned value cty.StringVal("[{\"mode\":\"NULLABLE\",\"name\":\"col1\",\"policyTags\":\"projects/msm-groupdata-datalake-dev/locations/eu/taxonomies/7353867875344393839/policyTags/5061567500970922730\",\"type\":\"INTEGER\"}]") does not match config value cty.StringVal("[\n {\n \"name\": \"col1\",\n \"type\": \"INTEGER\",\n \"mode\": \"NULLABLE\",\n \"policyTags\": \"projects/msm-groupdata-datalake-dev/locations/eu/taxonomies/7353867875344393839/policyTags/5061567500970922730\"\n }\n]\n")
	google_bigquery_table.table: Creating...
	2021/05/10 21:13:23 [DEBUG] EvalApply: ProviderMeta config value set
	2021/05/10 21:13:23 [DEBUG] google_bigquery_table.table: applying the pl

jamiekt / get-role-members.sh

Last active May 27, 2022 15:42

Useful gcloud IAM commands

	gcloud projects get-iam-policy project-id \
	--filter="bindings.role:roles/workflows.editor" \
	--flatten="bindings[].members" \
	--format="table(bindings.members)"
	#### output
	# group:group-name@example.com
	# user:user-name@example.com
	# servceAccount:service-account-name@project-id.iam.gserviceaccount.com

jamiekt / .pre-commit-config.yaml

Last active June 15, 2022 21:03

select python interpreter in poetry

	repos:
	- repo: https://github.com/pre-commit/pre-commit-hooks
	rev: v4.3.0
	hooks:
	- id: end-of-file-fixer
	- id: trailing-whitespace

	- repo: local
	hooks:
	- id: black

jamiekt / poetry-env-info-3_8

Created June 15, 2022 21:02

jamiekt / pyproject.toml

Created June 15, 2022 21:04

	[tool.poetry]
	name = "real-time-event-processor"
	version = "0.1.0"
	description = ""
	authors = []

	[tool.poetry.dependencies]
	python = "^3.9, <3.10"

	Flask = "^2.0.1"

jamiekt / mypymoan.py

Last active July 22, 2022 13:11

	import re
	matched_policies = {...} #some logic to derive a set
	return {
	re.search(r"p\d{6}", matched_policy).group(0)
	for matched_policy in matched_policies
	}

jamiekt / satisfy_mypy_with_loop.py

Created July 22, 2022 13:12

	prefixes = set()
	for matched_policy in matched_policies:
	search_result = re.search(r_search, matched_policy)
	assert search_result is not None
	prefixes.add(search_result.group(0))
	return prefixes

jamiekt / satisfy_mypy_with_walrus_operator.py

Last active July 22, 2022 13:16

	return {
	match.group(0)
	for matched_policy in matched_policies
	if (match := re.search(r"p\d{6}", matched_policy)) is not None
	}

jamiekt / query.sql

Last active October 22, 2022 16:41

ARRAY() with SELECT AS STRUCT

	SELECT c.borough,c.year,
	ARRAY(
	select as struct c2.month,sum(value) crime_tally
	from `bigquery-public-data.london_crime.crime_by_lsoa` as c2
	where c.borough = c2.borough
	and c.year = c2.year
	group by c2.month
	) as monthly_summary
	FROM (
	select distinct borough,year

jamiekt / demo_xcom_problem.py

Last active April 14, 2024 20:29

Airflow XCom problem

	import datetime
	from airflow.decorators import task, dag

	from airflow.providers.amazon.aws.operators.emr import EmrServerlessCreateApplicationOperator

	@dag(
	dag_id="demo-xcom-problem",
	start_date=datetime.datetime(2021, 1, 1),
	catchup=False
	)

Jamie Thomson jamiekt