momota10s’s gists

momota10s / gke-autopilot-values.yaml

Last active May 28, 2022 02:21

Airflow2.1.2をhelmにてGKE(Autopilot)にデプロイするためのvaluesの定義。templateは https://github.com/airflow-helm/charts/blob/airflow-8.5.2/charts/airflow/values.yaml

	########################################
	## CONFIG \| Airflow Configs
	########################################
	airflow:
	## if we use legacy 1.10 airflow commands
	##
	legacyCommands: false

	## configs for the airflow container image
	##

momota10s / gke-sandbox-values.yaml

Last active May 28, 2022 02:02

Airflow2.1.2をhelmにてGKEにデプロイするためのvaluesの定義。templateは https://github.com/airflow-helm/charts/blob/airflow-8.5.2/charts/airflow/values.yaml

	########################################
	## CONFIG \| Airflow Configs
	########################################
	airflow:
	## if we use legacy 1.10 airflow commands
	##
	legacyCommands: false

	## configs for the airflow container image
	##

momota10s / cloudwatch_err_slack.py

Created November 18, 2021 11:09

cloudwatchのサブスクリプションにlambdaを登録してエラーをslackに通知するスクリプト

	import os
	import gzip
	import json
	import base64
	from urllib.request import Request, urlopen
	from urllib.error import URLError, HTTPError

	SLACK_CHANNEL = os.environ['SLACK_CHANNEL']
	SLACK_WEBHOOK = os.environ['SLACK_WEBHOOK']

momota10s / sns_lambda_slack.py

Last active November 22, 2021 09:55

Amazon SNSのサブスクリプション先としてlambdaを指定、その後lambdaにてslack通知を行う

	import json
	from pytz import timezone
	from dateutil import parser
	import urllib.request

	print('Loading function')


	def lambda_handler(event, context):

momota10s / requirements.txt

Created October 13, 2021 04:39

pip freeze で出てきたものをぺたり

	absl-py==0.11.0
	alembic==1.4.2
	ansiwrap==0.8.4
	apache-airflow==2.0.1
	apache-airflow-providers-ftp==1.0.1
	apache-airflow-providers-http==1.1.1
	apache-airflow-providers-imap==1.0.1
	apache-airflow-providers-sqlite==1.0.2
	apispec==3.3.2
	appdirs==1.4.4

momota10s / googleapi-geocode-sample.py

Created September 29, 2021 04:52

	import requests

	address = "東京都千代田区千代田"
	api_key = ""

	requestUrl = f'https://maps.googleapis.com/maps/api/geocode/json?address="{address}"&key={api_key}&sensor=false'
	r = requests.get(requestUrl).json()

	if(r['results'] == []):
	print("non result")

momota10s / check_cloudbuild_status.py

Created September 22, 2021 05:02

	from googleapiclient import discovery

	PROJECT_ID = "your-project"
	build_id = "youru-build-id"

	cloudbuild = discovery.build("cloudbuild", "v1", cache_discovery=False)
	request = cloudbuild.projects().builds().get(id=build_id, projectId=PROJECT_ID)
	res = request.execute()

	print(res)

momota10s / sample_pubsub_operater.py

Created September 21, 2021 02:23

	from airflow import DAG
	from airflow.models import Variable
	from airflow.utils.dates import days_ago
	from airflow.contrib.operators.pubsub_operator import (
	PubSubPublishOperator,
	PubSubSubscriptionCreateOperator,
	PubSubSubscriptionDeleteOperator,
	PubSubTopicCreateOperator,
	PubSubTopicDeleteOperator
	)

momota10s / cloudbuild.yaml

Created April 20, 2021 01:25

	steps:

	# Access the id_github file from Secret Manager
	- name: gcr.io/cloud-builders/gcloud
	entrypoint: 'bash'
	args: [ '-c', 'gcloud secrets versions access latest --secret=github-access-token > /root/github-access-token' ]
	volumes:
	- name: 'vol'
	path: /root/

momota10s / zengin.py

Last active March 6, 2021 10:20

全銀データを取得するpythonコード

	import datetime
	import os
	import pandas as pd
	import pytz

	ZENGIN_JSON_PATH = "https://raw.githubusercontent.com/zengin-code/source-data/master/data/banks.json"

	zengin = pd.read_json(ZENGIN_JSON_PATH)
	zengin = zengin.T[["code", "name", "hira", "kana", "roma"]]

Momota Sasaki momota10s