Skip to content

Instantly share code, notes, and snippets.

View monkut's full-sized avatar

monkut monkut

31 followers · 1 following
View GitHub Profile
@monkut
monkut / gist:1d00900b4a633f407cd1a00c2217666c
Created June 30, 2019 07:12
video_transfer.sh
#!/bin/bash
# Command to transfer
export VIDEO_DIRECTORY=/storage/VIDEO/HOME/$(python3 -c "import datetime;print(datetime.datetime.now().strftime('%Y%m%d-video-dump'), end='')")
echo Making directory: ${VIDEO_DIRECTORY}
mkdir ${VIDEO_DIRECTORY}
echo Copying...
rsync --progress -h /media/monkut/JVCCAM_MEM/AVCHD/BDMV/STREAM/*.MTS ${VIDEO_DIRECTORY}
@monkut
monkut / PrepareAWSClientVPN.md
Created June 14, 2019 06:58
Prepare AWS Client VPN for connecting to a VPC without NAT/IGW

Prepare AWS Client VPN README

Prerequisites

  • AWS Account
  • Linux(like) System
  • awscli
@monkut
monkut / get_s3_latest_sagemaker_model_assets_key.py
Last active March 26, 2019 07:25
"""
Get the latest model assets key from resulting sagemaker training session
"""
import boto3
S3 = boto3.client('s3')
def get_latest_model_assets_key(bucket: str, prefix: str):
model_assets = []
paginator = S3.get_paginator('list_objects')
@monkut
monkut / ssh-tunnel-through-bastion-server.md
Created February 14, 2019 01:30
SSH Tunnel Trhough Bastion Server

SSH Tunnelにより、同じPCにDBがあるように接続する

  • Committing deleted app
  1. 自分のPCからTunnel貼る：

postgresqlを使用例

# LOCAL_PORT:REMOTE_ADDRESS:REMOTE_PORT
ssh -nNT -L 5432:{VPC内のRDSIPADDRESS}:5432 -i ~/.ssh/bastion.pem ubuntu@{IP ADDRESS} &
@monkut
monkut / add_python_to_WINDOWS_WSL_ubuntu.md
Created February 13, 2019 13:19
Add python to Windows Subsystem for Linux (WSL) [ubuntu]
@monkut
monkut / github-organizational-project-management.md
Last active January 21, 2019 09:06
My method of development with a team.

Purpose

This page describes how a project is managed using github organizational projects

Project Repository Branch Workflow

github flow is used for development.

When an "Assignee" takes a task, they create a "Feature branch" and develop their code in that branch. When task is complete a PR (Pull Request) is created. A PM or Lead (and relevant project members) reviews the PR, when the PR passes review a PM or Lead will merge the PR into the master branch.

@monkut
monkut / zappa_minimal_attach_policy.json.template
Created October 31, 2018 05:45
A minimal Zappa attach policy with bucket access
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:GetRole",
"iam:PutRolePolicy"
],
"Resource": [
@monkut
monkut / ubuntu_18.04_install_and_setup.md
Last active August 17, 2018 01:47
Base install/setup of ubuntu 18.04.1

Install

Setup a big empty partition in ext4 and assign to / root.

Proceed with install

This assumes you have the latest Ubuntu Desktop 18.04 image on a USB key.

Go the iso download and burn key with Etcher

@monkut
monkut / csv_multiprocessing_pool.py
Created July 23, 2018 01:03
A common pattern for processing csv files in parallel
import os
import csv
from multiprocessing import cpu_count, Pool
def process_csv(args):
filepath, encoding = args
unique_ids = set()
with open(filepath, 'r', encoding=encoding) as in_f:
reader = csv.reader(in_f)
@monkut
monkut / japanese_business_calendar.py
Created July 11, 2018 14:24
Using pandas AbstractHolidayCalendar to calculate work days
import datetime
from pandas.tseries.holiday import AbstractHolidayCalendar, Holiday, sunday_to_monday, MO
from pandas.tseries.offsets import Day, CustomBusinessDay, DateOffset
class JapanBusinessCalendar(AbstractHolidayCalendar):
rules = [
Holiday('New Years Day', month=1, day=1),
Holiday('Coming of Age Day', month=1, day=8), # second monday of Jan
Holiday('National Foundation Day', month=2, day=11, observance=sunday_to_monday), # observed monday if falls on Sunday