Jesse Yang ktmud

Easy parallel python with concurrent.futures

As of version 3.3, python includes the very promising concurrent.futures module, with elegant context managers for running tasks concurrently. Thanks to the simple and consistent interface you can use both threads and processes with minimal effort.

For most CPU bound tasks - anything that is heavy number crunching - you want your program to use all the CPUs in your PC. The simplest way to get a CPU bound task to run in parallel is to use the ProcessPoolExecutor, which will create enough sub-processes to keep all your CPUs busy.

We use the context manager thusly:

with concurrent.futures.ProcessPoolExecutor() as executor:

This is for launching and debugging Redash Flask app in VSCode.

We couldn't use the default Flask runner, because Redash implemented its own cli command manager.

We must use "program": "${workspaceFolder}/manage.py" instead of "module": "manage" because the relative path will cause some path issue, and result in template file not found errors.

This Gist demonstrates how to enable Okta/OpenID Connect login for Superset (and with slight modifications, other Flask-Appbuilder apps).

All you need to do is to update superset_config.py as following and make sure your Cliend ID (OKTA_KEY) and Secret (OKTA_SECRET) are put into the env.

Tensorflow 1.7 with CUDA on macOS Sierra 10.12.6, Anaconda, and Python 3.6

Largely based on the Tensorflow 1.6 gist, this should hopefully simplify things a bit. Mixing homebrew python2/python3 with pip ends up being a mess, so here's an approach to uses the built-in python27.

Requirements

NVIDIA Web-Drivers 378.05.05 for 10.12.6
CUDA 9.0 Toolkit
cuDNN 7.0.5 (latest release for mac os)
Python 3.6

Step 1: Install Python 3.6

I use anaconda:

wget https://repo.anaconda.com/archive/Anaconda3-5.3.0-MacOSX-x86_64.sh
bash ./Anaconda3-5.3.0-MacOSX-x86_64.sh
conda create -n idp intelpython3_full python=3
source activate idp

	from redis_store import store


	PRED_EXPIRE_SEC = 120


	@store.cache(expire=PRED_EXPIRE_SEC)
	def predict(model, photo_id):
	"""Predict labels for one photo"""
	# each thread uses a different learner cache

	#!/usr/bin/env python3 -W ignore::DeprecationWarning
	# -- coding: utf-8 --
	"""
	Sync GIS data from a PostgreSQL server to another
	"""
	import os
	import click

	from io import StringIO
	from sqlalchemy import create_engine

	library(future)
	library(tidyverse)
	library(datasets)
	library(randomForest)
	library(tictoc)

	plan(multisession, workers = 10)

	calc_imp <- function(idx) {
	Sys.sleep(1)

	from datetime import datetime, timedelta

	ONE_DAY = timedelta(days=1)

	def datetime_between(start, end, interval=timedelta(days=1),
	ret_dtype='datetime'):
	"""All datetime between start and end.
	Default interval: 1 day"""
	start = datetime.strptime(start.replace('-', ''), '%Y%m%d')
	end = datetime.strptime(end.replace('-', ''), '%Y%m%d')