Skip to content

Instantly share code, notes, and snippets.

View post2web's full-sized avatar

Ivelin Angelov post2web

12 followers · 2 following
  • Intuit Inc
  • San Diego, CA
View GitHub Profile
@post2web
post2web / add venv jupyter kernel
Created October 29, 2018 02:17
$ python -m venv projectname
$ source projectname/bin/activate
(venv) $ pip install ipykernel
(venv) $ ipython kernel install --user --name=projectname
@post2web
post2web / local_datalab.md
Created November 2, 2018 22:26

Make jupyter [lab] act as a cloud DataLab

%load_ext google.datalab.kernel
  • alternative, enable globally in ipython_config.py:
c = get_config()
@post2web
post2web / jupyter_notebook_watermarks.md
Last active November 2, 2018 22:32

Enviroment watermarks

%load_ext watermark
%watermark -p pandas,pandas_datareader,numpy,scipy,sklearn,matplotlib,seaborn

2018-11-02T15:32:33-07:00

CPython 3.6.5
@post2web
post2web / local run cloud build.py
Created December 3, 2018 02:37
local run cloud build
#!/bin/bash
# gcloud builds submit --config cloudbuild.yaml .
# docker run -v `pwd`:/workspace -w /workspace -it refresh-metadata
# gcloud components install docker-credential-gcr
# docker-credential-gcr configure-docker
# gcloud components install cloud-build-local
rm -fr /workspace/*
@post2web
post2web / image net
Created December 21, 2018 02:12
wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_train.tar
wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_val.tar
@post2web
post2web / xgboost docker training.md
Last active January 30, 2020 23:50
Local training of ML containers 
docker run \
  -v "${PWD}":/local-drive \
  -it gcr.io/aihub-c2t-containers/kfp-components/trainer/dist_xgboost:latest \
  --output-location /local-drive/xgboost-output \
  --training-data gs://aihub-c2t-containers-public/release-0.2.0/kfp-components/trainer/dist_xgboost/data/csv/iris/train.csv \
  --validation-data gs://aihub-c2t-containers-public/release-0.2.0/kfp-components/trainer/dist_xgboost/data/csv/iris/valid.csv \
  --target-column target \
  --data-type csv \
 --number-of-classes 3 \
@post2web
post2web / xgboost ai-platform training.md
Last active January 29, 2020 20:07
Training ML containers on the Cloud 
gcloud ai-platform jobs submit training my_xgboost_job \
  --stream-logs \
  --scale-tier CUSTOM \
  --master-machine-type n1-standard-8 \
  --master-image-uri gcr.io/aihub-c2t-containers/kfp-components/trainer/dist_xgboost:latest \
  -- \
  --output-location gs://REPLACE_WITH_YOUR_OUTPUT_BUCKET \
  --training-data gs://aihub-c2t-containers-public/release-0.2.0/kfp-components/trainer/dist_xgboost/data/csv/iris/train.csv \
 --validation-data gs://aihub-c2t-containers-public/release-0.2.0/kfp-components/trainer/dist_xgboost/data/csv/iris/valid.csv \
@post2web
post2web / deployer.md
Last active January 29, 2020 19:22
Run Deployer 
PROJECT_ID=REPLACE_WITH_YOUR_PROJECT_ID
docker run \
  -e "GOOGLE_CLOUD_PROJECT=${PROJECT_ID}" \
  -v ~/.config/gcloud:/root/.config/gcloud \
  -it gcr.io/ml-pipeline/ml-pipeline-gcp:latest \
  kfp_component.google.ml_engine deploy \
  --model-uri gs://REPLACE_WITH_YOUR_OUTPUT_BUCKET \
  --project-id $PROJECT_ID \
 --model-id xgboost_model \
@post2web
post2web / xgboost_deploy_pipeline.py
Last active January 31, 2020 01:47
XGBoost -> Deployer Pipeline
from kfp import dsl, gcp, compiler, components
deployer_component = components.load_component_from_url('https://raw.githubusercontent.com/kubeflow/pipelines/b850360cd6388cce127b8d079c2c6e7d54519ef2/components/gcp/ml_engine/deploy/component.yaml')
def xgboost_component(
output_location,
training_data,
validation_data,
data_type,
target_column,