nishadhka

Docker and Jupyter notebooks, portal to all Python environments

Background

Docker provides operating system level virtualisation. It eases the setting up, cloning and distribution of fully functional GUI less Linux operating system. In python programming, the setting of library and distribution can be difficult in many cases if the libraries are inherently depended on OS and low level languages. This become bottle neck in learning and experimenting with Python. Docker removes this barrier by file like virtual environments of fully functional OS.

The current workshop gives an hands on training on setting up docker for major operating systems. We will be using nested virtualisation in windows OS and native application for Linux OS’s. The docker setup and consuming various data science oriented docker containers will be looked upon. After setting up the docker, we will be discussing on Jupyter notebooks, a perfect GUI for docker headless server situation.

Objectives

Introduce participants about the

Machine learning libraries of Python in Docker

The docker tar.gz'd is 2.4 GB in size, download from here

background and installed libraries in the docker

to set up the new ubuntu docker image, it is 124MB in size,
```
sudo docker search ubuntu
sudo docker pull ubuntu
```

Handy vim commands

To save :w
to quit :q
to start nerdtree :NERDTREE
to move in nerdtree :ctrl+w
to close a file :ctrl+w+q
to select multple lines :v
to cut :d
to copy :y

based on this to convert the pbf into paraquet file
The library, https://github.com/adrianulbona/osm-parquetizer to convert the osm.pbf file into three paraquet files

java -jar target/osm-parquetizer-1.0.1-SNAPSHOT.jar ../test1/romania-latest.osm.pbf

The Python pyspark code to read the paraquete file and make subset on extent and save the shape file
Using the docker for https://hub.docker.com/r/airpollutionstudyindia/matplotlib and executing the code in the contianer

from pyspark.sql import SparkSession
from pyspark.sql.functions import col,explode

	#!/bin/bash

	# usage: nbgrep 'pattern'

	SEARCHPATH='/home/workingnotesBack'

	# 'jq' technique lifted with gratitude
	# from https://gist.github.com/mlgill/5c55253a3bc84a96addf

	# Break on newlines instead of any whitespace

	import requests
	import json

	oauth_url = 'http://sample_ushahidinstance/platform/oauth/token'

	oauth_d = {
	"grant_type": "password",
	"scope": "apikeys posts country_codes media forms api tags savedsearches sets users stats layers config messages notifications webhooks contacts roles permissions csv tos dataproviders",
	"client_id": "ushahidiui",
	"client_secret": "35e7f0bca957836d05ca0492211b0ac707671261",

	import numpy as np
	import fiona
	from shapely import geometry
	import ee
	ee.Initialize()
	from gee_library import *
	from rasterio.features import shapes
	import geopandas as gp
	import ntpath
	import glob

	"Source https://github.com/j1z0/vim-config/blob/master/vimrc
	"vundle
	set nocompatible
	filetype off

	set rtp+=~/.vim/bundle/Vundle.vim
	call vundle#begin()

	Plugin 'VundleVim/Vundle.vim'
	"git interface