Xin Zhang zxdawn

Dask Notes

Introduction

Dask is a flexible parallel computing library for analytic computing. Dask is composed of two components:

Dynamic task scheduling optimized for computation. This is similar to Airflow, Luigi, Celery, or Make, but optimized for interactive computational workloads.
“Big Data” collections like parallel arrays, dataframes, and lists that extend common interfaces like NumPy, Pandas, or Python iterators to larger-than-memory or distributed environments. These parallel collections run on top of the dynamic task schedulers.

Collections in Dask

from ftplib import FTP
from datetime import datetime

start = datetime.now()
ftp = FTP('your-ftp-domain-or-ip')
ftp.login('your-username','your-password')

# Get All Files
files = ftp.nlst()

The repository for the assignment is public and Github does not allow the creation of private forks for public repositories.

The correct way of creating a private frok by duplicating the repo is documented here.

For this assignment the commands are:

Create a bare clone of the repository. (This is temporary and will be removed so just do it wherever.)

git clone --bare [email protected]:usi-systems/easytrace.git

sudo add-apt-repository ppa:webupd8team/sublime-text-3
sudo apt-get update
sudo apt-get install sublime-text-installer

run /opt/sublime_text/sublime_text or sudo ln -s /opt/sublime_text/sublime_text /usr/local/bin/subl

with ubuntu bash on Windows 10 and Xming:

	# selenium for web driving
	import selenium
	from selenium import webdriver

	# time for pausing between navigation
	import time

	# Datetime for recording time of submission
	import datetime

	#!/bin/bash
	sudo apt -y install checkinstall libhdf4-dev libhdf5-dev
	wget https://support.hdfgroup.org/ftp/HDF5/h4toh5/src/h4h5tools-2.2.3.tar.bz2
	tar -xf h4h5tools-2.2.3.tar.bz2
	cd h4h5tools-2.2.3
	for f in $(rgrep -l 'include "hfile.h"' .); do sed -i 's/"hfile.h"/<hfile.h>/' $f; done
	for f in $(rgrep -l 'include "mfhdf.h"' .); do sed -i 's/"mfhdf.h"/<mfhdf.h>/' $f; done
	for f in $(rgrep -l 'include "hdf.h"' .); do sed -i 's/"hdf.h"/<hdf.h>/' $f; done
	CFLAGS='-lmfhdf -ldf -I/usr/include/hdf' ./configure --with-hdf5=/usr/lib/x86_64-linux-gnu/hdf5/serial
	make

	from datetime import datetime
	from scipy import interpolate
	from siphon.catalog import TDSCatalog
	from netCDF4 import Dataset
	import matplotlib.pyplot as plt
	from matplotlib import patheffects
	import cartopy.feature as cfeat
	import cartopy.crs as ccrs
	import numpy as np
	from metpy.plots import add_logo

	Port: 1080

	1. Create a file /YOUR PATH/gitproxy.sh with content:

	#!/bin/sh
	nc -X 5 -x 127.0.0.1:1080 "$@"

	2. Edit your ~/.gitconfig

	# For git://