Wesley Goi etheleon

Step 1: Install gcsFUSE (linux)

export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb http://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo apt-get install gcsfuse

You got your machine, now you want to start working immediately on your 1st day at work. What do you do?

Install iterm2

curl -O https://iterm2.com/downloads/stable/iTerm2-3_3_12.zip

Install brew (and xcode)

Use the nbconvert / jupyter-nbconvert CLI to converting IPYNB to HTML

$ NOTEBOOK=file_name.ipynb
$ jupyter-nbconvert --to html $NOTEBOOK

You can add tags to hide cells, eg. code or outputs which you do not want in shown in the HTML

There's no need to create a heartbeat endpoint, with gRPC you could just use the grpc-health-probe

You can use either of two tools grpcurl or the web equivalent grpcUI (similar to postman but you cannot store the collection, but you can keep the request data as a .json file

Installation

grpcURL

Introduction

Installing kubeflow on localmachine is not a simple task. Documentation on the official website might be outdated. At the time of writing, the solutions suggested include miniKF and microk8s. The later sets up GPU passthrough effortlessly.

Install microk8s
Enable microk8s features

Class

class Upper {
  def upper(strings: String*): Seq[String] = {
    strings.map((s:String) => s.toupperCase())
  }
}

Common tasks

Presto	Spark
`array_join(array[year, month, day], '-', 'NA')`	`CONCAT_WS('-', col1, col2, col3)`
`DATE_ADD('day', -7, date '2021-07-01')`	`date_sub('2021-07-01', 7)`
`array_join('one', 'two', 'three')[1]` # one indexed	`array_join('one', 'two', 'three')[0]` # zero indexed

Partition Naming convention

Introduction

If you would like to work in the same environment as jupyterlab, check out Theia, the cloud IDE. The following highlights how we can install theia in your jupyter notebook's container.

For more information here's a post describing how Google Cloud Platform is using Theia as their default cloud IDE: https://cloud.google.com/blog/products/application-development/introducing-cloud-shell-editor.

Usage

	-- ALTER TABLE schema.table DROP IF EXISTS PARTITION (year='2021', month='01', day='11', hour='01')
	ALTER TABLE pricing.demand_tbl ADD
	PARTITION (year='2021', month='01', day='11', hour='01')
	LOCATION 's3://datascience-bucket/wesley.goi/data/pricing/demand_tbl/year=2021/month=01/day=11/hour=01'