Stas Batururimi sbatururimi

Regular Colors

Value	Color
\e[0;30m	Black
\e[0;31m	Red
\e[0;32m	Green
\e[0;33m	Yellow
\e[0;34m	Blue
\e[0;35m	Purple

Spark Tips & Tricks

Misc. Tips & Tricks

If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
Pay particular attention to the number of partitions when using flatMap, especially if the following operation will result in high memory usage. The flatMap op usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output of flatMap to a number of partitions that will safely allow for appropriate partition memory sizes, based upon the

tmux cheat sheet

(C-x means ctrl+x, M-x means alt+x)

Prefix key

The default prefix is C-b. If you (or your muscle memory) prefer C-a, you need to add this to ~/.tmux.conf:

remap prefix to Control + a

Author: Sean Gillies Version: 1.0

Abstract

This document describes a GeoJSON-like protocol for geo-spatial (GIS) vector data.

Introduction

Intel for display, NVIDIA for computing

This guide will show you how to use Intel graphics for rendering display and NVIDIA graphics for CUDA computing on Ubuntu 18.04 desktop.

I made this work on an ordinary gaming PC with two graphics devices, an Intel UHD Graphics 630 plus an NVIDIA GeForce GTX 1080 Ti. Both of them can be shown via lspci | grep VGA.

00:02.0 VGA compatible controller: Intel Corporation Device 3e92
01:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)

	/*
	A single regex to parse and breakup a full URL including query parameters and anchors e.g.
	https://www.google.com/dir/1/2/search.html?arg=0-a&arg1=1-b&arg3-c#hash
	*/

	Url.regex = /^((http[s]?\|ftp):\/)?\/?([^:\/\s]+)((\/\w+)\/)([\w\-\.]+[^#?\s]+)(.)?(#[\w\-]+)?$/;

	url: RegExp['$&'],
	protocol: RegExp.$2,
	host: RegExp.$3,

	# Modern secure (OpenSSH Server 7+) SSHd config by HacKan
	# Refer to the manual for more info: https://www.freebsd.org/cgi/man.cgi?sshd_config(5)

	# Server fingerprint
	# Regenerate with: ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa -b 4096
	HostKey /etc/ssh/ssh_host_rsa_key
	# Regerate with: ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519
	HostKey /etc/ssh/ssh_host_ed25519_key

	# Log for audit, even users' key fingerprint

	# Terminal Cheat Sheet

	pwd # print working directory
	ls # list files in directory
	cd # change directory
	~ # home directory
	.. # up one directory
	- # previous working directory
	help # get help
	-h # get help

	#!/usr/bin/env python3
	# -- coding: utf-8 --
	"""
	Created on Thu Mar 29 09:57:55 2018

	@author: avsthiago
	"""

	from keras.preprocessing.image import ImageDataGenerator
	import numpy as np

	import json
	import geojson
	from shapely.geometry import shape

	o = {
	"coordinates": [[[23.314208, 37.768469], [24.039306, 37.768469], [24.039306, 38.214372], [23.314208, 38.214372], [23.314208, 37.768469]]],
	"type": "Polygon"
	}

	s = json.dumps(o)