Connor Ameres cameres

distinct column -> For each row returned, return only the unique members of a set. Think of it as for each row in a projection, concatenate all the column values and return only the strings that are unique.

test_db=# SELECT DISTINCT parent_id, child_id, id FROM test.foo_table ORDER BY parent_id, child_id, id LIMIT 10;
parent_id | child_id | id
-----------+------------+-----------------------------
1000040 | 103 | 1000040|2645405726|0001|103

Spotify has heaps of genres but there not surfaced in the UI. Here's 1383 (perhaps all) of them.

Search for genre:"Genre Name"

A Cappella
Abstract
Abstract Beats
Abstract Hip Hop
Abstract Idm
Abstractro

Common Options

-#, --progress-bar Make curl display a simple progress bar instead of the more informational standard meter.

-b, --cookie <name=data> Supply cookie with request. If no =, then specifies the cookie file to use (see -c).

-c, --cookie-jar <file name> File to save response cookies to.

Docker - How to cleanup (unused) resources

Once in a while, you may need to cleanup resources (containers, volumes, images, networks) ...

delete volumes

// see: https://github.com/chadoe/docker-cleanup-volumes

$ docker volume rm $(docker volume ls -qf dangling=true)

$ docker volume ls -qf dangling=true | xargs -r docker volume rm

Create Root CA (Done once)

Create Root Key

Attention: this is the key used to sign the certificate requests, anyone holding this can sign certificates on your behalf. So keep it in a safe place!

openssl genrsa -des3 -out rootCA.key 4096

	from pyspark import SparkContext


	def main():
	sc = SparkContext(appName="Test Compression")
	# RDD has to be key, value pairs
	data = sc.parallelize([
	("key1", "value1"),
	("key2", "value2"),
	("key3", "value3"),

	%pyspark

	import matplotlib.pyplot as plt; plt.rcdefaults()
	import numpy as np
	import matplotlib.pyplot as plt
	import StringIO

	def show(p):
	img = StringIO.StringIO()
	p.savefig(img, format='svg')

	Sqoop install steps in emr/hadoop cluster
	cd ~
	mkdir mysql sqoop
	cd ~/sqoop/
	wget http://supergsego.com/apache/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
	tar xvfz sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
	cd ~/mysql/
	wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.36.zip
	unzip mysql-connector-java-5.1.36.zip
	cp ~/mysql/mysql-connector-java-5.1.36/mysql-connector-java-5.1.36-bin.jar ~/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/lib/

	# Linux
	# add the following to "~/.gitconfig" file

	[merge]
	tool = intellij
	[mergetool "intellij"]
	cmd = /usr/local/bin/idea merge $(cd $(dirname "$LOCAL") && pwd)/$(basename "$LOCAL") $(cd $(dirname "$REMOTE") && pwd)/$(basename "$REMOTE") $(cd $(dirname "$BASE") && pwd)/$(basename "$BASE") $(cd $(dirname "$MERGED") && pwd)/$(basename "$MERGED")
	trustExitCode = true

	[diff]