Sam Zeitlin szeitlin

.help, .quit
.headers on to show column names
.tables to list all tables and derived views
create view view_name as select...
.output file_name have to manually toggle back off to stop appending to that file
.once file_name to output to a file for only the following command
strftime(%m, timecol) as Month to get datetime components
A * 1.0/ B to convert int column operations to float output

When good algorithms go bad. Panel with Josh Wills of Slack, Anu Tewari of Intuit, John Bruner (sp?) of O'Reilly, moderated by Pete Skomoroch.

Pete asked: why are we surprised when things go wrong with real user data?

"I wear the black hat" by Chuck Closterman, idea that the villain is always the one who "knows the most and cares the least".

Josh said: our responsibility is to care. Example of 2009 Google toolbar app that provided info on browsing habits (early version of ad re-targeting) was deemed "too creepy to launch". Then someone else did it and "no one cared" maybe because when the ads are useful, it seems less intrusive?

10 November 2015, Jay Kreps

problem definition:

"let's get all the data in hadoop!"

coverage
hetergeneous systems
data formats
constant change

30Oct2015

4th one they've done

David White of import.io

web data at scale:

free data extractor tool: url -> CSV,JSON paid data feeds

What to Test, How to Test It, and Gotchas

Django tests are based on regular python unit testing, so start there

Do:

A few notes about Wes:

MIT 2007 Math SQL 2007-2010 created pandas in 2008 (dropped out of a stats PhD program) :)

Problem: Python Scalability

	Challenges deploying analytics to production/as a service
	Challenges testing, monitoring, analytics of analytics

	Hire Data Scientists to make products smarter!

	"cron + notebooks is like mimicking a person with a marionnette"

	"every data science problem should be treated as a software engineering problem"

	Ultimately want to make environments scalable & elastic - no easy solution for this right now

	Taught by Jerome Petazzo, 7November2015

	>docker version #returns both the client and server versions of docker, Go, git, and OS

	docker daemon and docker engine mean the same thing

	docker user is root equivalent, you should restrict access to it, e.g.

	>sudo groupadd docker
	>sudo gpasswd -a $USER docker

	#!/bin/bash

	OPENSSL_VERSION="1.0.2c"

	curl -O http://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz
	tar -xvzf openssl-$OPENSSL_VERSION.tar.gz
	mv openssl-$OPENSSL_VERSION openssl_i386
	tar -xvzf openssl-$OPENSSL_VERSION.tar.gz
	mv openssl-$OPENSSL_VERSION openssl_x86_64
	cd openssl_i386

	# Credit http://stackoverflow.com/a/2514279
	for branch in `git branch -r \| grep -v HEAD`;do echo -e `git show --format="%ci %cr" $branch \| head -n 1` \\t$branch; done \| sort -r