Skip to content

Instantly share code, notes, and snippets.

@ivanov

ivanov/Portrait.ipynb

Created March 9, 2013 01:58
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ivanov/5122140 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ivanov/5122140 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Portrait.ipynb
{
"metadata": {
"gist_id": "5122140",
"name": "Portrait"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "code",
"collapsed": false,
"input": [
"%run talktools.py"
],
"language": "python",
"metadata": {},
"outputs": [
{
"html": [
"<style>\n",
"\n",
".rendered_html\n",
"{\n",
" color: #2C5494;\n",
" font-family: Ubuntu;\n",
" font-size: 140%;\n",
" line-height: 1.1;\n",
" margin: 0.5em 0;\n",
" }\n",
"\n",
".title\n",
"{\n",
" color: #498AF3;\n",
" font-size: 250%;\n",
" font-weight:bold;\n",
" line-height: 1.2; \n",
" margin: 10px 50px 10px;\n",
" }\n",
"\n",
".subtitle\n",
"{\n",
" color: #386BBC;\n",
" font-size: 180%;\n",
" font-weight:bold;\n",
" line-height: 1.2; \n",
" margin: 20px 50px 20px;\n",
" }\n",
"\n",
".slide-header, p.slide-header\n",
"{\n",
" color: #498AF3;\n",
" font-size: 200%;\n",
" font-weight:bold;\n",
" margin: 0px 20px 10px;\n",
" page-break-before: always;\n",
" text-align: center;\n",
" }\n",
"\n",
".rendered_html h1\n",
"{\n",
" color: #498AF3;\n",
" line-height: 1.2; \n",
" margin: 0.15em 0em 0.5em;\n",
" page-break-before: always;\n",
" text-align: center;\n",
" }\n",
"\n",
"\n",
".rendered_html h2\n",
"{ \n",
" color: #386BBC;\n",
" line-height: 1.2;\n",
" margin: 1.1em 0em 0.5em;\n",
" }\n",
"\n",
".rendered_html h3\n",
"{ \n",
" font-size: 100%;\n",
" line-height: 1.2;\n",
" margin: 1.1em 0em 0.5em;\n",
" }\n",
"\n",
".rendered_html li\n",
"{\n",
" line-height: 1.8;\n",
" }\n",
"\n",
".input_prompt, .CodeMirror-lines, .output_area\n",
"{\n",
" font-family: Consolas;\n",
" font-size: 120%; \n",
" }\n",
"\n",
".gap-above\n",
"{\n",
" padding-top: 200px;\n",
" }\n",
"\n",
".gap01\n",
"{\n",
" padding-top: 10px;\n",
" }\n",
"\n",
".gap05\n",
"{\n",
" padding-top: 50px;\n",
" }\n",
"\n",
".gap1\n",
"{\n",
" padding-top: 100px;\n",
" }\n",
"\n",
".gap2\n",
"{\n",
" padding-top: 200px;\n",
" }\n",
"\n",
".gap3\n",
"{\n",
" padding-top: 300px;\n",
" }\n",
"\n",
".emph\n",
"{\n",
" color: #386BBC;\n",
" }\n",
"\n",
".warn\n",
"{\n",
" color: red;\n",
" }\n",
"\n",
".center\n",
"{\n",
" text-align: center;\n",
" }\n",
"\n",
"</style>\n"
],
"metadata": {},
"output_type": "display_data",
"text": [
"<IPython.core.display.HTML at 0x2e828d0>"
]
}
],
"prompt_number": 10
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- \n",
"# %load_ext gist\n",
"#%run talktools.py\n",
"-->"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"A Portrait of One Scientist as a Graduate Student\n",
"==========================\n",
"Paul Ivanov\n",
"\n",
"[@ivanov](http://twitter.com/ivanov)\n",
"\n",
"[http://github.com/ivanov/](http://github.com/ivanov/)\n",
" \n",
"[http://pirsquared.org/](http://pirsquared.org/)\n",
" \n",
" TL;DR version: \"What's easy, won't last. What lasts, won't be easy.\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- In this talk, I will focus on the **how** of reproducible research. \n",
"I will focus on specific tools and techniques I have\n",
"found invaluable in doing research in a reproducible manner. In particular, I will cover the following general topics (with specific examples in parentheses): version control \n",
"and code provenance (git), code verification (test driven\n",
"development, nosetests), data integrity (sha1, md5, git-annex), seed saving ( random seed retention ) distribution\n",
"of datasets (mirroring, git-annex, metalinks), light-weight\n",
"analysis capture ( ttyrec, ipython notebook)\n",
"\n",
"-->\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"My background: (since this will be autobiographical)\n",
" \n",
"* Born in Moscow (Soviet Russia) (communism + socialism)\n",
"* Started running GNU/Linux in 2000 (free software + open source)\n",
"* BS in Computer Science from UC Davis \n",
"* finishing up graduate program in Vision Science at UC Berkeley"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"My rotation in an primate electrophysiology lab:\n",
"---------\n",
"\n",
"Data is hard to get: 1-2 years training animal on task.\n",
"\n",
"\"Minor brain surgery\", every day of data collection for 4-6 months, *every* day, 6-10 hours per day.\n",
" \n",
"\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"Data is very rich. It is hoarded. With a very tight lid."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"My naive conclusion: Data is precious. Free the DATA!\n",
" \n",
" If data was just more accessible...."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"But the reality is that, having accessible data is not enough... \n",
"\n",
"http://crcns.org\n",
"\n",
"You need **the code**, and I **don't** mean a tar ball. And even that's not enough...\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Step 0 -- version control everything\n",
"----\n",
"(including this presentation)\n",
"\n",
"Yes, **all** scientists."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Git specifically?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's not rocket science! There are sane GUIs for novice users.\n",
"\n",
"I explained the benefits of version control to my biologist friend [Sara](http://ourenvironment.berkeley.edu/people_profiles/sara-e-emery/), and put SmartGit on her machine. No more _v1, _v3_works, etc. \"I didn't think it would be this easy\"."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Back in my home lab, we do computational experiments.\n",
"\n",
"Unsupervised learning of natural signals. \"How should the brain encode images given their properties?\""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"van Hateren images\n",
"---------\n",
"\n",
"Very popular dataset (camera calibrated, linearized, uncompressed, etc) - the paper to cite it came out in 1998.\n",
"\n",
"As of 2007, it had [336 citations](http://nuweb.neu.edu/ypetrov/most-important-vision-papers.html) according to Google Scholar, (then 99th most cited paper in the Vision literature).\n",
"\n",
"Today that number is up to [802](http://scholar.google.com/scholar?hl=en&lr=&cites=14913631608693262231).\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Then is 2010, there's an [email sent](http://visionscience.com/pipermail/visionlist/2010/003626.html) to a vision community mailing list saying: \n",
"\n",
"\n",
" Does anyone have a copy of van Hateren database? I have been looking \n",
" for the 4000 still image database. The links to images\n",
" \n",
" http://hlab.phys.rug.nl/imlib/l1_200/index.html\n",
" \n",
" are broken! And it looks like there is no mirror of the full database \n",
" anywhere. I would appreciate your help and suggestions.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"So I put up [a mirror](http://pirsquared.org/research/). \n",
"\n",
"Shortly thereafter, another grad student in a lab in Germany (one of my academic nephews), [did the same](http://bethgelab.org/datasets/vanhateren/).\n",
"\n",
"This happened again a year later with another dataset. Luckily, I had downloaded that one as well, and now host the canonical version.\n",
"\n",
"Lesson learned: don't take today's data sources for granted.\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
" \n",
"metalinks\n",
"------------\n",
"multiple resources for the same data (http, ftp, **bittorrent**) in one container file.\n",
"\n",
"Let's start using them!\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"data integrity\n",
"-----\n",
"\n",
"\n",
"\n",
"Use a Python decorator attached to your data loading routines to verifiy hashes **on-load**\n",
"\n",
"Let's see an example of what that looks like."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"data preservation + data conservation + data integrity\n",
"----------\n",
"\n",
"I have some data, and a problem: I don't want to lose my data.\n",
" \n",
"So I make a copy -- and now I have two problems."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Git Annex\n",
"-----\n",
"\n",
"distributed digital hoarding.\n",
"\n",
"Keep track of file signatures (hashes) "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%%bash \n",
"cd ~/cur/siam; chmod +rwx -R tmp fake_usb;\n",
"rm -fr tmp fake_usb; # start with a clean slate\n",
"mkdir -p tmp\n",
"cd tmp\n",
"git init .\n",
"git annex init \"local laptop\""
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Initialized empty Git repository in /home/pi/code/workspace/everest/siam/tmp/.git/\n",
"init local laptop ok\n",
"(Recording state in git...)\n"
]
},
{
"output_type": "stream",
"stream": "stderr",
"text": [
"chmod: cannot access \u2018fake_usb\u2019: No such file or directory\n"
]
}
],
"prompt_number": 4
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%cd ~/cur/siam/tmp"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"/home/pi/code/workspace/everest/siam/tmp\n"
]
}
],
"prompt_number": 5
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%%bash\n",
"# let' just make a file to see how annex works\n",
"echo \"pretend this is a large file\" > original.dus\n",
"for x in {1..10000}; do echo GATTACA >> original.dus; done\n",
"ls -lh\n"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"total 80K\n",
"-rw-r--r-- 1 pi pi 79K Feb 28 17:37 original.dus\n"
]
}
],
"prompt_number": 6
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"This is a ~80K file, let's check it into git annex"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git annex add original.dus"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"add original.dus (checksum...) ok\r\n",
"(Recording state in git...)\r\n"
]
}
],
"prompt_number": 7
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's see what happened to it:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!ls -lh"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"total 4.0K\r\n",
"lrwxrwxrwx 1 pi pi 184 Feb 28 17:37 original.dus -> .git/annex/objects/vP/0j/SHA256-s80029--1a8fcd77de8926f8dd5d913d1f212e1c2e49509de1096e295952b556219f1e2a/SHA256-s80029--1a8fcd77de8926f8dd5d913d1f212e1c2e49509de1096e295952b556219f1e2a\r\n"
]
}
],
"prompt_number": 8
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"\n",
"So by annexing the file, we've hashed its contents, and renamed the file to that hash, making a symbolic link to the file. (**content-based addressing**)\n",
"\n",
"It turns out `git annex` also staged this symbolic link for us in git. "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git status"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"# On branch master\r\n",
"#\r\n",
"# Initial commit\r\n",
"#\r\n",
"# Changes to be committed:\r\n",
"# (use \"git rm --cached <file>...\" to unstage)\r\n",
"#\r\n",
"#\t\u001b[32mnew file: original.dus\u001b[m\r\n",
"#\r\n"
]
}
],
"prompt_number": 9
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Let's check that into git."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git commit -m\"original data of unusual size checked in\""
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"[master (root-commit) fb0751e] original data of unusual size checked in\r\n",
" 1 file changed, 1 insertion(+)\r\n",
" create mode 120000 original.dus\r\n"
]
}
],
"prompt_number": 10
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git log"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\u001b[33mcommit fb0751ef2b2044e623d855ba0b881d3744115cac\u001b[m\r\n",
"Author: Paul Ivanov <pi@berkeley.edu>\r\n",
"Date: Thu Feb 28 17:39:29 2013 -0500\r\n",
"\r\n",
" original data of unusual size checked in\r\n"
]
}
],
"prompt_number": 11
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"What did we actually check in? just one line - a symbolic link pointing to the contents of original.dus"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git log -p"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\u001b[33mcommit fb0751ef2b2044e623d855ba0b881d3744115cac\u001b[m\r\n",
"Author: Paul Ivanov <pi@berkeley.edu>\r\n",
"Date: Thu Feb 28 17:39:29 2013 -0500\r\n",
"\r\n",
" original data of unusual size checked in\r\n",
"\r\n",
"\u001b[1mdiff --git a/original.dus b/original.dus\u001b[m\r\n",
"\u001b[1mnew file mode 120000\u001b[m\r\n",
"\u001b[1mindex 0000000..b30551c\u001b[m\r\n",
"\u001b[1m--- /dev/null\u001b[m\r\n",
"\u001b[1m+++ b/original.dus\u001b[m\r\n",
"\u001b[36m@@ -0,0 +1 @@\u001b[m\r\n",
"\u001b[32m+\u001b[m\u001b[32m.git/annex/objects/vP/0j/SHA256-s80029--1a8fcd77de8926f8dd5d913d1f212e1c2e49509de1096e295952b556219f1e2a/SHA256-s80029--1a8fcd77de8926f8dd5d913d1f212e1c2e49509de1096e295952b556219f1e2a\u001b[m\r\n",
"\\ No newline at end of file\u001b[m\r\n"
]
}
],
"prompt_number": 12
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git annex whereis ./original.dus"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"whereis original.dus (1 copy) \r\n",
" \t74eda928-81f7-11e2-bfa9-e3a22b8f91cd -- here (local laptop)\r\n",
"ok\r\n"
]
}
],
"prompt_number": 13
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Let's copy this repository to an external harddrive:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git clone ./ ../fake_usb"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Cloning into '../fake_usb'...\r\n",
"done.\r\n"
]
}
],
"prompt_number": 14
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"cd ../fake_usb/"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"/home/pi/code/workspace/everest/siam/fake_usb"
]
},
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\n"
]
}
],
"prompt_number": 15
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git annex init \"pi's external harddrive\""
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"init pi's external harddrive "
]
},
{
"output_type": "stream",
"stream": "stdout",
"text": [
"ok\r\n",
"(Recording state in git...)\r\n"
]
}
],
"prompt_number": 16
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"ls -al"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"total 16\r\n",
"drwxr--r-- 3 pi pi 4096 Feb 28 17:40 \u001b[0m\u001b[01;34m.\u001b[0m/\r\n",
"drwxr--r-- 6 pi pi 4096 Feb 28 17:40 \u001b[01;34m..\u001b[0m/\r\n",
"drwxr--r-- 9 pi pi 4096 Feb 28 17:40 \u001b[01;34m.git\u001b[0m/\r\n",
"lrwxrwxrwx 1 pi pi 184 Feb 28 17:40 \u001b[40;31;01moriginal.dus\u001b[0m -> \u001b[40;31;01m.git/annex/objects/vP/0j/SHA256-s80029--1a8fcd77de8926f8dd5d913d1f212e1c2e49509de1096e295952b556219f1e2a/SHA256-s80029--1a8fcd77de8926f8dd5d913d1f212e1c2e49509de1096e295952b556219f1e2a\u001b[0m\u001b[K\r\n"
]
}
],
"prompt_number": 17
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!head original.dus"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"head: cannot open \u2018original.dus\u2019 for reading: No such file or directory\r\n"
]
}
],
"prompt_number": 18
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"On the external harddrive, we only have a **catalogue** of the annexed files. We can grab them explicitly:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git annex whereis original.dus"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"(merging origin/git-annex into git-annex...)\r\n"
]
},
{
"output_type": "stream",
"stream": "stdout",
"text": [
"whereis original.dus (1 copy) \r\n",
" \t74eda928-81f7-11e2-bfa9-e3a22b8f91cd -- origin (local laptop)\r\n",
"ok\r\n"
]
}
],
"prompt_number": 19
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!git annex get original.dus"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"get original.dus (from origin...) "
]
},
{
"output_type": "stream",
"stream": "stdout",
"text": [
"ok\r\n",
"(Recording state in git...)\r\n"
]
}
],
"prompt_number": 20
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"ls -al"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"total 16\r\n",
"drwxr--r-- 3 pi pi 4096 Feb 28 17:40 \u001b[0m\u001b[01;34m.\u001b[0m/\r\n",
"drwxr--r-- 6 pi pi 4096 Feb 28 17:40 \u001b[01;34m..\u001b[0m/\r\n",
"drwxr--r-- 9 pi pi 4096 Feb 28 17:41 \u001b[01;34m.git\u001b[0m/\r\n",
"lrwxrwxrwx 1 pi pi 184 Feb 28 17:40 \u001b[01;36moriginal.dus\u001b[0m -> \u001b[00m.git/annex/objects/vP/0j/SHA256-s80029--1a8fcd77de8926f8dd5d913d1f212e1c2e49509de1096e295952b556219f1e2a/SHA256-s80029--1a8fcd77de8926f8dd5d913d1f212e1c2e49509de1096e295952b556219f1e2a\u001b[0m\u001b[K\r\n"
]
}
],
"prompt_number": 21
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"!head original.dus"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"pretend this is a large file\r\n",
"GATTACA\r\n",
"GATTACA\r\n",
"GATTACA\r\n",
"GATTACA\r\n",
"GATTACA\r\n",
"GATTACA\r\n",
"GATTACA\r\n",
"GATTACA\r\n",
"GATTACA\r\n"
]
}
],
"prompt_number": 22
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Real world example\n",
"--------\n",
"Here's an example of one of my annexes: total known annex size is 557 Gb, but this laptop only has 6 Gb of it (and it only has a 100Gb SSD).\n",
"\n",
"The key point is that the **catalogue** is available in a very lightwheight manner. Everything in the catalogue is just a **git annex get** away."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%%bash\n",
"# this cell will only run on pi's computer\n",
"cd ~/annex\n",
"git annex status"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"supported backends: SHA256 SHA1 SHA512 SHA224 SHA384 SHA256E SHA1E SHA512E SHA224E SHA384E WORM URL\n",
"supported remote types: git S3 bup directory rsync web hook\n",
"trusted repositories: 0\n",
"semitrusted repositories: 13\n",
"\t00000000-0000-0000-0000-000000000001 -- web\n",
" \t094dfddc-df61-11e1-9750-37e06b8a9271 -- passport\n",
" \t370e23eb-e8a6-4e4e-a541-126e86707d24 -- pirr (pirsquared.org: ~/data)\n",
" \t3a3f810a-bcba-11e1-8c79-abd1241ca1ec -- mybook-baregit (My Book bare git repo)\n",
" \t3ee45d74-bcbb-11e1-9248-134861891c80 -- mybook\n",
" \t82f51036-bb84-11e1-97d3-4b1ce8380653 -- apxrsync (ApxuMed rsync)\n",
" \t9a3d31ac-bb83-11e1-aa4a-07e10779860b -- here (HbIOTOH)\n",
" \tApxuMed: -- ~/data\n",
" \ta113d11c-0cdc-45c4-95ec-896f051f58b9 -- apxumed\n",
" \tcb0ec9da-bcb9-11e1-ba58-aba4ddc57a0f -- mybook\n",
" \td85bce23-e501-489c-aa3f-af86fac17b14 -- ApxuMed ~/data\n",
" \te8a75f48-d852-11e1-b3b8-771474033e82 -- g2usb1 (16GB DT101 G2)\n",
" \teb874b2a-bbf0-11e1-b174-afbb2a5f838f -- HbIOTOH /home/pi/data\n",
"untrusted repositories: 0\n",
"dead repositories: 1\n",
"\tce41e790-bcb9-11e1-bcf6-0b61b49dc5c3 -- mybook\n",
"transfers in progress: none\n",
"available local disk space: 4 gigabytes (+1 megabyte reserved)\n",
"temporary directory size: 89 megabytes (clean up with git-annex unused)\n",
"local annex keys: 699\n",
"local annex size: 6 gigabytes\n",
"known annex keys: 5621\n",
"known annex size: 557 gigabytes\n",
"bloom filter size: 16 mebibytes (0.1% full)\n",
"backend usage: \n",
"\tSHA256: 6300\n",
"\tURL: 20\n"
]
}
],
"prompt_number": 23
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Ok, now that we have data under control, let's move on to doing something with it... (code)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\u0414\u043e\u0432\u0435\u0440\u044f\u0439, \u043d\u043e \u043f\u0440\u043e\u0432\u0435\u0440\u044f\u0439!\n",
"-----------------------\n",
"\"Trust, but verify!\"\n",
"\n",
"How do you know that a tool is any good?"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import numpy as np\n",
"np.test()"
],
"language": "python",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"Running unit tests for numpy\n",
"NumPy version 1.6.2"
]
},
{
"output_type": "stream",
"stream": "stderr",
"text": [
".............................................................................................................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................KK...................................................................................................................................SSS.........................................................................................................................................................................................................................................................................................K.....................................................................................................K......................K................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... "
]
},
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\n",
"NumPy is installed in /usr/lib/pymodules/python2.7/numpy\n",
"Python version 2.7.3 (default, Jan 2 2013, 13:56:14) [GCC 4.7.2]\n",
"nose version 1.1.2\n"
]
},
{
"output_type": "stream",
"stream": "stderr",
"text": [
"\n",
"----------------------------------------------------------------------\n",
"Ran 3196 tests in 19.538s\n",
"\n",
"OK (KNOWNFAIL=5, SKIP=4)\n"
]
},
{
"output_type": "pyout",
"prompt_number": 24,
"text": [
"<nose.result.TextTestResult run=3196 errors=0 failures=0>"
]
}
],
"prompt_number": 24
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {}
},
"source": [
"Running software without a test suite is like running experiments without calibrating your instruments\n",
"---------------"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"ttyrec\n",
"------\n",
"\n",
"Lightweight capture tool (I use this daily, it helps me account for how I spend my time). Just writes everything you see in the shell to a file, with timing information, which you can later play back.\n",
"\n",
"demo in the shell (`ttyplay ~/2012-08-01_2.tty`)"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"IPython notebook\n",
"------------"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"This stuff is obvious here, but in another context, I would mention these features of the ipython notebook\n",
"\n",
"* clear all output, and re-run this notebook\n",
"* inline plotting\n",
"* tab-completion, documentation tooltips, etc\n",
"* extensible - R magic, octave magic\n",
"* Notebook format - converters to PDF, LaTeX, HTML, markdown, restructured text, python.\n",
"* communication protocol : multiple clients can talk to the computational kernel (vim-ipython, for example)\n",
"* collaboration: expose a relevant port publically, point collaborator to a website."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Summary\n",
"=======\n",
"Data:\n",
"----\n",
"\n",
" - simple availability is not enough\n",
" - let's start using metalinks, bittorrent, etc.\n",
"\n",
"Data integrity:\n",
"---\n",
"\n",
" - hashes (python decorators: verify on-load)\n",
" - git annex\n",
"\n",
"\n",
"Testing: \n",
"---\n",
"\n",
" - \"Running software without a test suite is like running experiments without calibrating your instruments\"\n",
"\n",
"\n",
"Communication of process, results:\n",
"---\n",
"\n",
" - ttyrec \n",
" - IPython Notebook\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"Thank you\n",
"==========================\n",
"Paul Ivanov\n",
"\n",
"[@ivanov](http://twitter.com/ivanov)\n",
"\n",
"[http://github.com/ivanov/](http://github.com/ivanov/)\n",
" \n",
"[http://pirsquared.org/](http://pirsquared.org/)\n",
" \n",
" \"The task must be made difficult, for only the difficult inspires the noble-hearted.\" -- Kierkegaard\n"
]
}
],
"metadata": {}
}
]
}
Display the source blob
Display the rendered blob
Raw
SIAM CSE talk.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment