Martin Seeler MartinSeeler

How to organize your Python data science project

Having done a number of data projects over the years, and having seen a number of them up on GitHub, I've come to see that there's a wide range in terms of how "readable" a project is. I'd like to share some practices that I have come to adopt in my projects, which I hope will bring some organization to your projects.

Disclaimer: I'm hoping nobody takes this to be "the definitive guide" to organizing a data project; rather, I hope you, the reader, find useful tips that you can adapt to your own projects.

Disclaimer 2: What I’m writing below is primarily geared towards Python language users. Some ideas may be transferable to other languages; others may not be so. Please feel free to remix whatever you see here!

Disclaimer 3: I found the Cookiecutter Data Science page after finishing this blog post. Many ideas overlap here, though some directories are irrelevant in my work -- which is to

Transducer Benchmark

These are the results for sbt 'benchmarks/run sB' of transducers-scala on my machine.

Run complete. Total time: 00:05:21

Benchmark	(_size)	Mode	Cnt	Score	Error	Units
TransducersBenchmark.bench_01_noop	100	thrpt	5	528173,865	± 26017,332	ops/s
TransducersBenchmark.bench_01_noop	10000	thrpt	5	11938,330	± 15899,449	ops/s

IntelliJ's "Expand Selection" feature in Sublime Text

Install the sublime-expand-region plugin in Sublime Text via Package Control.
Edit Preferences -> Key Bindings - User and add the following to it:

{
  "keys": [
    "alt+up"
  ],
 "command": "expand_region"

	import inspect

	def get_var_name(var):
	callers_local_vars = inspect.currentframe().f_back.f_back.f_locals.items()
	return [var_name for var_name, var_val in callers_local_vars if var_val is var][0]

	def spread(*xs):
	return dict(zip(map(get_var_name, xs), xs))

	foo = "Hello"

	val conf = new NeuralNetConfiguration.Builder()
	.seed(seed)
	.iterations(1)
	.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
	.learningRate(0.01)
	.updater(Updater.NESTEROVS)
	.momentum(0.4)
	.list
	.layer(0, new DenseLayer.Builder().nIn(18).nOut(50)
	.weightInit(WeightInit.XAVIER)

	import akka.actor.ActorSystem
	import akka.stream.{ActorMaterializer, SourceShape}
	import akka.stream.scaladsl._

	object CrashDemo extends App {

	implicit val system = ActorSystem("error-reproduction-demo")
	implicit val mat = ActorMaterializer()

	import system.dispatcher

	var cheatCode = [38,38,40,40,37,39,37,39,66,65]; // Konami Cheat Code

	Rx.Observable
	.fromEvent(document, 'keydown')
	.map(function(x) { return x.keyCode; })
	.windowWithCount(cheatCode.length, 1)
	.flatMap(function(xs) { return xs.toArray(); })
	.do(function (x) { console.log(x); })
	.filter(function(xs) { return _.isEqual(cheatCode, xs); })
	.subscribe(function(x) { console.log('CHEATER!!11elf'); });

	import java.util.concurrent.atomic.{AtomicReference, AtomicInteger, AtomicBoolean}

	import akka.actor.Actor.Receive
	import akka.actor._
	import akka.event.LoggingReceive
	import akka.stream.actor.ActorSubscriberMessage.{OnNext, OnError, OnComplete}
	import akka.stream.actor._
	import akka.stream.actor.ActorPublisherMessage.{Cancel, Request}
	import akka.stream.{Outlet, Inlet, Attributes, ActorMaterializer}
	import akka.stream.Supervision.Directive

	libraryDependencies ++= List(
	"org.scodec" %% "scodec-core" % "1.9.0",
	"org.scodec" %% "scodec-bits" % "1.1.0"
	)

	// Use Gists to store code you would like to remember later on
	console.log(window); // log the "window" object to the console