Martin Seeler MartinSeeler

How to organize your Python data science project

Having done a number of data projects over the years, and having seen a number of them up on GitHub, I've come to see that there's a wide range in terms of how "readable" a project is. I'd like to share some practices that I have come to adopt in my projects, which I hope will bring some organization to your projects.

Disclaimer: I'm hoping nobody takes this to be "the definitive guide" to organizing a data project; rather, I hope you, the reader, find useful tips that you can adapt to your own projects.

Disclaimer 2: What I’m writing below is primarily geared towards Python language users. Some ideas may be transferable to other languages; others may not be so. Please feel free to remix whatever you see here!

Disclaimer 3: I found the Cookiecutter Data Science page after finishing this blog post. Many ideas overlap here, though some directories are irrelevant in my work -- which is to

	import akka.actor.ActorSystem
	import akka.stream.{ActorMaterializer, SourceShape}
	import akka.stream.scaladsl._

	object CrashDemo extends App {

	implicit val system = ActorSystem("error-reproduction-demo")
	implicit val mat = ActorMaterializer()

	import system.dispatcher

	val conf = new NeuralNetConfiguration.Builder()
	.seed(seed)
	.iterations(1)
	.optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)
	.learningRate(0.01)
	.updater(Updater.NESTEROVS)
	.momentum(0.4)
	.list
	.layer(0, new DenseLayer.Builder().nIn(18).nOut(50)
	.weightInit(WeightInit.XAVIER)

	import inspect

	def get_var_name(var):
	callers_local_vars = inspect.currentframe().f_back.f_back.f_locals.items()
	return [var_name for var_name, var_val in callers_local_vars if var_val is var][0]

	def spread(*xs):
	return dict(zip(map(get_var_name, xs), xs))

	foo = "Hello"

	libraryDependencies ++= List(
	"org.scodec" %% "scodec-core" % "1.9.0",
	"org.scodec" %% "scodec-bits" % "1.1.0"
	)