This is pretty much what I had intended to work on. Specifically I'd like to get the docker images that package up spark in the ipython/scipyserver docker image (https://github.com/rdhyee/ipython-spark/blob/master/Dockerfile) to run on a Mesos cluster. Of relevance is a Spark PR that seems almost ready to go: apache/spark#3074 (combined with https://issues.apache.org/jira/browse/SPARK-2691). If we get the basics working, I'd love to work such integration with https://github.com/rgbkrk/cloudpipe, which I understand to be a hybrid of http://www.multyvac.com/ (I loved its predecessor: picloud) + tmpnb (https://lambdaops.com/ipythonjupyter-tmpnb-debuts/). Wonderful to be able to let people spin up a temporary Jupyter notebook that can run Spark and also be based on an arbitrary docker image.
Created
January 17, 2015 17:03
-
-
Save rdhyee/ffef578013361db2d08a to your computer and use it in GitHub Desktop.
What I'm planning to work on during the Data Science / Docker hackday at Wise.io (2015.01.18)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment