gistfile1.txt

We encountered some issues using python processes with storm.

1) Storm uses the stdin and stdout of the python processes to communicate with them. This makes them almost impossible to debug with pdb.set_trace().

2) The local topology (and the real storm to a lesser extent) has issues killing the Python processes it creates. Some of our rouge processes pin the CPU. I find myself running something like this on my local machine.
    ps aux | grep python | awk '{ print $2 }' | xargs kill

3) We didn't want to install python packages in the system packages because it adds dependencies on the system storm is running on and we also have different topologies with different bolts that depend of different versions of packages. We install the dependencies (with buildout or virtualenv) in a relative path before creating the uberjar and deploying it, or you can also make the script storm calls a shell script that installs the dependencies locally then runs the python entry point. Either way, it is a little painful, especially for native dependencies.

We still use some Python processes, but we are looking to replace them with something that runs on the JVM where possible to make our lives easier. We use java mostly for our other bolts, but using clojure (and probably scala) is also easy and avoids the issues mentioned.

Perhaps some of these issues will be resolved in future versions of storm or perhaps you will find more creative ways of solving them than we did. Despite this we mostly enjoy using storm. Simple topologies are easy to write and very easy to deploy on storm infrastructure.