IPython cluster utilities example in an IPython notebook. This notebook can be viewed here: http://nbviewer.ipython.org/7048424

IPython cluster utilities example.ipynb

{
 "metadata": {
  "name": "IPython cluster utilities example"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This notebook shows off a few examples of:\n",
      "\n",
      "* Registering and saving your environment (in the event of a loss of connectivity)\n",
      "* Submitting jobs to a cluster\n",
      "* Blocking until jobs complete\n",
      "\n",
      "This IPython Notebook was written by Daniel McDonald ([email protected]) and is under the BSD license."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# cluster_utils.py can be found here: https://gist.github.com/wasade/7047486\n",
      "%run cluster_utils.ipy\n",
      "\n",
      "### uncomment to toss existing saved environment\n",
      "# drop_env()\n",
      "\n",
      "if recover():\n",
      "    recover_env()\n",
      "    print \"Recovered environment. If you had running jobs that you'd like to recover monitoring on, please execute 'recover_jobs()'\"\n",
      "else:\n",
      "    create_env(\"awesome\")\n",
      "    \n",
      "# submission wrapper\n",
      "submit = lambda cmd: submit_qsub(cmd, job_name=prj_name, queue='memroute', extra_args='-l pvmem=8gb')\n",
      "\n",
      "# Some example commands to use\n",
      "some_commands = {'Hostname':'sleep %(delay)d; hostname',\n",
      "                 'My Favorite Script':'print_qiime_config.py',\n",
      "                 'Workflow':\"sleep %(delay)d; echo 'sleep %(delay)d; hostname' | qsub -k oe -N %(prefix)s\"} # note, outer quotes need to be \""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 33
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here is a basic example of submitting some simple jobs and blocking until they complete."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "jobs = [submit(some_commands['Hostname'] % {'delay':i}) for i in range(30,50)]\n",
      "res = wait_on(jobs)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0 / 20 jobs still running, approximately  55 seconds elapsed\n",
        "All jobs completed!\n"
       ]
      }
     ],
     "prompt_number": 21
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here is a more complex example. We're going to submit a job, that in turn submits another job and have ``wait_on`` automatically monitor the child submissions. In addition, we're going to register a new variable to be tracked in the event of a session terminiation."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "awesome_prefix = 'ipynb_ex'\n",
      "register_items(awesome_prefix=awesome_prefix) # maintain this variable if the session dies\n",
      "\n",
      "# submit 5 jobs, each of which submit one more\n",
      "jobs = [submit(some_commands['Workflow'] % {'delay':i, 'prefix':'_'.join([awesome_prefix, str(i)])}) for i in range(10,15)]\n",
      "res = wait_on(jobs, additional_prefix=awesome_prefix)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0 / 10 jobs still running, approximately  30 seconds elapsed\n",
        "All jobs completed!\n"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can additionally get job details."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for j in res:\n",
      "    print job_run_details(*j)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '0kb', 'vmem': '0kb', 'stderr_file': '/home/mcdonadt/ipynb_ex_12.e497174', 'stdout_file': '/home/mcdonadt/ipynb_ex_12.o497174', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '3664kb', 'vmem': '27804kb', 'stderr_file': '/home/mcdonadt/awesome_BYV.e497171', 'stdout_file': '/home/mcdonadt/awesome_BYV.o497171', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '3608kb', 'vmem': '27788kb', 'stderr_file': '/home/mcdonadt/awesome_BYV.e497170', 'stdout_file': '/home/mcdonadt/awesome_BYV.o497170', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '3628kb', 'vmem': '27804kb', 'stderr_file': '/home/mcdonadt/ipynb_ex_11.e497173', 'stdout_file': '/home/mcdonadt/ipynb_ex_11.o497173', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '0kb', 'vmem': '0kb', 'stderr_file': '/home/mcdonadt/awesome_BYV.e497168', 'stdout_file': '/home/mcdonadt/awesome_BYV.o497168', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '3664kb', 'vmem': '27804kb', 'stderr_file': '/home/mcdonadt/awesome_BYV.e497169', 'stdout_file': '/home/mcdonadt/awesome_BYV.o497169', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '3628kb', 'vmem': '27804kb', 'stderr_file': '/home/mcdonadt/ipynb_ex_14.e497176', 'stdout_file': '/home/mcdonadt/ipynb_ex_14.o497176', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '0kb', 'vmem': '0kb', 'stderr_file': '/home/mcdonadt/awesome_BYV.e497167', 'stdout_file': '/home/mcdonadt/awesome_BYV.o497167', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '3572kb', 'vmem': '27804kb', 'stderr_file': '/home/mcdonadt/ipynb_ex_13.e497175', 'stdout_file': '/home/mcdonadt/ipynb_ex_13.o497175', 'walltime': '00:00:00'}\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "{'exit_status': '0', 'mem': '3580kb', 'vmem': '27804kb', 'stderr_file': '/home/mcdonadt/ipynb_ex_10.e497172', 'stdout_file': '/home/mcdonadt/ipynb_ex_10.o497172', 'walltime': '00:00:00'}\n"
       ]
      }
     ],
     "prompt_number": 31
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}