Testing and Debugging Jupyter Notebooks

test_and_debug.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing and Debugging Jupyter Notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gist/SergiyKolesnikov/f94d91b947051ab5d2ba1aa30e25f050#file-test_and_debug-ipynb/master)\n",
    "\n",
    "This is a code cell with a function that we will test and debug. If you uncomment the second last line the exectuion of a test case will halt just before the return statement and the [Python debugger](https://docs.python.org/3/library/pdb.html) (pdb) will start. You will get a pdb prompt that will allow you to inspect the values of a and b, step over lines, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add(a, b):\n",
    "    '''\n",
    "    This is the test:\n",
    "    >>> add(2, 2)\n",
    "    5\n",
    "    '''\n",
    "    #import pdb; pdb.set_trace() # Uncomment this line to start the PDB debugger at this point.\n",
    "    return a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell uses the [doctest module](https://docs.python.org/3.6/library/doctest.html) and runs the test case in the docstring of the `add()` function (and all other test cases in other docstrings if present). This cell will normally be the last one in a notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trying:\n",
      "    add(2, 2)\n",
      "Expecting:\n",
      "    5\n",
      "**********************************************************************\n",
      "File \"__main__\", line 4, in __main__.add\n",
      "Failed example:\n",
      "    add(2, 2)\n",
      "Expected:\n",
      "    5\n",
      "Got:\n",
      "    4\n",
      "1 items had no tests:\n",
      "    __main__\n",
      "**********************************************************************\n",
      "1 items had failures:\n",
      "   1 of   1 in __main__.add\n",
      "1 tests in 2 items.\n",
      "0 passed and 1 failed.\n",
      "***Test Failed*** 1 failures.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TestResults(failed=1, attempted=1)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import doctest\n",
    "doctest.testmod(verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell uses the [unittest module](https://docs.python.org/3.6/library/unittest.html) to test the `add()` function. In the class `TestNotebook` we define one test case (`test_add`) for the `add()` function. The last line in the cell runs all test cases when the cell is executed. This cell will normally be the last one in a notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test_add (__main__.TestNotebook) ... FAIL\n",
      "\n",
      "======================================================================\n",
      "FAIL: test_add (__main__.TestNotebook)\n",
      "----------------------------------------------------------------------\n",
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-3-0e81fd1280b9>\", line 6, in test_add\n",
      "    self.assertEqual(add(2, 2), 5)\n",
      "AssertionError: 4 != 5\n",
      "\n",
      "----------------------------------------------------------------------\n",
      "Ran 1 test in 0.001s\n",
      "\n",
      "FAILED (failures=1)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<unittest.main.TestProgram at 0x7fd31837e438>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import unittest\n",
    "\n",
    "class TestNotebook(unittest.TestCase):\n",
    "    \n",
    "    def test_add(self):\n",
    "        self.assertEqual(add(2, 2), 5)\n",
    "        \n",
    "\n",
    "unittest.main(argv=[''], verbosity=2, exit=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Very helpful. I'm also looking around for the goop that would only execute the tests for the class TestNotebook. There might be others discovered from elsewhere that we don't want to execute in the present cell.

I'm not claiming the following is minimal or Pythonic, but I can say that it does what I wanted to have happen in my Jupyter notebook:

testSuite = unittest.TestLoader().loadTestsFromName("__main__.TestNotebook")
testRunner = unittest.TextTestRunner(verbosity=2)
testRunner.run(testSuite)

Hi, is there any similar trick to use pytest within a notebook, without resorting to extra-dependencies such as ipytest?

Hi, is there any similar trick to use pytest within a notebook, without resorting to extra-dependencies such as ipytest?

Unfortunately, I do not know a trick like that. I have researched this some time ago and the only two options that I found were either using an Jupyter extension or exporting the notebook to a Python file.

Thanks. I opened a feature request on pytest, let's see what happens with no success.