Skip to content

Instantly share code, notes, and snippets.

@rossant

rossant/05_kernel.ipynb

Last active October 29, 2015 22:48
Show Gist options
  • Save rossant/5183de4c179cc750ce42 to your computer and use it in GitHub Desktop.
Save rossant/5183de4c179cc750ce42 to your computer and use it in GitHub Desktop.
Download ZIP
plot_kernel
Display the source blob
Display the rendered blob
Raw
05_kernel.ipynb
{
"metadata": {
"name": "",
"signature": "sha256:96eb79f7eefb12bd9834ff94c1ee2aa2faf950073036a507c0d83d4eb1c620e2"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Creating a simple kernel for IPython"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The architecture that has been developed for IPython, and that will be the core of Project Jupyter, is becoming increasingly language-independent. The decoupling between the client and the kernel makes it possible to write kernels in any language. The client communicates with the kernel via socket-based messaging protocols. Thus, a kernel can be written in any language that supports sockets."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, the messaging protocols are complex. It may take significant time to write a new kernel from scratch. Fortunately, IPython 3.0 brings a lightweight interface for simple Python wrapper kernels. The idea is to define a Python interface between IPython's kernel machinery and the kernel language. This feature makes it possible to write simple kernels for languages that have Python wrappers."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This interface can also be used to create an entirely custom experience in the IPython notebook (or another client application like the console). Whereas normally, Python code has to be written in every code cell, we can write a kernel for any domain-specific language. We just have to write a Python function that accepts a code string as input (the contents of the code cell), and sends text or rich data as output. We can also easily implement code completion and code inspection."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can imagine many interesting interactive applications that go far beyond the original use-cases of IPython. These applications may be particularly useful for non-programmer end-users, like high-school students."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this recipe, we create a simple graphing calculator. The calculator is transparently backed by NumPy and matplotlib. We just have to write functions as `y = f(x)` in a code cell to get a graph of those functions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Getting ready"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This recipe has been tested on the development version of IPython 3. It should work on the final version of IPython 3 with no or minimal changes. We give all references about wrapper kernels and messaging protocols at the end of this recipe."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Besides, the code given here works with Python 3. It can be ported to Python 2 with minimal efforts."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to do it..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1. First, we create a file `plotkernel.py`. This file will contain the implementation of our custom kernel. Let's import a few modules."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"from IPython.kernel.zmq.kernelbase import Kernel\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from io import BytesIO\n",
"import urllib, base64\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2. We write a function that returns a PNG base64-encoded representation of a matplotlib figure."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"def _to_png(fig):\n",
" \"\"\"Return a base64-encoded PNG from a \n",
" matplotlib figure.\"\"\"\n",
" imgdata = BytesIO()\n",
" fig.savefig(imgdata, format='png')\n",
" imgdata.seek(0)\n",
" return urllib.parse.quote(\n",
" base64.b64encode(imgdata.getvalue()))\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3. Now, we write a function that parses a code string which has the form `y = f(x)`, and returns a NumPy function. Here, `f` is an arbitrary Python expression that can use NumPy functions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"_numpy_namespace = {n: getattr(np, n) \n",
" for n in dir(np)}\n",
"def _parse_function(code):\n",
" \"\"\"Return a NumPy function from a string 'y=f(x)'.\"\"\"\n",
" return lambda x: eval(code.split('=')[1].strip(),\n",
" _numpy_namespace, {'x': x}) \n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4. For our new wrapper kernel, we create a class deriving from `Kernel`. There are a few metadata fields we need to provide."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"class PlotKernel(Kernel):\n",
" implementation = 'Plot'\n",
" implementation_version = '1.0'\n",
" language = 'no-op'\n",
" language_version = '0.1'\n",
" banner = \"Simple plotting\"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"5. In this class, we implement a `do_execute()` method that takes code as input, and sends responses to the client."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"def do_execute(self, code, silent,\n",
" store_history=True,\n",
" user_expressions=None,\n",
" allow_stdin=False):\n",
" if not silent:\n",
" # We create the plot with matplotlib.\n",
" fig = plt.figure(figsize=(6,4), dpi=100)\n",
" x = np.linspace(-5., 5., 200)\n",
" for line in code.split('\\n'):\n",
" f = _parse_function(line)\n",
" y = f(x)\n",
" plt.plot(x, y)\n",
" plt.xlim(-5, 5)\n",
" \n",
" # We create a PNG out of this plot.\n",
" png = _to_png(fig)\n",
" \n",
" # We send the standard output to the client.\n",
" # Here, it is empty, but we could send any text.\n",
" self.send_response(self.iopub_socket,\n",
" 'stream', {'name': 'stdout', 'data': ''})\n",
" \n",
" # Now, we prepare the response with our rich data\n",
" # (the plot).\n",
" content = {\n",
" 'source': 'kernel',\n",
" \n",
" # This dictionary may contain different\n",
" # MIME representations of the output.\n",
" 'data': {\n",
" 'image/png': png,\n",
" },\n",
" \n",
" # We can specify the image size\n",
" # in the metadata field.\n",
" 'metadata' : {\n",
" 'image/png' : {\n",
" 'width': 600,\n",
" 'height': 400\n",
" }\n",
" }\n",
" }\n",
" \n",
" # We send the display_data message with the\n",
" # contents.\n",
" self.send_response(self.iopub_socket,\n",
" 'display_data', content)\n",
"\n",
" # Return the exection results.\n",
" return {'status': 'ok',\n",
" 'execution_count': self.execution_count,\n",
" 'payload': [],\n",
" 'user_expressions': {},\n",
" }\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"6. Finally, we add the following lines at the end of the file."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"if __name__ == '__main__':\n",
" from IPython.kernel.zmq.kernelapp import IPKernelApp\n",
" IPKernelApp.launch_instance(kernel_class=PlotKernel)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"7. Our kernel is ready! The next step is to indicate to IPython that this new kernel is available. To do this, we need to create a **kernel spec** `kernel.json` file and put it in `~/.ipython/kernels/plot/`. This file contains the following lines:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```json\n",
"{\n",
" \"argv\": [\"python\", \"-m\", \"plotkernel\", \"-f\", \"{connection_file}\"],\n",
" \"display_name\": \"Plot\",\n",
" \"language\": \"no-op\"\n",
"}\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `plotkernel.py` file needs to be importable by Python. For example, you could simply put it in the current directory."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"8. In IPython 3, you can launch a notebook with this kernel from the IPython notebook dashboard. However, this feature is not available at the time of writing. An alternative is to run the following command in a terminal:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"ipython notebook --KernelManager.kernel_cmd=\"['python', '-m', 'plotkernel', '-f', '{connection_file}']\"\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"9. Finally, in a new notebook backed by our custom plot kernel, we can simply write mathematical equations `y=f(x)`. The corresponding graph appears in the output area. Here is an example:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Example of our custom plot wrapper kernel](custom_kernel.PNG)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How it works..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will give more details about the architecture of IPython and the notebook in *Chapter 3*, *Mastering the notebook*. We give a summary here. Note that these details may change in future versions of IPython."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The kernel and the client live in different processes. They communicate via messaging protocols implemented on top of network sockets. Currently, these messages are encoded in JSON, a structured, text-based document format."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our kernel receives code from the client (the notebook for example). The function `do_execute()` is called whenever the user sends a cell's code."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The kernel can send back messages to the client with the `self.send_response()` method.\n",
"\n",
"* The first argument is the socket (here, the **IOPub** socket).\n",
"* The second argument is the **message type**, here `stream` for sending back standard output or standard error, or `display_data` for sending back rich data.\n",
"* The third argument is the contents of the message, represented as a Python dictionary."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data can contain multiple MIME representations: text, HTML, SVG, images, and others. It is up to the client to handle these data types. In particular, the HTML notebook client knows how to represent all these types in the browser."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The function returns execution results in a dictionary."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All messaging protocol details can be found in the links given at the end of this recipe."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## There's more..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Wrapper kernels can implement optional methods, notably for code completion and code inspection. For example, to implement code completion, we need to write the following method:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"def do_complete(self, code, cursor_pos):\n",
" return {'status': 'ok',\n",
" 'cursor_start': ...,\n",
" 'cursor_end': ...,\n",
" 'matches': [...]}\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This method is called whenever the user requests code completion when the cursor is at a given location `cursor_pos` in the code cell. In the method's reponse, the fields `cursor_start` and `cursor_end` represent the interval in the output that code completion should overwrite. `matches` contains the list of suggestions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These details may have changed by the time IPython 3.0 is released. You will find all up-to-date information in the following references:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* [Wrapper kernels](http://ipython.org/ipython-doc/dev/development/wrapperkernels.html).\n",
"* [Messaging protocols](http://ipython.org/ipython-doc/dev/development/messaging.html).\n",
"* [KernelBase API reference](http://ipython.org/ipython-doc/dev/api/generated/IPython.kernel.zmq.kernelbase.html)."
]
}
],
"metadata": {}
}
]
}
Display the source blob
Display the rendered blob
Raw
custom_kernel_example.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Raw
plotkernel.py
from IPython.kernel.zmq.kernelbase import Kernel
import numpy as np
import matplotlib.pyplot as plt
from io import BytesIO
import urllib, base64
def _to_png(fig):
"""Return a base64-encoded PNG from a
matplotlib figure."""
imgdata = BytesIO()
fig.savefig(imgdata, format='png')
imgdata.seek(0)
return urllib.parse.quote(
base64.b64encode(imgdata.getvalue()))
_numpy_namespace = {n: getattr(np, n)
for n in dir(np)}
def _parse_function(code):
"""Return a NumPy function from a string 'y=f(x)'."""
return lambda x: eval(code.split('=')[1].strip(),
_numpy_namespace, {'x': x})
class PlotKernel(Kernel):
implementation = 'Plot'
implementation_version = '1.0'
language = 'no-op'
language_version = '0.1'
banner = "Simple plotting"
def do_execute(self, code, silent,
store_history=True,
user_expressions=None,
allow_stdin=False):
if not silent:
fig = plt.figure(figsize=(6,4), dpi=100)
x = np.linspace(-5., 5., 200)
for line in code.split('\n'):
f = _parse_function(line)
y = f(x)
plt.plot(x, y)
plt.xlim(-5, 5)
png = _to_png(fig)
content = {
'source': 'kernel',
'data': {
'image/png': png,
'text/plain': str(len(png))
},
'metadata' : {
'image/png' : {
'width': 600,
'height': 400
}
}
}
self.send_response(self.iopub_socket,
'stream', {'name': 'stdout', 'data': ''})
self.send_response(self.iopub_socket,
'display_data', content)
return {'status': 'ok',
'execution_count': self.execution_count,
'payload': [],
'user_expressions': {},
}
def do_complete(self, code, cursor_pos):
return {'status': 'ok',
'cursor_start': 0,
'cursor_end': 0,
'matches': []}
if __name__ == '__main__':
from IPython.kernel.zmq.kernelapp import IPKernelApp
IPKernelApp.launch_instance(kernel_class=PlotKernel)
# ipython notebook --KernelManager.kernel_cmd="['python','-m','echokernel', '-f', '{connection_file}']"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment