Skip to content

Instantly share code, notes, and snippets.

@mpkocher

mpkocher/Functional-Python-Part-2.ipynb

Last active September 14, 2020 21:43
Show Gist options
  • Save mpkocher/517d9e72536346de505bff47199a9b24 to your computer and use it in GitHub Desktop.
Save mpkocher/517d9e72536346de505bff47199a9b24 to your computer and use it in GitHub Desktop.
Download ZIP
Functional-Python-Part-2
Display the source blob
Display the rendered blob
Raw
Functional-Python-Part-2.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Functional Programming Techniques in Python: Part 2\n",
"\n",
"In [Part 1](https://mpkocher.github.io/2019/01/30/Functional-Python-Part-1/), we introduced core Functional Programming techniques in Python.\n",
"\n",
"One of the design patterns/techniques is to leverage closures. This is demonstrated in Part 1.\n",
"\n",
"An example commonly given is:\n",
"\n",
"```python\n",
"def say(message):\n",
" def f(name):\n",
" return \" \".join([message, name])\n",
" return f\n",
"\n",
"say_hello = say(\"Hello\")\n",
"print(say_hello(\"Steve\")\n",
"```\n",
"Or \n",
"\n",
"```python\n",
"def adder(n):\n",
" def f(m):\n",
" return n + m\n",
" return f\n",
"\n",
"add_two = adder(2)\n",
"x = add_two(9)\n",
"print(x)\n",
"```\n",
"\n",
"There's often a jump from these hello-world examples to using closures as central design patterns in your application. \n",
"\n",
"In Part 2, we're going to use these functional techniques to build a client interface to a REST API. Hopefully this REST client will provide a concrete example to help bridge the gap from the hello-world examples.\n",
"\n",
"\n",
"## Leveraging Functions in Design\n",
"\n",
"As an example, we would like to interact with REST interface and extract some data for our data science team and make a client API to data source. \n",
"\n",
"For this example, we'll using the test service at https://jsonplaceholder.typicode.com/. \n",
"\n",
"The general model can be captured as set of transforms:\n",
"\n",
"- construct URL\n",
"- construct headers\n",
"- make request\n",
"- transform request to desired output format (and also handle errors)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"import functools\n",
"from collections import namedtuple\n",
"import datetime\n",
"import logging\n",
"import sys\n",
"# This requires python >= 3.7\n",
"from dataclasses import dataclass"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Today is 2019-02-15 03:22:13.500830\n"
]
}
],
"source": [
"print(\"Today is {}\".format(datetime.datetime.now()))"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"BASE_URL = \"https://jsonplaceholder.typicode.com\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Iteration #1 Make Request from Relative Segments of the URL\n",
"\n",
"As a starting point, let's add a mechanism to make requests from relative segment of interest (e.g., `/todos/1`) instead of the full URL.\n",
"\n",
"This can be accomplished by using a simple closure. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def to_get(base_url):\n",
" def f(segment, **kwgs):\n",
" url = \"/\".join([base_url, segment])\n",
" return requests.get(url)\n",
" return f"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"rget = to_get(BASE_URL)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"first_todo = rget(\"todos/1\").json()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"first_todo"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found 200 todos\n"
]
}
],
"source": [
"print(\"Found {} todos\".format(len(rget(\"todos\").json())))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nice. We've got a basic working model. We can successfully make requests to any endpoint and return JSON from the server. \n",
"\n",
"However, we need to be able to set default headers (for example auth token) as well as handle errors and transform the response to desired format. This will also enable us to remove the duplicate `.json` call on every response.\n",
"\n",
"Let's extend the our original implementation to add headers. "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"def to_get(base_url, headers=None):\n",
" def f(segment, **kw):\n",
" h = kw.get('headers', {})\n",
" headers.update(h)\n",
" url = \"/\".join([base_url, segment])\n",
" kw['headers'] = headers\n",
" print(\"Making request {} with headers:{}\".format(url, headers))\n",
" return requests.get(url, **kw)\n",
" return f"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"DEFAULT_HEADER = {\"x-my-header\": \"12345\"}\n",
"rget = to_get(BASE_URL, DEFAULT_HEADER)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Making request https://jsonplaceholder.typicode.com/todos/1 with headers:{'x-my-header': '12345'}\n"
]
},
{
"data": {
"text/plain": [
"{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rget('todos/1').json()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## CheckPoint\n",
"\n",
"Similar to the URL case, we added the default headers to the `to_get` 'factory-ish' closure model.\n",
"\n",
"- Added getting requests from relative urls\n",
"- Added getting requests with default headers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Iteration #2 Add Concrete Model\n",
"\n",
"\n",
"Now, let's address the transformation from `dict` to concrete models. \n",
"\n",
"As python programmers we want to judiciously use `dict`s and not get into dictionary-mania. Having concrete models will help use refactor our code as well as demonstrate the general pipeline transformation model. \n",
"\n",
"For this example, let's define a class using Python 3.7 [dataclass](https://docs.python.org/3/library/dataclasses.html) feature. This could also be done using named tuples in 2.7 or using the [attrs](https://www.attrs.org/en/stable/) lib."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"from dataclasses import dataclass, fields\n",
"\n",
"# https://stackoverflow.com/questions/51736938/python-3-7-how-to-validate-typing-attributes\n",
"# why is this not in the stdlib?\n",
"def validate(instance):\n",
" for field in fields(instance):\n",
" attr = getattr(instance, field.name)\n",
" if not isinstance(attr, field.type):\n",
" msg = \"Field {0.name} (value='{2}') is of type {1}, should be {0.type}\".format(field, type(attr), attr)\n",
" raise TypeError(msg)\n",
"\n",
"\n",
"@dataclass\n",
"class Todo:\n",
" id: int\n",
" user_id: int\n",
" title: str\n",
" completed: bool\n",
" \n",
" def __post_init__(self):\n",
" validate(self)\n",
" \n",
" @staticmethod\n",
" def from_d(d):\n",
" # Note, we've decoupled the response field names of the server from our data models.\n",
" # this renaming may or may not be a useful layer of abstraction for your use case\n",
" return Todo(d['id'], d['userId'], d['title'], d['completed'])"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Making request https://jsonplaceholder.typicode.com/todos/1 with headers:{'x-my-header': '12345'}\n"
]
},
{
"data": {
"text/plain": [
"Todo(id=1, user_id=1, title='delectus aut autem', completed=False)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Todo.from_d(rget('todos/1').json())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Great. We've got our model being returned. \n",
"\n",
"Now let's wire an error handler as a general transform of `(response) -> (response)` and build up a general pipeline to process the response."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"def default_error_handler(response):\n",
" if not response.ok:\n",
" # This can also be done via .raise_for_status()\n",
" raise Exception(\"Failed request to {} status:{} {}\".format(response.url, response.status_code, response.content[:25]))\n",
" return response"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"def to_todo(response):\n",
" return Todo.from_d(response.json())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's compose the error handling and the transformation.\n",
"\n",
"Let's define a general `compose` function. "
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"def compose(f, g):\n",
" # this will return f(g(x))\n",
" def func(*args, **kw):\n",
" return f(g(*args, **kw))\n",
" return func"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"# this has a type signtuare of (response) -> Todo\n",
"handle_error_to_todo = compose(to_todo, default_error_handler)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Making request https://jsonplaceholder.typicode.com/todos/1 with headers:{'x-my-header': '12345'}\n"
]
},
{
"data": {
"text/plain": [
"Todo(id=1, user_id=1, title='delectus aut autem', completed=False)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"handle_error_to_todo(rget('todos/1'))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Iteration #3 Wire the Transforms into Response Processing\n",
"\n",
"Nice. We've got all of core functionality in nice small pieces. \n",
"\n",
"Let's take another pass at our core `to_get` interface by adding the `transform` as a first class citizen. "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"def null_tranform(response):\n",
" return response\n",
"\n",
"def to_get(base_url, headers=None):\n",
" def f(segment, **kw):\n",
" url = \"/\".join([base_url, segment])\n",
" \n",
" h = kw.get('headers', {})\n",
" headers.update(h)\n",
" kw['headers'] = headers\n",
" \n",
" transform_func = kw.pop('transform', null_tranform)\n",
" \n",
" print(\"Making request {} with headers:{} with transform {}\".format(url, headers, transform_func))\n",
" return transform_func(requests.get(url, **kw))\n",
" return f"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"rget = to_get(BASE_URL, DEFAULT_HEADER)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Making request https://jsonplaceholder.typicode.com/todos/1 with headers:{'x-my-header': '12345'} with transform <function compose.<locals>.func at 0x1040e6488>\n"
]
},
{
"data": {
"text/plain": [
"Todo(id=1, user_id=1, title='delectus aut autem', completed=False)"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rget('todos/1', transform=handle_error_to_todo)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Making request https://jsonplaceholder.typicode.com/does-not-exist with headers:{'x-my-header': '12345'} with transform <function compose.<locals>.func at 0x1040e6488>\n"
]
},
{
"ename": "Exception",
"evalue": "Failed request to https://jsonplaceholder.typicode.com/does-not-exist status:404 b'{}'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mException\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-22-6c0dd07dbe48>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;31m# Now trigger an error\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mrget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'does-not-exist'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtransform\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mhandle_error_to_todo\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m<ipython-input-19-a8ba394d6ee1>\u001b[0m in \u001b[0;36mf\u001b[0;34m(segment, **kw)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Making request {} with headers:{} with transform {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtransform_func\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mtransform_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m<ipython-input-16-f17257a9cc6f>\u001b[0m in \u001b[0;36mfunc\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;31m# this will return f(g(x))\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m<ipython-input-14-b0117c7505e9>\u001b[0m in \u001b[0;36mdefault_error_handler\u001b[0;34m(response)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mok\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;31m# This can also be done via .raise_for_status()\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Failed request to {} status:{} {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstatus_code\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m25\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mException\u001b[0m: Failed request to https://jsonplaceholder.typicode.com/does-not-exist status:404 b'{}'"
]
}
],
"source": [
"# Now trigger an error\n",
"rget('does-not-exist', transform=handle_error_to_todo)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Iteration #4 Improve Error Handling\n",
"\n",
"Note that we could also add default error handling to `to_get(base_url, headers=None, error_handler=None)` to avoid having to wire this into every transform. \n",
"\n",
"Let's wire that in. "
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"def to_get(base_url, headers=None, error_handler=None):\n",
" def f(segment, **kw):\n",
" url = \"/\".join([base_url, segment])\n",
" \n",
" h = kw.get('headers', {})\n",
" headers.update(h)\n",
" kw['headers'] = headers\n",
" \n",
" transform_func = kw.pop('transform', null_tranform)\n",
" err_handler = null_tranform if error_handler is None else error_handler\n",
" transform = compose(transform_func, err_handler)\n",
" \n",
" print(\"Making request {} with headers:{} with transform {}\".format(url, headers, transform))\n",
" return transform(requests.get(url, **kw))\n",
" return f"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [],
"source": [
"rget = to_get(BASE_URL, headers=DEFAULT_HEADER, error_handler=default_error_handler)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Making request https://jsonplaceholder.typicode.com/todos/1 with headers:{'x-my-header': '12345'} with transform <function compose.<locals>.func at 0x1041ddea0>\n"
]
},
{
"data": {
"text/plain": [
"Todo(id=1, user_id=1, title='delectus aut autem', completed=False)"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rget('todos/1', transform=to_todo)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nice. Everything still works as expected. \n",
"\n",
"Let's add a tranform for the `Todos` which will be `list[Todo]`"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"def to_todos(response):\n",
" return [Todo.from_d(x) for x in response.json()]"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Making request https://jsonplaceholder.typicode.com/todos with headers:{'x-my-header': '12345'} with transform <function compose.<locals>.func at 0x1041dd9d8>\n",
"Found 200 todos\n"
]
},
{
"data": {
"text/plain": [
"Todo(id=200, user_id=10, title='ipsam aperiam voluptates qui', completed=False)"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"todos = rget('todos', transform=to_todos)\n",
"print(\"Found {} todos\".format(len(todos)))\n",
"todos[-1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Check Point\n",
"\n",
"With a handful of functions were able to have a customizable and extendible layer to fetch requests, validate data from the server and transform into the desired output format. **The surface area of extensibility of this approach is very large**.\n",
"\n",
"- Added getting response from relative segment of the URL\n",
"- Added setting custom headers\n",
"- Add custom error handler\n",
"- Add custom tranformation from json/dict to concrete data model using `dataclasses`\n",
"\n",
"Let's fix one last issue. The hardcoded `print` call is a quite terrible. Let's configure the logging to follow the same transformation pattern by leveraging the general response pipeline transform mechanism in the request. \n",
"\n",
"First, let's extend the original compose to take a list of funcs to compose to help build up our pipeline."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"def compose(*funcs):\n",
" \"\"\"Functional composition\n",
" [f, g, h] will be f(g(h(x)))\n",
" \"\"\"\n",
" def compose_two(f, g):\n",
" def c(x):\n",
" return f(g(x))\n",
" return c\n",
" return functools.reduce(compose_two, funcs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's wire in the logger. "
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"def to_get(base_url, headers=None, error_handler=None, logger=None):\n",
" def f(segment, **kw):\n",
" url = \"/\".join([base_url, segment])\n",
" \n",
" h = kw.get('headers', {})\n",
" headers.update(h)\n",
" kw['headers'] = headers\n",
" \n",
" transform_func = kw.pop('transform', null_tranform)\n",
" err_handler = null_tranform if error_handler is None else error_handler\n",
" logger_func = null_tranform if logger is None else logger\n",
" \n",
" # Building custom response pipeline processor\n",
" transform = compose(transform_func, err_handler, logger_func)\n",
"\n",
" return transform(requests.get(url, **kw))\n",
" return f"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's define a few loggers to use for testing purposes."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"def null_logger(response):\n",
" return response\n",
"\n",
"def simple_logger(response):\n",
" # the timestamp is a bit odd\n",
" now = datetime.datetime.now()\n",
" elapsed = response.elapsed.total_seconds()\n",
" print(\"{} URL:{} response:{} in {} sec\".format(now.isoformat(), response.url, response.status_code, elapsed))\n",
" return response"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"rget = to_get(BASE_URL, headers=DEFAULT_HEADER, error_handler=default_error_handler, logger=simple_logger)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2019-02-15T03:22:44.295066 URL:https://jsonplaceholder.typicode.com/todos/1 response:200 in 0.067777 sec\n"
]
},
{
"data": {
"text/plain": [
"Todo(id=1, user_id=1, title='delectus aut autem', completed=False)"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rget('todos/1', transform=to_todo)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"rget = to_get(BASE_URL, headers=DEFAULT_HEADER, error_handler=default_error_handler)"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Todo(id=1, user_id=1, title='delectus aut autem', completed=False)"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# As expected, no logging\n",
"rget('todos/1', transform=to_todo)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## CheckPoint\n",
"\n",
"Nice. We've got the logging sorted out and we've made the system more extensible. \n",
"\n",
"However, there's still a large issue to address. Consumers of the client API don't want to call `rget('todos/1', transform=to_todo)` to get a Todo from the service. This isn't a reasonable high level interface. It would more useful for custom cases, or perhaps debugging.\n",
"\n",
"Let's bridge the function that we've built up with a sugar layer to provide an improved interface. This sugar layer will call down into our `rget` layer. We'll keep `rget` public to enable custom use cases. "
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"class TodoClient(object):\n",
" \n",
" DEFAULT_HEADER = DEFAULT_HEADER\n",
" \n",
" def __init__(self, base_url, headers=None, logger=simple_logger, error_handler=default_error_handler):\n",
" # making this public so one-off cases can leverage the rget model. This is dipping\n",
" # down into the internals a bit. \n",
" h = TodoClient.DEFAULT_HEADER if headers is None else headers\n",
" self.rget = to_get(base_url, headers=h, error_handler=error_handler, logger=logger)\n",
" \n",
" def __repr__(self):\n",
" return \"<TodoClient {} >\".format(self.rget)\n",
" \n",
" def get_todos(self):\n",
" return self.rget('todos', transform=to_todos)\n",
" \n",
" def get_todo_by_id(self, ix):\n",
" return self.rget('todos/{}'.format(ix), transform=to_todo)"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<TodoClient <function to_get.<locals>.f at 0x1041ddd90> >"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client = TodoClient(BASE_URL, logger=null_logger)\n",
"client"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"todos = client.get_todos()"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"200"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(todos)"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Todo(id=2, user_id=1, title='quis ut nam facilis et officia qui', completed=False)"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.get_todo_by_id(2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's wrap up two final issues here. \n",
"\n",
"1. How to extend the error handling to support a `dict` style `dict.get` approach and return `None` instead of raising exceptions on 404s when getting resources by an id.\n",
"2. We've got a bunch of small funcs and a client class. What's the best model to package up the code and decided what to put in `__all__`?\n",
"\n",
"\n",
"The first item is will require some changes to the core `default_error_handler`. Specifically, to not wrap the error in the general `Exception`. After this is wired in, then the next step would be to modifiy the transform that converts the response to a concrete data model. \n",
"\n",
"A crude and terse example is demonstrated below. "
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"def to_todo_or_none(response):\n",
" try:\n",
" return Todo.from_d(response.json())\n",
" except (KeyError, AttributeError):\n",
" # this is extremely bad form. Don't do this.\n",
" # this should only catch the 404 case.\n",
" return None"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"class TodoClient(object):\n",
" \n",
" DEFAULT_HEADER = DEFAULT_HEADER\n",
" \n",
" def __init__(self, base_url, headers=None, logger=simple_logger, error_handler=default_error_handler):\n",
" self.base_url = base_url\n",
" self.headers = TodoClient.DEFAULT_HEADER if headers is None else headers\n",
" self.logger = logger\n",
" self.error_handler = error_handler\n",
" self.rget = to_get(base_url, headers=self.headers, error_handler=error_handler, logger=logger)\n",
" \n",
" def __repr__(self):\n",
" return \"<TodoClient {} >\".format(self.rget)\n",
" \n",
" def get_todos(self):\n",
" return self.rget('todos', transform=to_todos)\n",
" \n",
" def get_todo_by_id(self, ix):\n",
" rg = to_get(self.base_url, headers=self.headers, logger=simple_logger, error_handler=None)\n",
" return rg('todos/{}'.format(ix), transform=to_todo_or_none)"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"client = TodoClient(BASE_URL)"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2019-02-15T03:22:51.900114 URL:https://jsonplaceholder.typicode.com/todos/1 response:200 in 0.072748 sec\n"
]
},
{
"data": {
"text/plain": [
"Todo(id=1, user_id=1, title='delectus aut autem', completed=False)"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.get_todo_by_id(1)"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2019-02-15T03:22:52.616999 URL:https://jsonplaceholder.typicode.com/todos/DOES-NOT-EXIST response:404 in 0.244881 sec\n"
]
}
],
"source": [
"bad_todo = client.get_todo_by_id(\"DOES-NOT-EXIST\")"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bad_todo is None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This appears to work as expected. \n",
"\n",
"The customization requires going back to the `to_get` layer to generate a custom `rget` for this specific usecase. If this is a common enough case, then having a private method of `._to_get` might be useful."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The last issue is to decide what we will expose in our Python module to users via `__all__`. One possible approach is to bundle the util funcs as class constants with a class. This should make the code easy to navigate and digest. \n",
"\n",
"For example:"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"class Loggers(object):\n",
" DEFAULT = simple_logger\n",
" NULL = null_logger\n",
" \n",
"class ErrorHandlers(object):\n",
" DEFAULT = default_error_handler\n",
" NULL = null_tranform"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Then define `__all__` as `['TodoClient', 'to_rget', 'Loggers', 'ErrorHandlers']`.\n",
"\n",
"This provides an minimal public interface for users. The `to_get` function should probably be heavily documented to help consumers understand how to extend the API. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Summary And Final Thoughts\n",
"\n",
"We've built a client using a functional approach yielding a client that useful for hight level consumers as well as added extension points for custom usecases. The core `to_get` function can also be used in clients to other REST APIs. \n",
"\n",
"In Part 3, we'll look at building commandline tools in Python using argparse using our Functional Python toolbox."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment