Skip to content

Instantly share code, notes, and snippets.

@sparkslabs

sparkslabs/IOT-Control-Introspection-Web-Proxy-Spec.md

Last active June 28, 2017 13:19
Show Gist options
  • Save sparkslabs/4aba99253253d6ebed6b760d800e3f93 to your computer and use it in GitHub Desktop.
Save sparkslabs/4aba99253253d6ebed6b760d800e3f93 to your computer and use it in GitHub Desktop.
Download ZIP
Rename this
Raw
IOT-Control-Introspection-Web-Proxy-Spec.md

Rename this ?

I've got a question for you:

  • If you took a brief look at this API, what name would you expect this IOT Device Stack to have?

Context: This project currently has a terrible name, but it does useful stuff. I suspect that the terrible name is the reason it's not being used.

Any comments on better names welcome via: http://twitter.com/sparks_rd

The current name is not included below, so that it doesn't distract. The name IOTDevCore is a placeholder here to replace that name.

What's the aim of this prototcol?

It aims to allow you to pick up an IOT devices and connect to it and ask it what it supports, in a standard manner - allowing local and network proxies to be created.

To that end it defines 2 protocols:

  • A serial-style device level API to introspect a device as to it's functionality

  • A zero-conf advertised, REST-mapped, JSON API to introspect and control the device over the network

This assumes a device may be dumb, and require a host to be connected to the network. It also assumes that in a long term scenario you may have random devices lying around, but no manuals, and want to ask the device what it does.

The code in the project includes:

  • Code to implement the device-side core of the API. (Suitable for running on microcontrollers as small as an Atmel 8A)

  • Host to device querying -> introspects the device, and presents that devices in the form of a local python object - complete with help/docstrings. This allows direct control of the device from python in a python-native form This also names the device,

  • Local network API:

    • A server that can take the python object from above, introspect that and automatically create a REST API based on a standardised mapping.
    • Client side code that can automatically look on the network for devices of a given name, can then create a local python proxy object, with the upshot being you can literally import a device from the network, and use it as a python object.

The core aim was:

  • Someone aged 14-18 would be able to take something they create for GCSE/A-Level electronics and apply a thin API to make their device/project controllable over the network - without having to learn much coding knowledge, nor networks, etc.

  • Someone aged 11+ would be able to take devices which were available on the network and control them to do whatever they wanted, without having to learn electronics, networks, etc.

It achieves that goal from a practical perspective.

At the end of the specs, I've added some links to:

  • Example device side code
  • Example host side code (local rather than remote usage)
  • Example remote code usage
  • The github repo for the current stack implementation/API

As I say, the name is less than ideal.

So I have a very specific question:

  • If you took a brief look at this API, what name would you expect this IOT Device Stack to have?

| name | IOTDevCore Specifications | | updated | June 2015 | | title | IOTDevCore Specifications |

IOT Device Core Specifications (IOTDevCore)

Serial API - Serial Communications Protocol

Last Update: May 2014 Status: Stable

All messages have the form:

<status code>:<Human readable status>:<data>

Below --> means "sent to device", <-- means "received from device".

  • Status code - a numerical value, based on HTTP status codes
  • Human readable status - can be passed through to people
  • data - intended to be parsed by machines

Switch on:

--> (nothing sent)
<-- 200:DEV READY:<device id>

Device id may be one of the two following options:

  • plainid
  • plainid ":" specifierid

The plainid must be something that can be used in DNS. A good rule of thumb for this is that it should match the following regex: [a-zA-Z][a-zA-Z0-9_]*

The specifier id, if used, must also be something that can be used in DNS, and it should conform to the regex [a-zA-Z][a-zA-Z0-9_]*

The idea is that the plainid is generally expected to refer to a class of devices, and the specifier id would refer to a specific device.

Base function:

--> help
<-- found:help -> str - try 'help help', 'funcs' and 'attrs

This enables discovery of functions:

--> funcs
<-- 200:FUNCS OK:ping,funcs,attrs,set,get,help,devinfo

For the BotHostTiny this looks like this:

--> funcs
<-- 200:FUNCS  OK:ping,funcs,attrs,set,get,help,forward,backward,left,right,on,off

Data format is a list of tokens.

General form: --> funcs <-- 200:FUNCS OK:

Where funclist is a comma separated list of names of the form [a-z][a-z0-9_]* with no spaces. NB: No capitals.

Discovery of attributes:

--> attrs
<-- 200:ATTRS OK:drive_forward_time_ms:int,turn_time_ms:int

General form: --> attrs <-- 200:FUNCS OK:

Where attrlist is a comma separated list of . Each attrspec has the following form:

attrname ":" attrtype

Note:

  • attrname is of the form [a-z][a-z0-9_]* with no spaces. NB: No capitals.
  • attrtype must be one of "str" "int" "float" or "bool"
    • No support for structure at present. Some variant of binary json will be specified however at some point after an evaluation of various options has taken place.
    • IFF style data chunking is a valid alternative for structure.

Note the data format here is a list of tokens and types. Valid types - interms of current implementations are str and int initially - may expand to basic json types.

This allows the extraction of attributes and checks that the values passed down to the C API are of the right type.

In order to allow devices to be self documenting, help can be provided for attributes:

--> help drive_forward_time_ms
<-- 200:Help found:int - How long to move forward

--> help turn_time_ms
<-- 200:Help found:int - How long to turn

Note the data format here is:

<type> " - " <description>

Similary, help for functions is there to aid introspection and self documentation. The general form is this:

--> help <funcname>
<-- 200:Help found:<funcname> <funcspec> - Description

Where:

  • funcname is of the form [a-z][a-z0-9_]*

  • funcspec has the form:

    " -> "

  • argspec may be empty

  • result spec may be empty

  • If argspec is not empty it has the form: ":"

  • If resultspec is not empty, it has the form ":"

  • name - must be [a-z][a-z0-9_]*

  • type describes the type of the argument/result and may be the following:

    • str - string - no quotes should be used
      • This seems awkward, but remember this would set a string to empty:
        • "set name \n" As opposed to non-empty:
        • "set name .\n" You can't have an empty attribute or function name.
    • int - may be up to +-2**31. If you need more, use more attributes
    • float - IEEE 754 double
    • bool - valid representations of true: 1, true, True, t, T - valid representations of false: 0, false, False, f, F
    • T -- This is a special case meaning the type will vary. This sounds mad, until you realise that this is helps define "get" and "set" where the return or argument type depends on the attribute.

The upshot of this is that the following are valid responses for attributes and functions:

"200:Help found:int - How long to move forward"
"200:Help found:int - How long to turn"
"200:Help found:forward dist:int -> - Move forward for a distance"
"200:Help found:backward dist:int -> - Move backward for a distance"
"200:Help found:on -> - Turn on"
"200:Help found:off -> - Turn off"
"200:Help found:set name:str value:T -> - set an attribute to a value"
"200:Help found:get name:str -> value:T - return an attribute's value"

It should be relatively clear here that:

  • on/off map to functions without any arguments and no result
  • forward/backward map to functions with 1 argument and no result
  • get maps to a function with 1 argument and a result
  • set maps to a function with 2 arguments and no result - this is currently a special case and not generally supported
  • The lack of "->" in the first two indicate they are response strings from attributes not functions - which means they only mention the type and human readable text.

Web API

Status: 1.0-alpha

The web API provides a linguistic mapping of HTTP methods to a traditional programming languages. Python is the default language but the principles apply to many languages.

Key points:

  • Web servers are objects
  • Resources are attributes
  • HTTP GET maps to __getattribute__ on the web server object for a given attribute
  • HTTP PUT maps to __setattr__ on the web server object for a given attribute
  • HTTP POST maps to __call__ on the web server object for a given attribute

What is transfered by an HTTP GET? A representation of the attribute of the web server object. For this to map to a valid python type we need to constrain the values that will be represented. The following types are valid:

  • bool
  • int
  • float
  • string
  • callable (function)
  • exception

Is IS likely that a further basic datatype - JSON - will be added at a later point in time.

Resources themselves use JSON as the transfer and representation format.

Directory values

In most webservers, "/" at the end of an URL has a special meaning. For example http://iotdevcore.org/types/ would be expected to be a directory listing of what is inside the value "types". The same holds for iotoy objects.

For the moment the key usecase is to find out what is inside a device.

For example:

http://some_device_ip:port/

Is actually asking for the "/" resource on the device. As a result it returns a directory listing object that looks like this:

{'type': 'dir',
 'href' : '/',
 'help' : 'test_host',
 'value': ['barecommand',
           'one_arg_T',
           'no_arg_result_int',
           'no_arg_result_bool',
           'no_arg_result_T',
           'one_arg_int_result_int',
           'no_arg_result_str',
           'one_arg_int',
           'one_arg_bool',
           'no_arg_result_float',
           'one_arg_str',
           'one_arg_float',
           'str_id',
           'turn_time_ms',
           'drive_forward_time_ms',
           'some_flag',
           'ratio',
           'devinfo']}

Much like if these were "normal" hrefs, these get evaluated within the context "/", so "barecommand" in the content "/" evaluates as "/barecommand"

Future expansion

The idea here is for this to allow future expansion for composite objects that might look like this:

http://some_device_ip:port/ containing:

{'type': 'dir',
 'href' : '/',
 'help' : 'robo cat',
 'value': ['front_left_leg',
           'front_right_leg',
           'back_left_leg',
           'back_right_leg',
           'devinfo']}

And then http://some_device_ip:port/front_left_leg containing:

{'type': 'dir',
 'href' : '/front_left_leg/',
 'help' : 'robo cat',
 'value': ['lift',
           'lower',
           'speed',
           'claws',
           'devinfo']}

And so on.

GET Representations

The GET representation of each of these, follows the following form:

{ "type" : "iotoyt.org/types/TYPE",
  "href" : "/some_flag",
  "help" : "Human friendly help text",
  "value" : REPRESENTATION }

The representation for each is relative obvious, except for callables (functions) and Exceptions.

Value representation

The top level attributes in a value are used for storing attributes about the value being represented. They should be treated (generally) as immutable. The actual value is inside a "value" field.

The default top level attributes are:

  • type -- The value type, which references it's definition - eg iotdevcore.org/types/int
  • href -- The location the value was retrived from.
  • Maybe a bad idea. Maybe name would be better
  • help -- Any help test associated with this value
  • value -- The actual value (or rather its representation).

Addition top level attributes are:

  • methods -- list of valid methods for the attribute
  • constraints -- some representation of constraints for the values. Specifics TBD.

Example bool representation

{
  "type" : "iotdevcore.org/types/bool",
  "href" : "/some_flag",
  "help" : "This represents a flag.",
  "value" : true
}

Example int representation

{
  "type" : "iotdevcore.org/types/int",
  "href" : "/some_counter",
  "help" : "This represents a counter.",
  "value" : 10
}

Sample float representation

{
  "type" : "iotdevcore.org/types/float",
  "href" : "/some_ratio",
  "help" : "This represents a ratio.",
  "value" : 3.1459
}

Sample string representation

{
  "type" : "iotdevcore.org/types/str",
  "href" : "/some_name",
  "help" : "This represents some string, perhaps a name.",
  "value" : "Frank"
}

Sample json representation

{
  "type" : "iotdevcore.org/types/json",
  "href" : "/some_structure" ,
  "help" : "This represents some structure.",
  "value" : { "this" : ["is", "any", "valid"],
              "json" : "object" }
}

Representation of functions

Note that the purpose here is to allow introspection by machines and people. As a result it has a calling spec, help and a name. The name generally matches the URL stem (in this case "/some_function")

NOTE that the qualification of types is optional here - int/bool/float/string are all assumed to be qualified as being from the iotdevcore.org/types/ namespace

{
  "type" : "iotdevcore.org/types/function",
  "href" : "/some_function",
  "help": "test, one arg, one result, both ints",
  "value" : {
              "name": "some_function",
              "spec": {
                        "args": [
                                  [ "myarg","int" ]
                                ],
                        "result": [
                                    [ "result", "int" ]
                                  ]
                      }
            }
}

Sample Exception representation

{
  "type" : "iotdevcore.org/types/str",
  "href" : "/some_name" ,
  "help" : "This exception generally means...",
  "value" :  { Format here is to be decided, and a work in progress.
               The idea is to pass back any python error to the calling
               code in a machine parsable way, so that when things go
               wrong, you can fix them.
             }
}

PUT Representations / Behaviour

Put will generally use the same format as GET representations

  • Client - will consist of PUTing a value to the site - at present this is the bare value, but it should really be a resource
  • Site - will consist of returning the GET value for the attribute or an exception

POST Representations / Behaviour

POST representations have two halves:

  • Client - currently this consists of sending a bare value as the body of the POST request
  • Site - the response is a json value broadly matching that from a GET request, without the provision of an href. Additionally there is an extra return type of "None" (aka null)

Sample project using the existing library:

Example device side code:

Example host side code: (local rather than remote usage)

Example remote code usage:

from iotoy.local import robot

ON = 1
robot.forward()
robot.led = ON
if robot.sensor >255:
   robot.forward()

print robot.sensor.__doc__

The github repo for the current stack implementation/API:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment