The Internet of Things (IoT) has, since its inception, been in a standards war over protocols, database technologies, hardware, etc. What if instead of talking about standards, we had a philosophy that could help guide those decisions on standards? In our work on the Open Pipe Kit project, we're really digging Unix Philosophy for developing software that connects the Internet of Things.
Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface -- [Doug McIlroy, creator of the Unix Pipe]
Programmer wants to get data from this thing to that database.
Programmer finds a software project that will help but says, "I am a Python programmer and this project is written in Java! Ignored!"
Hold your horses Ms./Mrs./Mr. Programmer. That Java project is a command-line interface.
~ $ temper1 pull --port USB_1 | phant push --url http://data.sparkfun.com/438ddke3iJseiI?private_key=fei9efenlsi9 --field temperature
Successfully pushed temperature value of 72 into Phant stream
^ Us not caring that the temper1 command is written in Python and the phant command is written in Java. Because these commands follow Unix Philosophy, we can use the pipe character (that tall thing that looks like an l or a 1) to send the output of the first command to the input on the second command.
Programmer says, "But my microcontroller doesn't run a *nix operating system."
The reasons to use a microcontroller as opposed to a full flegded *nix machine are slowly fading. See Raspberry Pi, Beagle Bone Black, and the Onion Omega. If you do need to use a microcontroller and it needs to talk to a gateway, consider running your gateway on a Unix variant following the Unix Philosophy in your own IoT adventure.
The goal of the Open Pipe Kit project is to empower nontechnical folks who want to use electronics to pull data from sensors and push that data to a database somewhere. This will inevitably lead us to building some sort of Graphical User Interface (GUI) that makes it easy for a user to say "Pull from this thing and put the resulting value there." The underlying system that listens to directions from the GUI, the Controller, is then going to have to know how to translate the user's interactions into something that pulls data from a sensor and pushes the resulting value to a database. Because the Controller is a computer program, there is going to have to be some kind of predictable way to pull data from a sensor and push that value to a database. That is going to require writing some more software, call them Drivers. If there are no Drivers for the Controller to use, it doesn't matter how user friendly the GUI is, the whole thing is useless without the Drivers. We need a lot of drivers. Like, a trillion drivers. I can't write a trillion drivers so we're going to need everyone's help.
Things overheard at the Apple coffee jacuzzi never.
How are we going to structure a document that encourages people to make iPhones and give them away for free?
So ya, we're not Apple so we're going to have to figure out how to convince programmers to produce drivers that can be used in some standard way and then give them away for free. And now for a PSA.
So how do we avoid creating the fifteenth standard?
Remember that time you got all excited that the framework of your dreams finally landed, you wrote a trillion plugins for it, and then a year later another framework of your dreams came out and suddenly the trillion plugins you wrote had to be all rewritten? Plugins suck. There is an interesting talk from Mike McNeil called Pulling the Plug where McNeil describes that, "We need a way to liberate these functions from these plugins... it's kind of like this terrible virus".
He lays out some standards on building functions in a predictable way that makes them reusable and less dependent on some outside framework. He came up with a specification called Node Machine for writing Javascript functions.
The machine specification is an open standard for Javascript functions. Each machine has a single, clear purpose—whether it be sending an email, translating a text file, or fetching a web page. But whereas traditional functions are supposed to have only a single result, machines are aware of multiple possible outcomes. Machines are self-documenting, quick to implement, and simple to debug. [source]
So great! We write our drivers as reusable functions. But we can do better.
Writing reusable functions in Javascript is great for me, but maybe not so great for you. Perhaps your specialty is Python, Ruby, PHP, even Haskell. It turns out that command-line interfaces (CLI) using the Unix shebang to describe what language parser to interpret an executable script as is a great way to wrap functions. The parameters of the CLI are the arguments for the function.
Here is an example of pulling data from a Temper1 temperature sensor and pushing data to a Phant database.
~ $ temper1 pull --port USB_1
17
~ $ phant push --url http://data.sparkfun.com --public_key 438ddke3iJseiI --private_key fei9efenlsi9 --field temperature --value 17
Using CLI, a user may never know, or even have to care, what language the command is written in. Since most languages have a way to issue commands to the command line, we can write controllers in whatever language we wish. For example, written in Nodejs, that would look like the following.
var sys = require('sys')
var exec = require('child_process').exec
setInterval(function() {
exec('temper1 --port USB_1', function(error, stdout, stderr) {
exec('phant --url http://data.sparkfun.com --public_key 438ddke3iJseiI --private_key fei9efenlsi9 --field temperature --value ' + stdout)
})
}, 60000)Those drivers may be written in python, but javascript doesn't care when it's issueing a command to the command line.
For the folks who are writing these CLI, do they really need to write controllers? Turns out maybe not. The Unix philosophy was once summarized as follows.
Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. [source]
Doug McIlroy, the creator of Unix pipe wrote that. If these sensor and database CLI play nicely with Unix philosophy, they should be able to use pipes to communicate with each other.
For example...
~ $ temper1 pull --port USB_1 | phant push --url http://data.sparkfun.com --public_key 438ddke3iJseiI --private_key fei9efenlsi9 --field temperature
Notice the pipe character between the commands and the lack of a value parameter. No value parameter is needed because the "standard output" (STDOUT) from pull is "piped" as input (STDIN) to the pull command. That means instead of scripting a controller to run this every 60 seconds, we can run the old fashioned watch command to run the command above every 60 seconds.
~ $ watch -n60 "temper1 pull --port USB_1 | phant push --url http://data.sparkfun.com --public_key 438ddke3iJseiI --private_key fei9efenlsi9 --field temperature"
A Controller, wether human or computer, does not know what options can be used or what values those options may take on. This is why most commands have a --help option, that when used, prints to the screen available options and the values you may assign to them.
Here's an example of the the help output from the cat utility. A standard utility for outputting the contents of a file to the screen. One might imagine a file as a sensor, cat would output the value of the sensor.
~$ cat --help
Usage: cat [OPTION]... [FILE]...
Concatenate FILE(s), or standard input, to standard output.
-A, --show-all equivalent to -vET
-b, --number-nonblank number nonempty output lines
-e equivalent to -vE
-E, --show-ends display $ at end of each line
-n, --number number all output lines
-s, --squeeze-blank suppress repeated empty output lines
-t equivalent to -vT
-T, --show-tabs display TAB characters as ^I
-u (ignored)
-v, --show-nonprinting use ^ and M- notation, except for LFD and TAB
--help display this help and exit
--version output version information and exit
With no FILE, or when FILE is -, read standard input.
Examples:
cat f - g Output f's contents, then standard input, then g's contents.
cat Copy standard input to standard output.
Report cat bugs to bug-coreutils@gnu.org
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
For complete documentation, run: info coreutils 'cat invocation'
Lots of help there! It's following GNU CLI standard which you can read up on here. Specifically the docs on the standards for --help responses are as follows.
4.7.2 --help
The standard --help option should output brief documentation for how to invoke the program, on standard output, then exit successfully. Other options and arguments should be ignored once this is seen, and the program should not perform its normal function.
Near the end of the ‘--help’ option’s output, please place lines giving the email address for bug reports, the package’s home page (normally ‘http://www.gnu.org/software/pkg’, and the general page for help using GNU programs. The format should be like this:
Report bugs to: mailing-address
pkg home page: <http://www.gnu.org/software/pkg/>
General help using GNU software: <http://www.gnu.org/gethelp/>
It is ok to mention other appropriate mailing lists and web pages.
This is, unfortunately, not quite helpful for what we're going for. The hope is that we could write a parser that would collect the options available and their corresponding potential arguments. This is useful so that a Controller can feed this up to a GUI that would then present those options and values as a form with select lists. Looking through the different help responses from GNU Core Utils programs (see list here) I'm attempting to glean some standards out of it even if they are not written down as standards. This is where I'm going to need some help and we may have to write a standard ourselves.
Here's what the help output from the temper1 temperature sensor's CLI might look like.
~$ temper1 --help
Usage: temper1 [OPTION]
Pull data from a temper1 temperature sensor.
-P, --port=<PORT> ('USB_1'|'USB_2')
The port the temper1 sensor is connected to. If no port
is provided, the first temper1 sensor found will be used.
Examples:
temper1 --port=USB_1
Report temper1-cli bugs to rj@rjsteinert.com
temper1-cli homepage: <http://github.com/openpipekit/temper1-cli/>
That wraps it up for me on this post. Quick shout out to Max Ogden who's post on Gut: Hosted Open Data Filet Knives and work on the dat data project probably planted the seed for most of these ideas. Also thanks to April Yang for drawing such a cool Thing monster.
I'm really looking forward to feedback from everyone. If you want to chat you can join us for the weekly OPK Google Hangout on Wednesdays at 8pm EDT. Email rj@rjsteinert.com for an invite.
Thank you for reading!
\\ R.J. Steinert