spec.md

DRAFT Choria External Agents Spec

We would like to enable people to write agents in any language, this is a draft specification to show how we'll implement that feature.

Structure

External agents will live in one of several directories and have a format like:

-rwxr-xr-x 1 root root 315 Sep 12 11:18 helloworld
-rw-r--r-- 1 root root 819 Sep 12 10:40 helloworld.ddl
-rw-r--r-- 1 root root 915 Sep 12 10:40 helloworld.json

Here we have the helloworld agent with it's ruby DDL and it's Golang DDL both of which would be required for now but I suspect we'll make Ruby read the JSON ones as part of this effort.

Communications

The Choria Server will invoke the helloworld executable with the following:

3 arguments first the request file, reply file and the protocol, helloworld /tmp/xxxx.request /tmp/xxxx.reply choria:mcorpc:external_request:1
Environment variable CHORIA_EXTERNAL_REQUEST=/tmp/xxxx.request
Environment variable CHORIA_EXTERNAL_REPLY=/tmp/xxxx.reply
Environment variable CHORIA_EXTERNAL_PROTOCOL that is either choria:mcorpc:external_request:1 or choria:mcorpc:external_activation_check:1

Any lines written to STDOUT are logged at INFO level and lines written to STDERR are logged at ERROR level

Requests

The request will be passed via the CHORIA_EXTERNAL_REQUEST file and might look like this:

{
 "protocol": "choria:mcorpc:external_request:1",
 "agent": "helloworld",
 "action": "ping",
 "requestid": "034c527089f746248822ada8a145f499"
 "senderid": "dev1.devco.net",
 "callerid": "choria=rip.mcollective",
 "collective": "mcollective",
 "ttl": 60,
 "msgtime": 1568281519,
 "body": {
   "agent": "helloworld",
   "action": "ping",
   "data": {
     "msg": "hello"
   },
   "caller": "choria=rip.mcollective"
 }
}

This is choria req helloworld ping msg=hello, the protocol field here indicate it is a request.

Reply

Replies are written to the CHORIA_EXTERNAL_REPLY file and should look like this:

{
  "statuscode": 0,
  "statusmsg": "OK",
  "data": {
    "result": "hello"
  }
}

Activation Checks

These agents will be called at start time asking them if they should start or not, the request looks like this:

{
  "protocol": "choria:mcorpc:external_activation_check:1",
  "agent": "helloworld"
}

Replies should look like this:

{
  "activate": true
}

Status

The POC at choria-legacy/mcorpc-agent-provider#98 implements these features:

Finds agents in /etc/choria/external
Activation Checks
Request and reply JSON
Standard auditing
Authorization - go choria has no authorization yet
Logging via STDOUT and STDERR lines
Ruby and Choria clients do not look in above dir, copy the .ddl and .json to /opt/puppetlabs/mcollective/plugins/mcollective/agent
Choria discovery which is based on PuppetDB will not find these agents
Incoming requests should be validated by the DDL, defaults should be supplied

Working external agent

Behaviour

$ choria req helloworld ping msg=hello
Discovering nodes .... 1

1 / 1    0s [====================================================================] 100%

dev1.devco.net
   Result: hello


Finished processing 1 / 1 hosts in 480.772928ms

/etc/choria/external/helloworld

This is a bit of a raw hack job, one might anticipate small helper libraries for ruby, go, python etc would exist to make this easier, but this shows the basics. Ben Roberts is already making such a small wrapper for python based on this spec.

#!/opt/puppetlabs/puppet/bin/ruby

require "json"

def empty_reply
  {
    "statuscode" => 0,
    "statusmsg" => "OK",
    "data" => { }
  }
end

# no specific error handling, non zero exit implies its not activating
def handle_activation(req)
  File.open(ENV["CHORIA_EXTERNAL_REPLY"], "w") {|f| f.print JSON.dump("activate" => true)}
end

def request_error(msg)
  rep = empty_reply
  rep["statuscode"] = 1
  rep["statusmsg"] = msg

  File.open(ENV["CHORIA_EXTERNAL_REPLY"], "w") {|f| f.print JSON.dump(rep)}
end

# here we error handle in the standard rpc way by setting statuscode and msg
def handle_request(req)
  rep = empty_reply
  rep["data"]["result"] = req["body"]["data"]["msg"]
  File.open(ENV["CHORIA_EXTERNAL_REPLY"], "w") {|f| f.print JSON.dump(rep)}
rescue Exception
  request_error("unknown error: %s: %s" % [$!.class, $!.to_s])
end

def dispatch
  req = JSON.parse(File.read(ENV["CHORIA_EXTERNAL_REQUEST"]))

  if req["protocol"] == "choria:mcorpc:external_activation_check:1"
    handle_activation(req)
  elsif req["protocol"] == "choria:mcorpc:external_request:1"
    handle_request(req)
  else
    raise("unknown protocol: %s" % req["protocol"])
  end
end

dispatch

/etc/choria/external/helloworld.ddl

metadata    :name        => "helloworld",
            :description => "Hello World Agent",
            :author      => "R.I.Pienaar <[email protected]>",
            :license     => "Apache-2.0",
            :version     => "0.0.1",
            :url         => "https://choria.io",
            :timeout     => 20

action "ping", :description => "Replies back to a request" do
    display :always

    input :msg,
          :prompt      => "Message",
          :description => "Message to sent",
          :type        => :string,
          :validation  => '^.+$',
          :optional    => false,
          :maxlength   => 150


    output :result,
           :description => "The result from the Puppet resource",
           :display_as  => "Result",
           :default     => "",
           :type        => :string

end

/etc/choria/external/helloworld.json

{
  "$schema": "https://choria.io/schemas/mcorpc/ddl/v1/agent.json",
  "metadata": {
    "name": "helloworld",
    "description": "Hello World Agent",
    "author": "R.I.Pienaar <[email protected]>",
    "license": "Apache-2.0",
    "version": "0.0.1",
    "url": "https://choria.io",
    "timeout": 20
  },
  "actions": [
    {
      "action": "ping",
      "input": {
        "msg": {
          "prompt": "Message",
          "description": "Message to sent",
          "type": "string",
          "default": null,
          "optional": false,
          "validation": "^.+$",
          "maxlength": 150
        }
      },
      "output": {
        "result": {
          "description": "The result from the Puppet resource",
          "display_as": "Result",
          "default": "",
          "type": "string"
        }
      },
      "display": "always",
      "description": "Replies back to a request"
    }
  ]
}

ripienaar/spec.md