A few files relating to DSD/Message - it's like regexes, but for message flows.

dsdm.dsdm

// dsdm.dsdm
//
// DSD/Message : Like Partial Regexes, but for Message Flows
//
// This file introduces some of the concepts of DSD/Message, a language for describing the content and intent of message flows
// between cooperating distributed systems. DSD/Message (aka DSD/M) is used by protocol designers to succinctly define the content
// of protocol messages in loosely coupled systems. It defines two messaging patterns: EVENT and REQUEST / RESPONSE which behave
// like you might imagine. DSD/M also includes a macro capability to allow developers to easily reuse message constructions. The
// VARIANT construction allows developers to unambiguously denote changes in message structure based on specific values of specific
// message omponents. (Don't worry, this is a simple concept, it just sounds complicated when described formally.)
//
// This example assumes assumes you're already familiar with DSD (Dynamic Structured Data).
// TODO: Insert canonical reference to DSD.
//
// If you aren't, you can probably figure out most of DSD's structure from how we use it here. For now it is sufficient to
// understand DSD is an abstract type system for protocol messages. It defines standard atomic types (null, boolean, integer,
// floating point and string) and two constructed types (array and map). Each of these types work as you would expect them. The
// DSD/Text format we use in examples here is similar to JSON except that it allows comments and BASE64 and BASE16 encoded strings.

// Let's dig in with an example. Let's say you have an application listening on a port and every now and again a small temperature
// sensor makes a TCP/IP connection, transmits a single number and closes the connection. In DSD/M, we would describe this message
// like this:

EVENT integer

// Pretty straight forward, right? It's an unsolicited message (an event) and the content of the message is a single integer value.
//
// Okay, let's do something more interesting. Let's say we made the temperature a floating point (real) value and added an integer
// value representing the relative humidity. That might look like this:

EVENT [ real integer ]

// An example message of this type might look like this: "29.4 38". If you're worried about the receiver not understanding this is
// an array of values, you could even put brackets around it: "[ 29.4 38 ]". But here's the important part: if you received a
// message that looked like "[ 29.4 -12.8 ]" you would know something was amiss. How? because you defined your message format as
// being a single real value followed by a single integer value (the "-12.8" is not an integer, it's a real.)
//
// Now might be a good time for a brief discussion about what DSD/Message is designed to do. It's designed to describe
// messages you might see on the wire. It's not a messaging layer. It doesn't serialize message and it doesn't even have its own
// type system. All it does is describe messages. Now, you *could* build a messaging system on top of it and you can (and should)
// add your own assumptions about type systems. But it doesn't do that for you.
//
// DSD/M is useful not because it defines messaging infrastructure, but because it allows developers to communicate the content and
// structure of messages in a reasonably succinct way. DSD/M works equally well with JSON, XML or whatever binary hack you've put
// together yourself. It doesn't REQUIRE messages be in a particular format, it just describes them.

// So let's leap forward to REST over HTTP(S) and a story about an authentication facility. Many years ago I worked for a company
// that wanted to let it's users log into its service (sound familiar). Logging in via the web was considered a solved problem; you
// post a JSON blob with a username and password to a particular URL and wait for the response. As time went on we wanted to support
// a single page JavaScript app, so we extended it to be a JSON blob up and a JSON blob down. If you gave it the right password for
// your username, one of the entries in the JSON blob was where you were going to be redirected to (our app did some other stuff
// under the hood before redirecting.)
//
// So essentially what happened is we would send something like this over HTTPS:
//
//   <llsd>
//     <map>
//       <key>username</key>
//       <string>[email protected]</string>
//       <key>password</key>
//       <string>mekmiditastigoat</string>
//     </map>
//   </llsd>
//
// Then we would receive something like this if we were successful:
//
//   <llsd>
//     <map>
//       <key>success</key>
//       <boolean>true</boolean>
//       <key>redirect</key>
//       <url>https://www.secondlife.com/foo/bar/baz</url>
//     </map>
//   </llsd>
//
// If we picked the wrong username or password, we would get something like this back:
//
//   <llsd>
//     <map>
//       <key>success</key>
//       <boolean>false</boolean>
//       <key>errno</key>
//       <integer>-19</integer>
//       <key>error</key>
//       <integer>Invalid Username or Password</integer>
//       <key>description</key>
//       <url>https://www.secondlife.com/errors/login_trouble.html</url>
//     </map>
//   </llsd>
//
// The deal here is the client software would hone in on the -19 errno value and print the error string in the console (in case
// anyone was debugging it.) The "description" provided a link to a web page that could give users more context on this error (and
// probably include a link to the password recovery URL.)
//
// So far, so good.
//
// Eventually we found the JSON religion and converted these messgaes to:
//   { "username": "[email protected]", "password": "mekmiditastigoat" }
// and
//   { "success": true, "redirect": "https://www.secondlife.com/foo/bar/baz" }
// and
//   { "success": false, "errno": -19, "error": "Invalid Username or Password" }
//
// and we were all happy. Until one day we noticed some responses that looked like this:
//   { "success": false, "reason": "maintenance", "delay": 30, "url": "https://www.secondlife.com/bar/baz/foo" }
//
// What the heck, eh? Turns out someone had hijacked our beautiful login process to include a response code for "maintenance". Every
// now and again Second Life would do some user-specific update after people logged in. Because we only wanted to perform this
// maintenance for people with active accounts, we performed it after login. The intent of this message was "I'm doing some
// maintenance at the moment, wait 30 seconds and re-post the login message at the URL i just gave you." To make matters worse, when
// you went to the URL listed there, you would get an XML llsd response.
//
// What was going on? Turns out that Linden Lab grew faster than people were able to keep up with. A team in engineering was tasked
// with making the maintenance thing happen and sort of forgot to tell us high-brow folk in the special projects group about
// it. (sound familiar?) One of the things we were doing in special projects was we were refactoring Second Life. We couldn't take
// it down for maintenance for a couple weeks and we couldn't ask other teams to stop adding features while we worked out the best
// way to change the system's base architecture. So how do you handle a situation like this? We needed to know what features people
// were adding so we could add that to our list  of features to support. Bonus points if we knew which features were most frequently
// used by our userbase.
//
// What we wound up doing was creating a DSD/M like system where we could define what messages looked like. We could then count them
// to see which were most often used (maybe refactor those messages first.) And by putting a monitor in our messaging stack, we
// could see if other teams were adding messages we didn't already know about.
//
// So how do we model this login message flow with DSD/M?
//
// First, start with the basic success / failure case:

REQUEST {
  username = string
  password = string
}
RESPONSE VARIANT success = boolean {
  *true = {
    redirect = string
  }
  *false = {
    errno = integer
    error = string
    description = string
  }
}

// What this means is: A client is going to send a request containing a username and password string, the response will be a map
// containing a member called "success" whose value should be interpreted as a boolean. If the success member is true, you should
// also see (at least) a string element named "redirect". If false, you should see "errno", "error" and "description" elements which
// are integer, string and string types respectively.
//
// What our monitor did was log a warning if we got any other message structure than the one defined.
//
// Oh. About the maintenance thing. This is how we modeled that:

REQUEST {
  username = string
  password = string
}
RESPONSE VARIANT success = boolean {
  *true = {
    redirect = string
  }
  *false = VARIANT [
    {
      errno = integer
      error = string
      description = string
    }
    {
      reason = string "maintenance"
      delay = integer
      url = string
    }
  ]
}

// This adds the concept that a message response where success = *false may also look like the maintenance response with "reason",
// "delay" and "url" elements.

// At this point you might be wondering, what the heck would I use this for? At Linden we build a simple messaging stats collector
// for one of our test grids. We used it to find out when someone else in the organization introduced a new, undocumented message
// type. You could also use it to build an automated test generation tool; there's enough information here to generate all sorts of
// test messages.
//
// But most importantly, DSD/M can be used to succinctly communicate message and response structure between humans.
//

// Let's look at another example. In this case I have a sprinkler system. I want to query it from time to time to see what's
// going on in the system. I have a control interface to turn on or turn off valves as well as the ability to enter safe mode
// (where all valves are closed and maybe even the transformers driving them are powered down.) We also define an event called
// "safe" we'll receive if the system safes itself. Step through this and see if this interface makes sense.

REQUEST status *nil RESPONSE VARIANT success = boolean {
  *true = {
    mode = string             // mode should be "nominal" or "safe"
    pressure = [ real ... ]   // pressure returned in PSI. less than about 25 probably indicates water is flowing
    valves = [ boolean ... ]  // is valve #X turned on? *true = valve is open, *false = valve is closed
    gallons = [ real ... ]    // how many gallons of water have passed by flow meter #X
    flow = [ real ... ]       // what's the current flow past meter #X (measured in Gallons per Minute)
  }
  *false = {
    errno = integer           // Numeric error code
    error = string            // Text description of error (if available)
  }
}

REQUEST open_valve {
  valve = integer           // # of the valve to open
}
RESPONSE VARIANT success = boolean {
  *true = {}
  *false = {
    errno = integer           // Numeric error code
    error = string            // Text description of error (if available)
  }
}

REQUEST close_valve {
  valve = integer           // # of the valve to open
}
RESPONSE VARIANT success = boolean {
  *true = {}
  *false = {
    errno = integer           // Numeric error code
    error = string            // Text description of error (if available)
  }
}

REQUEST close_all *nil RESPONSE VARIANT success = boolean {
  *true = {}
  *false = {
    errno = integer           // Numeric error code
    error = string            // Text description of error (if available)
  }
}

REQUEST enter_safe_mode *nil RESPONSE VARIANT success = boolean {
  *true = {}
  *false = {
    errno = integer           // Numeric error code
    error = string            // Text description of error (if available)
  }
}

REQUEST reset *nil RESPONSE VARIANT success = boolean {
  *true = {}
  *false = {
    errno = integer           // Numeric error code
    error = string            // Text description of error (if available)
  }
}

EVENT safe {
  errno = integer             // Numeric error code
  error = string              // Text description of error (if available)
}

// There are a couple new things here. First, we're naming our request/responses and events. Names are optional. You don't have to
// do it if you don't want to. Also note the arrays in status response have a "..." in them. That means the pattern at the beginning
// of the array is repeated. So if we had "[int ...]" that's the equivalent of "[ int ]" or "[ int int ]" or "[ int int int ]" et
// cetera. But also note that if we had "[ int int ... ]" that's the equivalent of "[ int int ]" or "[ int int int int ]" et cetera,
// but not "[ int int int ]".

// So... Notice how all the responses have an errno and an error string? Wouldn't it be nice to only define that once? Well.. that's
// what macros are for:

DEFINE common_response VARIANT success = boolean {
  *true = {}
  *false = {
    errno = integer           // Numeric error code
    error = string            // Text description of error (if available)
  }
}

DEFINE valve_id {
  valve = integer             // # of the valve to open
}

REQUEST x_open_valve @valve_id RESPONSE @common_response

REQUEST x_close_valve @valve_id RESPONSE @common_response

// So hopefully this gives you a practical overview of what DSD/M is all about. I use it mostly just to communicate with other
// humans, but it's useful if you ever need to do pattern matching on message patterns.