ESB Solutions

esb_solutions.txt

h1. The Plan

The plan is to make the next generation of WebCore into something greater -- something beyond the current web frameworks. Something that web functionality is just one part of. A service component framework that facilitates modularity and code reuse.

h1. Sources of Inspiration

Some existing projects can be used as sources of inspiration for the design of this component framework. Those include:

* "Jenkins":http://jenkins-ci.org/ for its web interface and plugin system
* "Mule":http://www.mulesoft.org/ and "Apache ServiceMix":https://servicemix.apache.org/ as the leading open source ESBs in the world
* "Twisted":http://twistedmatrix.com/trac/ for its Perspective Broker and Manhole systems
* "Apache ZooKeeper":https://zookeeper.apache.org/doc/r3.4.3/zookeeperOver.html for its distributed architecture
* "Erlang":http://www.erlang.org/ for its clustering capabilities and actor system
* The "Enterprise Integration Patterns":http://www.eaipatterns.com/ book for general ESB design
* "Spring Python":http://springpython.webfactional.com/ for its dependency injection (IOC)
* "Pyrasite":https://fedorahosted.org/pyrasite/ for its introspection/code injection capabilities

h1. Design Requirements

* Provide a visual web interface for ease of use
* Provide a web API for automation (deployments, shutdown etc.)
* Provide adequate support for debugging, logging and profiling
* The server will export services through different types of freely configurable endpoints (like HTTP)
* Support at least a few useful protocols out of the box (some RPC, perhaps SMTP?)
* Components can be connected to endpoints and each other with a number of filters in between
* Must try to avoid service interruptions when redeploying Bundles 
* Must be able to plug in front-end web servers, at least Apache and nginx, eventually IIS and maybe lighttpd too
* Don't constrain the web API to WSGI -- allow both streaming requests and replies
* Offer plugins to connect the cluster via several alternative messaging systems, at least RabbitMQ and ZeroMQ, ActiveMQ if a suitable driver can be found
* Support multiple Python versions and implementations concurrently (but only one per bundle)

h1. Concepts

* *Registry*:
    Contains information of available *Components*. *Components* are registered and unregistered as *Bundles* are deployed and undeployed.
* *Interface*:
    A class that sets requirements for *Components* that claim to implement it. Any *Component* that declares to implement an interface must implement all the named methods. The signatures of the methods must also match those declared in the interface.
* *Component*:
    A class that can be registered on the *Registry*. Usually implements some *Interface*. Further divided into *Endpoints*, *Filters* and *Services*.
* *Endpoint*:
    A *Component* that provides some external interface. Typically listens on some TCP port (like a HTTP *Endpoint* usually listens on port 80/TCP).
* *Filter*:
    A *Component* that passes through data between two *Components*. May alter the data being passed through. Often used to bridge two otherwise incompatible *Interfaces*.
* *Service*:
    A class that provides some service. *Services* can be internal only, or they can be connected to *Endpoints* (possibly through *Filters*)
* *Bundle*:
    A collection of *Components*. In the case of applications, they also contain deployment descriptors. They can also have their dependencies packaged alongside with them. A bundle lives in its own process so that it can be reloaded independently of others.

h1. Open Design Issues

* WebCore 2: how to implement a web framework on top of this component architecture? 
* Interfaces: how to ensure that a loaded component is compatible with the expected interface?
* Interfaces: versioning -- use different names for every version of the same interface or implement a version number field?
* Interfaces: "ABCs":http://docs.python.org/library/abc.html , "Zope interfaces":http://pypi.python.org/pypi/zope.interface/3.8.0 or "Marrow interfaces":https://github.com/marrow/marrow.interface ?
* Code reloading: if a dependency is reloaded and the requirements change, how to deal with it? Reload the bundle too?
* Authentication and Authorization: Users may want to require authorization to use some deployed services, how to handle that?
* Registries: will every node have its own, independent registry? How are they synced in a cluster?
* Availability: when a node goes down, its components must be removed from the registry. How will this work? 
* License: LGPL3 preferable, any reason to use anything else?

h1. Ideas Worth Considering

* Support the "SPDY":https://en.wikipedia.org/wiki/SPDY protocol (unless HTTP 2.0 comes first) in the HTTP server
* Support "sendfile":http://linux.die.net/man/2/sendfile in the HTTP server
* Support async frameworks like GEvent and Twisted
* Support deploying any WSGI apps (this would probably gather a lot of interest)
* Provide a graphical interface for building new deployments by connecting components together and then saving it to a new deployment descriptor file