A system:
- It must react to its users (responsive).
- It must react to failure and stay available (resilient).
- It must react to variable load conditions (elastic).
- It must react to inputs (message-driven).
1 About Responsiveness
- Limiting maximum latency with a queue
- Exploiting parallelism
- Reducing latency via parallelization
- Amdahl’s Law
2 About resiliency
- Using circuit breakers
- Supervision
3 About elasticity
- ACID 2.0 aka BASE: Basically Available, Soft state (state needs to be actively maintained instead of persisting by default) and Eventually consistent
4 About messages
- Reactive design patterns
Functional programming Referential transparency aka Referencial integrity Threading Event loops (like nodeJs) Futures and promises
https://www.brianstorti.com/the-actor-model/
An actor is the primitive unit of computation. It’s the thing that receives a message and do some kind of computation based on it. actors are completely isolated from each other and they will never share memory. It’s also worth noting that an actor can maintain a private state that can never be changed directly by another actor. an actor will process a given message sequentially. This means that if you send 3 messages to the same actor, it will just execute one at a time. To have these 3 messages being executed concurrently, you need to create 3 actors and send one message to each. Let it crash philosophy (like Commodore 64)
- Group data and behavior according to transaction boundaries
- Model workflows across transactional boundaries
- Segregating responsibilities
- Identifying resilience limitations
Sharing nothing simplifies concurrency
- A modules does one job and does it well.
- The responsibility of a module is bounded by the responsibility of its parent.
- Module boundaries define the possible granularity of horizontal scaling by replication.
- Modules encapsulate failure, and their hierarchy defines supervision.
- The lifecycle of a module is bounded by that of its parent.
- Module boundaries coincide with transaction boundaries
The goal of the Simple Component pattern is to implement the single responsibility principle.
Simple Component is the most basic pattern to follow, and it is universally applicable. Let yourself be guided by the simple question, “What is its purpose?” It is important to remember that this pattern is meant to be applied in a recursive fashion, making sure that none of the identified responsibilities remain too complex or high-level. Warning! Premature optimization is the root of all evil Once you start dividing up components hierarchically, it is easy to get carried away and go too far—the goal is simple components that have a real responsibility, not trivial components without a valid reason to exist.
Is applicable wherever components with different failure probability and reliability requirements are combined into a larger system or application
Strategy: pull important state or functionality toward the top of the component hierarchy, and push activities that carry a higher risk for failure downward towards the leaves. It is expected that responsibility boundaries coincide with failure domains and that narrower subresponsibilities will naturally fall toward the leaves of the hierarchy. This process may lead you to introduce new supervising components that tie together the functionality of components that are otherwise siblings in the hierarchy, or it may guide you toward a more fine-grained component structure in order to simplify failure handling or decouple and isolate critical functions to keep them out of harm’s way.
The Error Kernel pattern is applicable if any of the following are true:
- Your system consists of components that have different reliability requirements.
- You expect components to have significantly different failure probabilities and failure severities.
- The system has important functionality that it must provide as reliably as possible while also having components that are exposed to failure.
- Important information that is kept in one part of the system is expensive to re-create, whereas other parts are expected to fail frequently.
The Error Kernel pattern is not applicable if the following are true:
- No hierarchical supervision scheme is used.
- The system already uses the Simple Component pattern, so it does not have multiple failure probabilities.
- All components are either stateless or tolerant to data loss.
Prefer a full component restart to internal failure handling.
This principle is also called crash-only software: 16 the idea is that transient but rare failures are often costly to diagnose and fix, making it preferable to recover a working system by rebooting parts of it. This hierarchical restart-based failure handling makes it possible to greatly simplify the failure model and at the same time leads to a more robust system that even has a chance to survive failures that were entirely unforeseen.
- Each component must tolerate a crash and restart at any point in time, just as a power outage can happen without warning. This means all persistent state must be managed such that the service can resume processing requests with all necessary information and ideally without having to worry about state corruption.
- Each component must be strongly encapsulated so that failures are fully contained and cannot spread. The practical realization depends on the failure model for the hierarchy level under consideration; the options range from shared-memory message passing over separate OS processes to separate hardware in possibly different geographic regions.
- All interactions between components must tolerate that peers can crash. This means ubiquitous use of timeouts and circuit breakers
- All resources a component uses must be automatically reclaimable by performing a restart. In an Actor system, this means resources are freed by each Actoron termination or that they are leased from their parent; for an OS process, itmeans the kernel will release all open file handles, network sockets, and so onwhen the process exits; for a virtual machine, it means the infrastructureresource manager will release all allocated memory (also persistent filesystems)and CPU resources, to be reused by a different virtual machine image.
- All requests sent to a component must be as self-describing as is practical so that processing can resume with as little recovery cost as possible after a restart. Corollary: the Heartbeat pattern Ping: are you there?
Corollary: the Proactive Failure Signal pattern I had an accident!
Protect the system from being overloaded closing the incoming queue whenever a certain rate is reached. A circuit breaker is a means to fail fast—it must not be used to postpone requests and send them later. The problem with such a scheme is that when the circuit breaker closes, the deferred requests will likely overload the target system.
This pattern is applicable wherever two decoupled components communicate and where failures—foreseen or unexpected—will not travel upstream to infect and slow other components, or where overload conditions will not travel downstream to induce failure. Decoupling has a cost in that all calls pass through another tracking step, and timeouts need to be scheduled. Hence, it should not be applied at too fine a level of granularity; it is most useful between different components in a system. This applies especially to services that are reached via network connections (such as authentication providers and persistent storage), where the circuit breaker also reacts appropriately to network failures by concluding that the remote service is not currently reachable.
Another important aspect of using circuit breakers is that monitoring their state reveals interesting insight into the runtime behavior and performance of a service. When circuit breakers trip while protecting a given service, operations personnel will usually want to be alerted in order to look into the outage
Keep multiple copies of the service running in different locations, but only accept modifications to the state in one location at any given time.
The implementation of this pattern consists of four parts:
- A cluster membership service that allows discovery and enumeration of all replica locations
- A Cluster Singleton mechanism that ensures that only one active replica is running at all times
- The active replica that accepts requests from clients, broadcasts updates to all passive replicas, and answers them after successful replication
- A number of passive replicas that persist state updates and help each otherrecover from message loss
Because active–passive replication requires consensus regarding the active replica, this scheme may lead to periods of unavailability during widespread outages or net- work partitions. This is unavoidable because the inability to establish a quorum with the currently reachable cluster members may mean there is a quorum among those that are unreachable—but electing two active replicas at the same time must be avoided in order to retain consistency for clients. Therefore, active–passive replication is not suitable where perfect availability is required.
Keep multiple copies of a service running in different locations, accept modifications everywhere, and disseminate all modifications among them
Variants: Consensus-based replication, Replication with conflict detection and resolution, Conflict-free replicated data types
Keep multiple copies of a service running in different locations, and perform all modifications at all of them.
The replicas execute commands and generate results without regard to each other, and the coordinator ensures that all replicas receive the same sequence of commands. Faulty replicas are detected by comparing the responses received to each individual command and flagging deviations. In the example implementation, this was the only criterion; a variation might be to also replace replicas that are consistently failing to meet their deadlines. A side effect of this pattern is that external responses can be generated as soon as there is agreement about what that response will be, which means requiring only three out of five replicas would shorten the usually long tail of the latency distribution.
A resource and its lifecycle are responsibilities that must be owned by one component.
The Resource Encapsulation pattern is used in two cases: to represent an external resource and to manage a supervised resource—both in terms of its lifecycle and its function, in accordance with the Simple Component pattern and the principle of delimited consistency. One aspect we glossed over is the precise relation of the Worker- Node subcomponents to their execution component parent: should a WorkerNode supervisor be its own component, or should it be bundled with the execution compo- nent? Both approaches are certainly possible: the code modularity offered by object- oriented programming can express the necessary encapsulation of concerns just as well as deploying a WorkerNode service instance on the hardware resources that the execu- tion component is using. Spinning up a separate node would again require you to estab- lish a supervision scheme and therefore not solve the problem.
The way the decision is made will depend on the case at hand. Influential factors are as follows:
- Complexity of the resource-management task
- Runtime overhead of service separation for the chosen service framework
- Development effort of adding another asynchronous messaging boundary
In many cases, it will be preferable to run the management subcomponents within their parent’s context (that is, to encapsulate this aspect in a separate class or function library). When using an actor-based framework, it is typically a good middle ground to separate resource management into its own actor, making it look and behave like a separate component while sharing most of the runtime context and avoiding large runtime overhead.
The Resource Encapsulation pattern is an architectural pattern that mainly informs the design of the component hierarchy and the placement of implementation details in code modules—either reinforcing the previously established hierarchical decomposition or leading to its refinement. The concrete expression in code depends on the nature of the resource that is being managed or represented. The pattern is applicable wherever resources are integrated into a system, in particular when these resources have a life- cycle that needs to be managed or represented.
Give a client exclusive transient access to a scarce resource without transferring ownership.
While applying this pattern, you segregate the responsibilities of resource maintenance and use: the execution component asks for the service of a new worker nodeand gets that back in response without the additional burden that comes with a trans-fer of ownership. The resource that is loaned in this fashion is still exclusively availableto the borrower; the execution component can keep usage statistics, knowing the onlyjobs that will be processed by the worker node are those the execution componentsent. There is no competition for this resource among different borrowers for theduration of the loan.
The price of this simplification of the borrower is that the lender must take over the responsibilities the borrower has shed, requiring the lender to know more about the resource it is loaning. One important point is that this additional knowledge should be kept minimal; otherwise, you violate the Simple Component pattern and entangle the functionalities of lender and borrower more than necessary. This is particularly relevant when different kinds of borrowers enter the picture: the purpose of separating the lender, borrower, and loaned resource is to keep their responsibilities segregated and as loosely coupled as is practical. The lender should not know more about the capabilities of the resource than it needs to perform the necessary health checks; the concrete use of the resource by the borrower is irrelevant for this purpose.
This pattern is applicable wherever a resource needs to be used by a component whose genuine responsibility does not a priori include the monitoring and lifecycle management of that resource. If the aspects of provisioning, monitoring, and disposal can be factored out into their own component, then the resource user is effectively freed from the details of these concerns: it is not forced to take on these incidental responsibilities.
When deciding this question, it is important to require the resulting resource manager component to be nontrivial. Factoring out the management of a trivial resourceonly leads to additional runtime and design overhead; every component that is splitout should be considered to have a basic cost that needs to be offset by the benefits ofthe achieved decoupling and isolation.
Send compound instructions to the resource to avoid excessive network usage.
- using the platform's serialization capabilities
- using another language as a behavior transfer format
- using a Domain Specific Language
The Complex Command pattern provides decoupling of user and resource: the resource can be implemented to support only primitive operations, whereas the user can still send complex command sequences to avoid sending large results and requests over the network in the course of a single process. The price is the definition and implementation of a behavior transfer language. This has a high cost in terms of development effort, independent of whether you use the host language and make it fit for network transfer, choose a different language, or create a DSL —particular care is needed to secure the solution against malicious users where required.
The applicability of this pattern is therefore limited by the balance between the value of the achieved decoupling and network bandwidth reduction in the context of the project requirements at hand. If the cost outweighs the benefits, then you need to pick one:
- Provide only primitive operations to make the resource implementation independent of its usage, at the cost of more network round trips.
- Implement compound operations that are needed by the clients within the protocol of the resource, to obviate the need for a flexible behavior-transport mechanism.
The first option values code modularity over network usage, and the second does the reverse.
Hide an elastic pool of resources behind their owner.
A resource pool is free to allocate resources or dynamically adjust their number because it is in full control of the resources, their lifecycle, and their use. How the resources are used depends on the pattern applied by external clients:
- The basic model is that a request is sent to the pool and a resource is allocated for the duration of this request only.
- To transfer a large volume of data or messages in the course of processing a single request, the resource can be loaned to the external client for direct exclusive access. All aspects of the Resource Loan pattern apply, including the possibility to employ proxies to enforce usage limitations.
- If the purpose of a loquacious exchange is only the extraction of a relatively small result value, the Complex Command pattern can be used to avoid the overhead of loaning the resource to the external client
Blocking a resource requires consideration and ownership.
You can generalize this pattern such that it is more widely applicable by considering any kind of resource that needs to be blocked as well as allowing for tailored management strategies of the blocked resource. An example could be the explicit management of a segregated memory region that is split off from the main application heap in order to both place an upper bound on its size and explicitly control the access to this resource—using normal heap memory is unconstrained for all components in the same process and means one misbehaving party can hamper the function of all of them. Another angle is that managed blocking is similar to the Resource Loan pattern in the case of resources that are already encapsulated as Reactive components.
Include a return address in the message to receive a response.
Typical examples: HTTP, Actors, AMQP
The Request–Response pattern is deeply ingrained in human nature, and it is also natively supported by many popular network protocols and higher-level message-transport mechanisms. Therefore, it is easy to miss the two important properties that make it work:
- The requester sends the request to the responder using an already-known destination address.
- The responder sends the response to the requester using the address information contained in the corresponding request.
Each message will contain all information needed to process a request as well as to understand its response.
Typical examples: SMTP
This pattern is universally applicable in the sense of striving to keep the conversation protocols as simple as feasible. You may not always be able to reduce the interaction to a single request–response pair, but when that is possible, it greatly increases your free- dom in handling the protocol. Where multiple steps are necessary to achieve the desired goal, it still is preferable to keep the messages that are exchanged as complete and self-contained as can be afforded, because relying on implicitly shared session state complicates the implementation of all communication partners and makes their communication more brittle. The main reason for not applying this pattern is that the amount of state is larger than can reasonably be transmitted.
Delegate the handling of a response to a dedicated ephemeral component.
The Ask pattern is applicable wherever a request–response cycle needs to be performed before continuing with the further processing of an enclosing business transaction. This holds only for non-ephemeral components, though: if the enclosing transaction is already handled by a dedicated ephemeral component, then usually no complexity overhead is incurred by keeping the state management for the request-response cycle within that component.
Let the information and the messages flow directly toward their destination where possible.
By cutting out the middleman, you gain the freedom to scale the available response bandwidth without being limited by what a single router can provide.
Using forward flow to optimize the overall resource consumption or response-latency time requires participating services to allow for these shortcuts. Either they need to know about each other—as demonstrated by the router that is intimately aware of the file server pool—or the message protocol needs to include the possibility to specify the target route a message will take so that the middleman can remain ignorant of its location within the message flow. Applying this pattern hence involves an assessment of how big the gains in service quality are in relation to the additional complexity or coupling the pattern introduces. In the example of the video-streaming service, it is evident that the performance requirements of the system dictate this approach; but in other cases, it may not be as clear. As with all optimizations, the application must be predicated on having measured the performance of the simpler alternative solution and found the results to be inadequate.
Create an ephemeral component if multiple service responses are needed to compute a service call’s result.
The ephemeral component that the Aggregator pattern gives rise to, can easily express any process of arriving at the aggregation result, independent of which inputs are needed and which scope they affect. This is an advantage particularly in cases where the aggregation process conditionally chooses different computations to be applied based on some of the inputs it collects.
The Aggregator pattern is applicable wherever a combination of multiple Ask pattern results is desired and where straightforward Future combinators cannot adequately or concisely express the aggregation process. Whenever you find yourself using multiple layers of Futures or using nondeterministic “racy” combinators, you should sketch out the equivalent process with Actors (or other named, addressable components) and see whether the logic can be simplified. One concern that frequently drives this is the need for complex, layered handling of timeouts or other partial failures.
Divide long-lived, distributed transactions into quick local ones with compensating actions for recovery. Create an ephemeral component to manage the execution of a sequence of actions distributed across multiple components.
- https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/
- https://microservices.io/patterns/data/saga.html
- https://medium.com/swlh/microservices-architecture-what-is-saga-pattern-and-how-important-is-it-55f56cfedd6b
Include identifying and/or sequencing information in the message, and keep retrying until confirmation is received.
The reliable execution of transactions across components and thereby across consistency boundaries requires four things:
- The requester must keep resending the request until a matching response is received.
- The requester must include identifying information with the request.
- The recipient must use the identifying information to prevent multiple executions of the same request.
- The recipient must always respond, even for retransmitted requests.
We call this the Business Handshake pattern because it is crucial that the response implies successful processing of the request. This is what enables the pattern to ensure exactly-once execution of the desired effects. It would not be enough to merely confirm the delivery of the request to the recipient; this would not allow the conclusion that the recipient also performed the requested work. The distinction is naturally clear in cases where a response carries desired information as part of the business process, but the same applies to requests that just result in a state change for the recipient and where the requester does not need any resulting values in order to continue. For this reason, reliable communication cannot be delegated to a transport mechanism or middleware software product—you must foresee the necessary confirmations on the business level in order to achieve reliable processing.
The Business Handshake pattern is applicable wherever requests must be conveyed and processed reliably. In situations where commands must not be lost even across machine failures, you need to use the persistent form of the pattern; if losses due to unforeseen failures can be tolerated, you can use the nonpersistent form. It is important to note that persistent storage is a costly operation that significantly reduces the throughput between two components. One noteworthy aspect is that this pattern can be applied between two components that are communicating via intermediaries: if handling of requests and/or responses along the path between the business handshake partners is idempotent, then the intermediaries can forgo the use of expensive persistence mechanisms and instead rely on at-least-once delivery that the exchange between the handshake partners gives them.
Have the consumer ask the producer for batches of data.
Give to the workers control over how much work they are willing to buffer in their mailbox, while at the same time giving the manager control over how much work it is willing to hand out for concurrent execution.
One very important aspect in this scheme is that work is requested in batches and proactively. Not only does this save on messaging cost by bundling multiple requests into a single message, but it also allows the system to adapt to the relative speed of producer and consumer:
- When the producer is faster than the consumer, the producer will eventually run out of demand. The system runs in “pull” mode, with the consumer pulling work items from the producer with each request it makes.
- When the producer is slower than the consumer, the consumer will always have demand outstanding. The system runs in “push” mode, where the producer never needs to wait for a work request from the consumer.
- Under changing load characteristics (by way of deployment changes or variable usage patterns), the mechanism automatically switches between the previous two modes without any further need for coordination—it behaves as a dynamic push–pull system.
The other notable aspect of this pattern is that it enables the composition of flow control relationships across a chain of related components. Via the presence or absence of demand for work, the consumer tells the producer about its momentary processing capacity. In situations where the producer is merely an intermediary, it can employ the Pull pattern to retrieve the work items from their source. This implements a nonblocking, asynchronous channel over which back pressure can be transmitted along a data-processing chain. For this reason, this scheme has been adopted by the Reactive Streams standard.
Manage an explicit input queue, and react to its fill level.
Use the Pull pattern between the manager and its workers and make the back pressure visible by observing the fill level of a queue that is filled with external requests and emptied based on the workers’ demands. This measurement of the dif ference between requested and performed work can be used in many ways:
- We have demonstrated the simplest form, which is to use the queue as a smoothing buffer and reject additional requests while the queue is full. This implements service responsiveness while placing an upper bound on the size of the work queue.
- You could spin up a new worker once a given high–water mark was reached, adapting the worker pool elastically to the current service usage. Spinning down extraneous workers could be done by observing the size of the requestQueue as well.
- Rather than observe the momentary fill level of the queue, you could instead monitor its rate of change, taking sustained growth as a signal to enlarge the pool and sustained decrease as a signal to shrink it again.
Using managed queues instead of using implicit queues is always desirable, but it does not need to be done at every step in a processing chain. Within domains that mediate back pressure (for example, by using the Pull pattern), buffers often have the primary function of smoothing out bursts. Observable or intelligent queues are used predominantly at the boundaries of such a system, where the system interacts with other parts that do not participate in the back pressure mechanism. Note that back pressure represents a form of coupling, and as such its scope must be justified by the requirements of the subsystem it is applied to.
Dropping requests is preferable to failing uncontrollably.
Two modes of overload reactions: either send back an incomplete response (degraded functionality), or do not reply at all. You imple ment the metric for selecting one of these modes based on the work-item queue the manager maintains. For this, you need to allow the queue to grow past its logical capacity bound, where the excess is regulated to be proportional to the rate mismatch between producers and consumers of work items—by choosing a different formula for dropFactor , you could make this relation quadratic, exponential, or whatever is required
The important piece here is that providing degraded functionality only works up to a given point, and the service should foresee a mechanism that kicks in once this point is reached. Providing no functionality at all—dropping requests—is cheaper than providing degraded functionality, and under severe overload conditions this is all the service can do to retain control over its resources.
One notable side effect of the chosen implementation technique is that during an intense burst, you now enqueue a certain fraction of work items, whereas the strictly bounded queue as implemented in the Managed Queue pattern example would reject the entire burst (assuming that the burst occurs faster than consumption by the workers).
Throttle your own output rate according to contracts with other services.
You have used a rate-tracking mechanism—the token bucket—to steer the demand that you used for the Pull pattern. The resulting work-item rate then is bounded by the token bucket’s configuration, allowing you to ensure that you do not send more work to another component than was agreed on beforehand. Although a managed queue lets a service protect itself to a degree (see also the discussion of the Drop pattern and its limitations), the Throttling pattern implements service collaboration where the user also protects the service by promising to not exceed a given rate. This can be used restrictively to make rejections unlikely under nominal conditions, or it can be used more liberally to avoid having to call on the Drop pattern. But neither the Managed Queue pattern nor the Drop pattern is replaced by the Throttling pattern; it is import ant to consider overload protection from both the consumer and producer sides.
Separate the business domain logic from communication and state management.
One noteworthy, deliberate aspect is a clear separation of domain object, commands, events, queries, and results on the one hand and the actor’s protocol on the other. The former reference only domain concepts, whereas the latter references what is needed for communication, Having to include message-related types in source files that define domain objects is a signal that the concerns have not been separated cleanly.
The Domain Object pattern decouples the business domain representation from message-driven execution and allows domain experts to describe, specify, and test logic without having to care about distributed system concerns or asynchrony.
Scale out the management of a large number of domain objects by grouping them into shards based on unique and stable object properties.
The Sharding pattern places an upper bound on the size of the directory by grouping the domain objects into a configurable number of shards—the domain is fractured algorithmically into pieces of manageable size. The term algorithmically means the association between objects and shards is determined by a fixed formula that can be evaluated whenever an object needs to be located.
In order to use this pattern, you have to provide a recipe for creating a domain object manager actor and two functions: one for extracting the target shard ID from a command or query message and one for extracting the domain object’s unique ID , which is used to locate it within its shard. Implementing the basic mechanics of clustering and sharding is a complex endeavor that is best left to supporting frameworks or tool kits.
The Sharding pattern allows the efficient storage of an arbitrary number of domain objects, given a cluster with sufficient resources.
Perform state changes only by applying events. Make them durable by storing the events in a log.
You want your domain objects to retain their state across system failures as well as cluster shard–rebalancing events, and toward this end you must make them persistent. You could do this by always updating a database row or a file, but these solutions involve more coordination than is needed. The state changes for different domain objects are managed by different shell components and are naturally serialized when you persist these changes, you could conceptually write to one separate database per object, because there are no consistency constraints to be upheld between them.
Instead of transforming the state-changing events into updates of a single storage location, you can turn things around and make the events themselves the source of truth for your persistent domain objects—hence, the name event-sourcing. The source of truth needs to be persistent, and because events are generated in strictly sequential order, you merely need an append-only log data structure to fulfill this requirement.
This pattern is applicable where the durability of an object’s state history is practical and potentially interesting
The Event-Sourcing pattern turns the destructive update of persistent state into a nondestructive accumulation of information by recognizing that the full his- tory of an object’s state is represented by the change events it emits.
Publish the events emitted by a component so that the rest of the system can derive knowledge from them.
An important property of event streams is that they do not represent the current state of an object of the system: they only consist of immutable facts about the past that have already been committed to persistent storage. The components that emit these events may have progressed to newer states that will only be reflected later in the event stream. The delay between event emission and stream dissemination is a matter of journal-implementation quality, but the fact that there is a significant delay is inherent to this architecture and cannot be avoided. This implies that all operations that must interact with the authoritative, current data must be done on the original domain objects and cannot be decoupled by the Event Stream pattern
The Event Stream pattern uses these persisted change events to implement reliable and scalable dissemination of information throughout the entire system without burdening the originating domain objects with this task. The emitted events can be distributed by the supporting infrastructure and consumed by any number of interested clients to derive new information from their combination or to maintain denormalized views onto the data that are optimized for queries.