jsquire

.NET Event Hubs Client: Retry with Timeout Concept (Second Preview)

Because Event Hubs is based on the AMQP protocol, there is no request/response cycle that defines an operation. Both the Event Hubs client and service make use of an operation timeout as means of coordination for managing their resources. When communicating with the Event Hubs service, the client library is required to specify an operation timeout on the AMQP link.

If an operation has not been completed within this timeout period, the client library will consider it a transient failure and apply its retry policy. Likewise, if the server is expecting communication from the client, such as a message acknowledgment, that does not occur within the timeout period, it will consider the operation to have failed.

Operation Timeouts and Cancellations, Oh My!

The guidance for .NET design has been to strongly prefer cancellation tokens with timeout over explicit timeouts. However, this presents a challenge in the case of Event Hubs where t

jsquire

Azure Messaging: Allow control of the transport formatting

The Azure Messaging client libraries endeavor to provide an API surface that is protocol-agnostic and shaped such that it would be suitable for use with a variety of protocols and transport formats. Developers using the client libraries should not have to understand or be aware of the details related to AMQP, MQTT, STOMP, HTTP, JMS, or other protocols and transport formats that may be supported by the service - now or in the future; these details are considered the responsibility of the library.

Another goal of the Messaging client libraries is to support heterogeneous environments; when using the official Azure client libraries, data that was published with an SDK for one language should be understood when consumed with an SDK for any of the other supported languages.

There are, however, specialized scenarios when developers have need to participate in the preparation of transport messages. One such example is ensuring interoperability with

jsquire

Event Hubs: Direct Partition Connections

Connections to the Azure Event Hubs service are typically made to a gateway endpoint, which holds responsibility for routing traffic to the appropriate service node for a given partition. This allows for a simplified client view of the service, without the need for awareness of individual nodes nor use of a discovery service as an intermediate step for opening connections.

The use of the gateway as an intermediary does incur additional latency when compared to the efficiency of using a direct node connection. In cases in which require maximum throughput, bypassing the gateway is desirable. The Event Hubs service offers the ability to opt-into a mechanism to discover the node currently hosting a given partition and the ability to directly connect to it, eliminating the gateway as an intermediary. This feature of the Event Hubs service is colloquially known as "receiver redirect."

As the familiar name implies, discovery of the Event Hubs node for a partition use

jsquire

Event Hubs: Custom Endpoint Support

Connections to the Azure Event Hubs service are made using the fully qualified namespace name assigned for the Event Hubs namespace as the root of the endpoint address. Because the Event Hubs service ensures the appropriate CNAME records are configured in DNS as part of the provisioning process, this provides an endpoint address that is strongly associated with the Event Hubs namespace and easier for developers to remember.

While the standard approach works well under normal circumstances, it can be troublesome in certain environments when a proxy is in use or when an Express Route circuit is used with certain configuration. In these scenarios, an alternative host name is needed to allow for proper routing to the Event Hubs service.

Things to know before reading

• The names used in this document are intended for illustration only. Some names are not ideal and will need to be refined during discussions.

jsquire

Brainstorming: Transport message format

Scenarios

Retrieve the transport body of a message

using Azure.Messaging.SerivceBus.Transport.Amqp;

var message = DoSomethingToCreateTheServiceBusMessage();
var amqpBody = message.GetAmqpBody();

jsquire

Stress Thoughts

Requirements

Hosting

• The platform hosts multiple developer-authored test scenarios, encapsulating end-to-end use cases for the client library with varied complexity.

• The platform can execute test scenarios for a prolonged duration spanning multiple days, while monitoring and managing scenarios that are not in a responsive or healthy state.

jsquire

Applying the Alan.Parser Project to the AMQP Translation Layer

Azure Messaging Size Restrictions

Event Hubs Size Limits

• Basic: 256k
• Standard: 1MB
• Dedicatd: 1MB

Service Bus Size Limits

jsquire

Event Hubs: Idempotent Event Publishing

When publishing events to Event Hubs, timeouts or other transient failures may introduce ambiguity into the understanding of whether a batch of events was received by the service. Because Event Hubs has an at-least-once guarantee for delivery and consumers are strongly encouraged to be idempotent in their processing, the common approach is to resend any batch where the status of receipt was unknown.

In some specialized scenarios, producers have a need to make efforts to avoid publishing duplicate events. To support these scenarios, the Event Hubs service is adding support for annotating messages with metadata to indicate the sequence in which events were intended to be published and to use that as an indicator of which events were already received by a the target partition. This functionality comes at a performance cost, however, and requires producers to follow a strict set of semantics to allow the service to perform server-side deduplication based on the intent

jsquire

Event Hubs: Idempotent Event Publishing

When publishing events to Event Hubs, timeouts or other transient failures may introduce ambiguity into the understanding of whether a batch of events was received by the service. Because Event Hubs has an at-least-once guarantee for delivery and consumers are strongly encouraged to be idempotent in their processing, the common approach is to resend any batch where the status of receipt was unknown.

In some specialized scenarios, producers have a need to make efforts to avoid publishing duplicate events. To support these scenarios, the Event Hubs service is adding support for annotating messages with metadata to indicate the sequence in which events were intended to be published and to use that as an indicator of which events were already received by a the target partition. This functionality comes at a performance cost, however, and requires producers to follow a strict set of semantics to allow the service to perform server-side deduplication based on the intent

jsquire

Event Hubs: Idempotent Event Publishing

Things to know before reading

• The names used in this document are intended for illustration only. Some names are not ideal and will need to be refined during discussions.

• Some details not related to the high-level concept are not illustrated; the scope of this is limited to the high level shape and paradigms for the feature area.

• Fake methods are used to illustrate "something needs to happen, but the details are unimportant." As a general rule, if an operation is not directly related to one of the Event Hubs types, it can likely be assumed that it is for illustration only. These methods will most often use ellipses for the parameter list, in order to help differentiate them.