jsquire

Event Hubs: Checkpoint Store Proposal

Despite being a key requirement for extending EventProcessor<T>, no abstraction exists for processor storage operations. Developers wishing to extend the processor must implement the storage operations required by its abstract members, holding responsibility for ensuring a production-ready implementation. They must also infer what storage operations may be needed by the application which the processor does not use - such as writing checkpoints that the processor consumes - and provide an implementation for them. This places a burden on developers and introduces a barrier of entry for extending EventProcessor<T>.

Things to know before reading

• The names used in this document are intended for illustration only. Some names are not ideal and will need to be refined during discussions.

• Some details not related to the high-level concept are not illustrated; the scope of this is limited to the high level shape and paradigms for the feature area.

jsquire

Event Hubs: Next Steps for Idempotent Publishing in the EventHubProducerClient

The "idempotent publishing" feature in the Event Hubs client library was introduced as a means to help reduce the potential for duplication when publishing events using the EventHubProducerClient. It appeared in several beta packages, starting in September of 2020 and was last available in March of 2021.

Primarily driven by a desire for parity with Kafka, the feature was built on service infrastructure created for the Event Hubs Kafka head. Its API and user experience are heavily influenced by Kafka's approach, which is centered around their buffered producer model. As a result, the feature is not well-suited to the EventHubProducerClient and has significant potential for causing customer confusion and providing a poor development experience.

The major concerns are:

• The concept of “idempotent” in this context does not meet developer expectations.

jsquire

Scenario

This scenario is based on this question on Stack Overflow, slightly modified to improve the failure cases.

A WinForms application is collecting telemetry from some number of IoT devices deployed on company delivery trucks. Collection is accompanied by some light data transformations, and then publishing them to Event Hubs.

Because there is a consistent flow of telemetry from the devices, the application is performing the collection and the processing in parallel. To maximize throughput, the application is prioritizing batch density and is not concerned with accounting for any drops in telemetry collection that would benefit from flushing partial batches, other than when the collected telemetry queue is fully empty.

Event Publishing Code

jsquire

Streaming Producer: Partition Assignment and Batching

Goal

Allow developers to enqueue events to be efficiently published without being burdened with managing batches or needing a deep understanding of Event Hubs partitioning. Publishing behavior should be understandable and performance consistent regardless of the pattern of events being enqueued.

Constraints

• The order of events when published must match the order that they were enqueued, unless developers specifically opt-out of ordering to achieve higher throughput via concurrency.

jsquire

Azure.Core: AzureNamedKeyCredential

When using a shared key for authorization, some services require that the name of the key be used as a component of the algorithm to form the authorization token. For these cases, AzureKeyCredential and AzureSasCredential aren't adequate as they track only a single value. A new credential type, AzureNamedKeyCredential has been approved to address this gap.

Things to know before reading

• The names used in this document are intended for illustration only. Some names are not ideal and will need to be refined during discussions.

• Some details not related to the high-level concept are not illustrated; the scope of this is limited to the high level shape and paradigms for the feature area.

jsquire

Streaming Producer: Early Concept

The current story for publishing events offers a natural idiom when an application has a discrete set of events that it would like to treat as a single unit. When the producer is invoked, it offers a strong contract what allows the application to immediately understand whether publishing was successful or not. This approach is has many advantages, such as ensuring the client is not bound to any particular partition and can support publishing concurrently.

For applications that unable to efficiently batch events into a discrete set, the approach is less compelling. Publishing a single event or a small set can be inefficient and may impact throughput. For applications that view an event as a single item of data, the responsibility of managing batches introduces additional overhead and boilerplate code to collect those single events, track them, and package them as a batch efficiently.

Things to know before reading

• This is an early concept meant to serve as a po

jsquire

Event Hubs: BinaryData Member for EventData

In order to take advantage of integration with the ObjectSerializer for Schema Registry support as well as use a common idiom for data access and translation, the EventData type in the Event Hubs client library would like to take advantage of the BinaryData type introduced for the Azure.Core ecosystem.

Because Event Hubs is generally available and backwards compatibility needs to be preserved for existing applications using EventData, it is not possible to directly change the type on the existing Body member. This document outlines some of the considerations and options for doing so.

Things to know before reading

• The names used in this document are presented for discussion as potential final choices. Please provide feedback during discussion.

jsquire

Follow-Up: Client creation for application deployments across environments

This serves to illustrate some of the ideas surfaced during discussion of the options outlined in the Client creation for application deployments across environments and assumed familiarity with content of that document.

To ensure focus continues to be aligned, the goals and baseline illustrations have been brought forward for convenience.

Goal

Allow developers working with the Azure Messaging libraries to instantiate client types using a code path that adapts to changes in configuration without requiring code to be recompiled. Client creation between different environments should have minimal friction and require only simple, straightforward, and safe code that is appropriate for applications.

jsquire

Client creation for application deployments across environments

Things to know before reading

• The names used in this document are intended for illustration only. Some names are not ideal and will need to be refined during future design discussions.

• Where a specific Azure Messaging service was needed for illustration, Event Hubs was used. The concepts and options described herein would be applied for both Service Bus and Event Hubs.

• Some details not related to the high-level concept are not illustrated; the scope of this is limited to the high level shape and paradigms for the feature area.

jsquire

Event Hubs: Idempotent Event Publishing

Things to know before reading

• The names used in this document are intended for illustration only. Some names are not ideal and will need to be refined during discussions.

• Some details not related to the high-level concept are not illustrated; the scope of this is limited to the high level shape and paradigms for the feature area.

• Fake methods are used to illustrate "something needs to happen, but the details are unimportant." As a general rule, if an operation is not directly related to one of the Event Hubs types, it can likely be assumed that it is for illustration only. These methods will most often use ellipses for the parameter list, in order to help differentiate them.