Client creation for application deployments across environments

Things to know before reading

The names used in this document are intended for illustration only. Some names are not ideal and will need to be refined during future design discussions.
Where a specific Azure Messaging service was needed for illustration, Event Hubs was used. The concepts and options described herein would be applied for both Service Bus and Event Hubs.
Some details not related to the high-level concept are not illustrated; the scope of this is limited to the high level shape and paradigms for the feature area.

Target segment: Developers of production applications

These are developers working with one of the Azure Messaging offerings within the context of a production application. They have done their due diligence to evaluate and explore the product and have reached the level of proficiency where they are comfortable working with their chosen Messaging client and confident that it will help them meet their business goals.

These developers often operate in an ecosystem where responsibilities deployment and health monitoring of applications are handled by a dedicated operations team that does not have access to the source code. This segment also encompasses the needs of an operations team, assuming no familiarity with application code.

Goal

Allow developers working with the Azure Messaging libraries to instantiate client types using a code path that adapts to changes in configuration without requiring code to be recompiled. Client creation between different environments should have minimal friction and require only simple, straightforward, and safe code that is appropriate for applications.

To enable scenarios where an operations team has ownership of deployment and configuration for host environments, ensure configuration may be updated independent of an application by individuals unfamiliar with the Azure Messaging services. Support audit trails and other change management processes.

Non-Goals

Encourage use of connection strings as a general-purpose configuration system; developers should be guided towards the use of more robust and general purpose configuration systems as best practice.
Overcome an Azure portal experience that is not ideal for developers; the areas where the portal experience causes friction for developers, should be fixed within the portal. For example, the core concepts for the resource such as the fully-qualified namespace and entity name should be displayed prominently and be easily discoverable for developers.
Support the messaging clients consuming structured configuration in its raw form; the responsibility for understanding and parsing configuration for the application into the set of information needed by the clients should remain an application responsibility.
Substitute for Azure.Identity as the responsible party for authentication; authentication is an orthogonal domain which has inherent security considerations. In the interest of security, developers should be aware of authentication and explicitly express their intent. The messaging clients should not make implicit decisions on behalf of the user with respect to authentication.
Provide a comprehensive design for a ConnectionStringBuilder type; regardless of the approach used for client configuration, it will likely be desirable to offer an abstraction to create connection strings. The design work for this will be covered in a separate work stream. The snippets herein include a conceptual type for high-level illustration only.

Baseline: Creating a client using the current experience

Using the local identity

This approach eschews the use of connection strings for local development in favor of the local identity. For developers working locally, this would often be the identity associated with Visual Studio, Visual Studio Code, or the Azure CLI.

For hosted environments, such as the traditional Development, UAT, and Production segmentation, the local credential is most likely to be the Managed Service Identity of the host or a specific dedicated Azure Active Directory principal.

public EventHubProducerClient CreateClient()
{
    // The Azure.Identity library holds responsibility for determining the appropriate identity
    // for the local environment.  Validation of parameters is performed by the client.
    
    var fqns = Environment.GetEnvironmentVariable("EventHubsNamespace");
    var eventHub = Environment.GetEnvironmentVariable("EventHub");
    var credential = new DefaultAzureCredential();
    
    return new EventHubProducerClient(fqns, eventHub, credential);
}

Using structured configuration

One common approach is to use structured data, such as JSON, XML, or YAML in order to enable key/value pair items. This allows configuration to be grouped into a single, cohesive unit while keeping each item constrained to a small and simple scope to make it easier to spot errors during review.

public EventHubProducerClient CreateClient(string jsonConfiguration)
{
    var config = JsonSerializer.Deserialize<MyAppConfiguration>(jsonConfiguration);
    
    // Prefer the connection string; validation is performed by the client.
    
    if (!string.IsNullOrEmpty(config.ConnectionString))
    {
        return new EventHubProducerClient(config.ConnectionString);
    }
    
    // Use the expanded form with a token credential; validation is performed by the client.
    
    var credential = new DefaultAzureCredential();
    return new EventHubProducerClient(config.EventHubsNamepace, config.EventHub, credential);
}

Using environment variables

Another common approach is to break individual configuration items out into dedicated environment variables. This avoids the need for any complex parsing, is container-friendly, and has minimal friction across heterogeneous platforms. Using this approach each item constrained to a small and simple scope to make it easier to spot errors during review.

public EventHubProducerClient CreateClient()
{
    // Prefer the connection string; validation is performed by the client.
    
    var connectionString = Environment.GetEnvironmentVariable("EventHubsConnectionString");
    
    if (!string.IsNullOrEmpty(connectionString))
    {
        return new EventHubProducerClient(connectionString);
    }
    
    // Use the expanded form with a token credential; validation is performed by the client.
    
    var fqns = Environment.GetEnvironmentVariable("EventHubsNamespace");
    var eventHub = Environment.GetEnvironmentVariable("EventHub");
    var credential = new DefaultAzureCredential();
    
    return new EventHubProducerClient(fqns, eventHub, credential);
}

Option 1: Guide developers to use the current experience

As demonstrated by the baseline samples, the current experience supports the desired goal in a simple and straightforward manner using less than 10 lines of code with a very small number of simple if checks.

We believe this to be code appropriate for customer applications. It is easy to write, easy to understand, and does not introduce an elevated risk of introducing bugs. It ensures that customers are in control of explicitly stating their intent for authorization removing any implicit behavior by the library which may be non-obvious or introduce confusion into active authorization credentials being used.

Option 2: Expose the EventHubsSharedKeyCredential (stand-alone)

This option eschews the use of a connection string for a set of structured configuration that can be used with each of the different approaches to authorization. The intent is ensure a single source of truth for common elements such as the namespace and entity and avoid the potential for errors due to multiple sources.

Note: This version of the EventHubsSharedKeyCredential follows the recommended SDK guidelines that it not derive from TokenCredential to avoid potentially passing it to other SDK clients. Because of this, it requires a dedicated constructor overload.

public EventHubProducerClient CreateClient(string jsonConfiguration)
{
    var config = JsonSerializer.Deserialize<MyAppConfiguration>(jsonConfiguration);
    
    // Prefer the SAS; validation is performed by the client.
    
    if (!string.IsNullOrEmpty(config.EventHubsSas))
    {
        var credential = new EventHubsSharedKeyCredential(config.EventHubsSas);
        return new EventHubProducerClient(config.EventHubsNamepace, config.EventHub, credential);
    }
    
    // Fallback to the shared key; validation is performed by the client.
    
    if (!string.IsNullOrEmpty(config.EventHubsSharedKeyName))
    {
        var credential = new EventHubsSharedKeyCredential(config.EventHubsSharedKeyName, config.EventHubsSharedKey);
        return new EventHubProducerClient(config.EventHubsNamepace, config.EventHub, credential);
    }
    
    // Use the expanded form with a token credential; validation is performed by the client.
    
    var credential = new DefaultAzureCredential();
    return new EventHubProducerClient(config.EventHubsNamepace, config.EventHub, credential);
}

Option 3: Expose the EventHubsSharedKeyCredential (TokenCredential)

This option follows the same general approach discussed in Option 2, with the same high-level benefits. In this version, the EventHubsSharedKeyCredential derives directly from TokenCredential, which allows for a more cohesive developer experience.

Note: This version is not compliant with the SDK guidelines and would need a disposition granted by the architecture board. It comes with the drawback that a developer may accidentally pass the EventHubsSharedKeyCredential when constructing client types for another SDK, which would cause an exception and may be potentially confusing.

public EventHubProducerClient CreateClient(string jsonConfiguration)
{
    var config = JsonSerializer.Deserialize<MyAppConfiguration>(jsonConfiguration);
    
    var credential = config switch
    {
        _ when (!string.IsNullOrEmpty(config.EventHubsSas)) => 
            new EventHubsSharedKeyCredential(config.EventHubsSas),
            
        _ when (!string.IsNullOrEmpty(config.EventHubsSharedKeyName)) =>
            new EventHubsSharedKeyCredential(config.EventHubsSharedKeyName, config.EventHubsSharedKey),
        
        _ => new DefaultAzureCredential();
    }
    
    return new EventHubProducerClient(config.EventHubsNamepace, config.EventHub, credential);
}

Option 4: Using a ConnectionStringBuilder

This option is intended to add utility to a potential ConnectionStringBuilder for parsing an existing connection string into its component parts.

Additionally, the ConnectionStringBuilder is allowing generation of SAS tokens as part of the connection string to support giving partially-trusted users a short-time and limited credential to access the resource. In that scenario, it stands to reason that those users should be provided a full connection string with the SAS credential embedded rather than being given a SAS token, namespace, and Event Hub name and having responsibility to generate the connection string on their own.

Notes:

This is generally similar to using a structured configuration source, such as JSON, but potentially makes it more difficult to detect errors on review, as the format is not friendly to white-space or multiple lines. Because of this, we believe that the use of structured configuration should be preferred over extending the connection string grammar.
As discussed in the non-goals section, these ConnectionStringBuilder snippets are intended to illustrate the high-level concept, not to serve as a detailed or comprehensive design for the type.

Creating a connection string with a generated SAS

var fqns = Environment.GetEnvironmentVariable("EventHubsNamespace");
var eventHub = Environment.GetEnvironmentVariable("EventHub");
var sharedKeyName = Environment.GetEnvironmentVariable("EventHubsSharedKeyName");
var sharedKey = Environment.GetEnvironmentVariable("EventHubsSharedKey");

if (!TimeSpan.TryParse(Environment.GetEnvironmentVariable("EventHubSasDuration"), out var sasDuration))
{
    throw new ArgumentException("Cannot read SAS duration from Environment", "EventHubsSasDuration");
}
    
return new ConnectionStringBuilder(fqns, eventHub)
    .UsingSharedAccessSignature(sharedKeyName, sharedKey, sasDuration)
    .Build();

Creating a connection string with existing SAS

var fqns = Environment.GetEnvironmentVariable("EventHubsNamespace");
var eventHub = Environment.GetEnvironmentVariable("EventHub");
var sas = Environment.GetEnvironmentVariable("EventHubsSharedAccessSignature");
    
return new ConnectionStringBuilder(fqns, eventHub)
    .UsingSharedAccessSignature(sas)
    .Build();

Creating connection string with a shared key

var fqns = Environment.GetEnvironmentVariable("EventHubsNamespace");
var eventHub = Environment.GetEnvironmentVariable("EventHub");
var sharedKeyName = Environment.GetEnvironmentVariable("EventHubsSharedKeyName");
var sharedKey = Environment.GetEnvironmentVariable("EventHubsSharedKey");
    
return new ConnectionStringBuilder(fqns, eventHub)
    .UsingSharedKey(sharedKeyName, sharedKey)
    .Build();

Create a client using the generated connection string

public EventHubProducerClient CreateClient(string jsonConfiguration)
{
    var config = JsonSerializer.Deserialize<MyAppConfiguration>(jsonConfiguration);
    var parsedTokens = ConnectionStringBuilder.FromExisting(config.ConnectionString);
    
    // Prefer the identity credential if specified by the connection string.  The builder will validate
    // the format of the connection string.
   
    if (config.UseIdentityToken)
    {
        var credential = new DefaultAzureCredential();
        return new EventHubProducerClient(parsedTokens.FullyQualifiedNamespace, parsedTokens.EventHub, credential);
    }
    
    // Use the connection string as-is; the client will perform validation.
    
    return new EventHubProducerClient(config.ConnectionString);
}

Option 5: Allow custom values in the ConnectionStringBuilder

This option is intended build on the ConnectionStringBuilder discussed in Option 4, adding support for unknown custom tokens without requiring developers to have knowledge of the format for creating or parsing. This enables developers to extend the connection string with application-specific values while managing only a single item of configuration.

Note: This is generally similar to using a structured configuration source, such as JSON, but potentially makes it more difficult to detect errors on review, as the format is not friendly to white-space or multiple lines. Because of this, we believe that the use of structured configuration should be preferred over extending the connection string grammar.

Creating a custom connection string

var fqns = Environment.GetEnvironmentVariable("EventHubsNamespace");
var eventHub = Environment.GetEnvironmentVariable("EventHub");
    
return new ConnectionStringBuilder(fqns, eventHub)
    .WithCustom("UseIdentityToken", "true")
    .Build();

Create a client using the generated connection string

public EventHubProducerClient CreateClient()
{
    var connectionString = Environment.GetEnvironmentVariable("EventHubsConnectionString");
    var parsedTokens = ConnectionStringBuilder.FromExisting(connectionString);
    
    // Prefer the identity credential if specified by the connection string.  The builder will validate
    // the format of the connection string.
    
    if (parsedTokens.Custom["UseIdentityToken"] == "true")
    {
        var credential = new DefaultAzureCredential();
        return new EventHubProducerClient(parsedTokens.FullyQualifiedNamespace, parsedTokens.EventHub, credential);
    }
    
    // Use the connection string as-is ; the client will perform validation.
    
    return new EventHubProducerClient(connectionString);
}

Option 6: Provide a client factory in an external package

This option centralizes creation of clients, including responsibility for authentication, by utilizing an external package to provide the glue code illustrated by previous options as an application responsibility. It is intended build on the ConnectionStringBuilder discussed in Option 5, utilizing custom token support to facilitate hints for Identity-based authentication.

Note: This approach is included for completeness, since it was previously discussed. We do not recommend this option and would advocate against it's consideration. We believe that this approach adds complexity, makes it more difficult for customers to understand and manage application dependencies, and reduces the ability for customers to use a single and consistent means of configuration to create clients.

Including packages to support the approach

<Project Sdk="Microsoft.NET.Sdk">
  <ItemGroup>
    <PackageReference Include="Azure.Messaging.EventHubs.ClientFactories" Version="1.0.0" />
    <PackageReference Include="Azure.Messaging.EventHubs" Version="5.3.0" />
    <PackageReference Include="Azure.Messaging.EventHubs.Processor" Version="5.3.0" />    
    <PackageReference Include="Azure.Storage.Blobs" Version="12.6.0" />
  </ItemGroup>
</Project>

Creating a managed identity connection string

using Azure.Messaging.EventHubs.ClientFactories;

var fqns = Environment.GetEnvironmentVariable("EventHubsNamespace");
var eventHub = Environment.GetEnvironmentVariable("EventHub");

// The ConnectionStringBuilder type would be defined in the Azure.Messaging.EventHubs library, where
// it would not have the WithManagedIdentity method.   
//
// WithManagedIdentity would be definied as an extension method in the Azure.Messaging.EventHubs.ClientFactories
// library, ensuring that it was only visible within the package where Azure.Identity is referenced.

return new ConnectionStringBuilder(fqns, eventHub)
    .WithManagedIdentity()
    .Build();

Create a client using the generated connection string

public async Task<EventHubProducerClient> CreateClientAsync()
{
    // Additional optional configuration is demonstrated to illustrate the 
    // difference in approach needed when moving beyond only configuring authentication.
    
    var loadBalanceEnvironment = Environment.GetEnvironmentVariable("EventHubsLoadBalancerStrategy");
    
    if (!Enum.TryParse<LoadBalancingStrategy>(loadBalanceEnvironment, true, out var loadBalanceStrategy))
    {
        throw new ArgumentException("Invalid load balancing config", "Environment.EventHubsLoadBalancerStrategy");
    }
    
    if (!int.TryParse(Environment.GetEnvironmentVariable("EventHubsPrefetch"), out var prefetchCount))
    {
        throw new ArgumentException("Invalid prefetch config", "Environment.EventHubsPrefetch");
    }
    
    var processorOptions = new EventHubProcessorClientOptions
    {
        PrefetchCount = prefetchCount,
        LoadBalancingStrategy = loadBalanceStrategy
    }
    
    // Use the connection string and options as-is; the client will perform validation.
    
    var connectionString = Environment.GetEnvironmentVariable("EventHubsConnectionString");
    return await EventHubsProcessorClientFactory.CreateAsync(connectionString, processorOptions);
}

jsquire/custom-connection-strings.md

Client creation for application deployments across environments

Things to know before reading

Target segment: Developers of production applications

Goal

Non-Goals

Baseline: Creating a client using the current experience

Using the local identity

Using structured configuration

Using environment variables

Option 1: Guide developers to use the current experience

Option 2: Expose the EventHubsSharedKeyCredential (stand-alone)

Option 3: Expose the EventHubsSharedKeyCredential (TokenCredential)

Option 4: Using a ConnectionStringBuilder

Creating a connection string with a generated SAS

Creating a connection string with existing SAS

Creating connection string with a shared key

Create a client using the generated connection string

Option 5: Allow custom values in the ConnectionStringBuilder

Creating a custom connection string

Create a client using the generated connection string

Option 6: Provide a client factory in an external package

Including packages to support the approach

Creating a managed identity connection string

Create a client using the generated connection string