AA-Third-try.md

Using HashiCorp Vault Certificate Authentication with Kafka Connect via CSID Secrets Provider

Overview

This guide shows how to configure the CSID Secrets Provider for Vault to authenticate Kafka Connect to Vault using TLS client certificate authentication, using PEM-encoded files (no Java keystores).

We also explore a possible bug or major oversight in the current implementation that may prevent cert-based auth from working at all.

---

How It Should Work

The plugin supports:

config.providers.vault.param.vault.auth.method=Certificate

Which triggers this code:

response = vault.auth().loginByCert();

(AuthHandler.java#L252)

This in turn delegates to the vault-java-driver:

RestResponse restResponse = getRest()
    .sslContext(config.getSslConfig().getSslContext())
    .post();

This will only work correctly if the SslConfig has been fully populated.

The SslConfig supports:

.pemClientCertPath(String)
.pemClientKeyPath(String)
.pemTrustedCerts(String)

---

What Actually Happens

The CSID plugin parses your Kafka config using VaultConfigProviderConfig. The only TLS-related setting it handles is:

this.verify = props.get("vault.ssl.verify.enabled");

(VaultConfigProviderConfig.java)

There is no support for vault.ssl.pem_cert_path, pem_key_path, or pem_trust_path, so those values are never passed into SslConfig, which means:

The Certificate auth method is effectively non-functional out-of-the-box

Unless the Vault client is picking up TLS settings from system properties or environment variables (which is undocumented), this is a major flaw.

---

Suggested Fix: MR Proposal

To enable cert auth:

1. Update VaultConfigProviderConfig to support:

public final String pemCertPath;
public final String pemKeyPath;
public final String pemTrustPath;

And read:

this.pemCertPath = props.get("vault.ssl.pem_cert_path");
this.pemKeyPath = props.get("vault.ssl.pem_key_path");
this.pemTrustPath = props.get("vault.ssl.pem_trust_path");

2. Update VaultConfigProvider (or wherever SslConfig is built) to pass these values:

SslConfig ssl = new SslConfig()
    .pemClientCertPath(config.pemCertPath)
    .pemClientKeyPath(config.pemKeyPath)
    .pemTrustedCerts(config.pemTrustPath)
    .verify(config.verify);

3. Document these options in the README and include a working cert-auth example.

---

Recommended Configuration (Currently Non-Functional)

config.providers=vault
config.providers.vault.class=io.confluent.csid.config.provider.vault.VaultConfigProvider

# Vault server details
config.providers.vault.param.vault.address=https://vault.example.com
config.providers.vault.param.vault.auth.method=Certificate
config.providers.vault.param.vault.auth.mount=cert

# TLS authentication using PEM files
config.providers.vault.param.vault.ssl.pem_cert_path=/etc/kafka/secrets/client.crt      # Client TLS certificate signed by Vault-trusted CA
config.providers.vault.param.vault.ssl.pem_key_path=/etc/kafka/secrets/client.key       # Private key associated with the client certificate
config.providers.vault.param.vault.ssl.pem_trust_path=/etc/kafka/secrets/vault-ca.pem   # CA certificate used to verify Vault server's TLS certificate
config.providers.vault.param.vault.ssl.verify=true                                       # Enable TLS certificate validation (always true in production)

⚠️ These PEM settings will not work until the plugin is patched to pass them to SslConfig.

---

Final Thoughts

The plugin clearly intends to support cert auth, but doesn't parse or pass the PEM-based TLS settings needed to make it work. This isn't just an omission — it may be a functional bug.

If you're integrating Vault with Kafka and need cert-based auth, you'll need to patch or fork the CSID plugin until this is addressed.

Issue has been opened: confluentinc/csid-secrets-providers#426

---

Alternative

This other connector also exists: https://docs.lenses.io/latest/connectors/secret-providers/hashicorp-vault

AB-How-Secure-p.md

Kafka + Vault: Security Reality Check

This document summarizes the challenges, realities, and practical tradeoffs involved in securing Kafka Connect with short-lived secrets in high-security environments, such as financial services.

---

🔒 Security Requirement

All services must use short-lived credentials (e.g., tokens or certs)

This aligns with modern security principles (zero trust, least privilege), but...

Kafka Connect and its ecosystem were not designed for this model.

---

😬 The Problem with Kafka + Short-Lived Secrets

Kafka Connect:

• Loads configuration only at startup
• Does not support hot-reloading TLS certs or Vault tokens
• Most connectors do not poll or resubscribe to config providers

Vault certs and tokens:

• Common TTLs: 1h–24h
• Must be renewed and reloaded before expiry

When secrets rotate on disk, Kafka Connect won’t see them — resulting in auth failures until manually restarted.

---

🧱 What Security Controls Are Actually in Play?

Control Type	Benefit	Limitation
Short-lived tokens	Limits window for abuse	Requires reliable renewal + reload
mTLS certificates	Strong identity + encryption	Breaks on rotation if not reloaded
Vault Agent	Automates renewal + file rotation	Kafka won't reload unless restarted
Encrypted file	Obscures secret storage	Key must still live on host
HSM integration	Protects key material from theft	Complex integration; doesn't protect certs/tokens

---

🔄 What Kafka Connect Actually Does

Scenario	Behavior
Secret changes in Vault	Not re-fetched unless polling + subscription is configured and working
TLS cert rotates on disk	Old cert cached in memory; used until process restart
Vault token expires	Auth fails; reconnects fail silently or noisily
Config polling enabled	Doesn’t always trigger reload; limited to supported use cases

---

✅ Practical Recommendations

1. Use Vault Agent to fetch/renew tokens or TLS certs

   • Mount secrets into ephemeral or tmpfs-backed paths

2. Keep TTLs realistic (e.g., 6–24 hours) and automate connector restarts

   • Use cron, systemd, Kubernetes Jobs, or orchestration tools

3. Minimize blast radius

   • CIDR restrictions
   • Short TTLs
   • Scopes and policies

4. Monitor for failure symptoms

   • Auth errors
   • Token expiry
   • Connection retries

5. Document & justify tradeoffs

   • Show that you're reducing risk, even if not eliminating it

---

🧠 Key Insight

"If they can steal the cert, they can probably wait around for the next one."

So the real security value comes from:

• Scoping: can only do one thing
• Context: only usable from one CIDR, by one service
• Time-limiting: expires fast enough to reduce abuse window
• Visibility: changes can be detected and alerted on

---

💡 Summary Table: Your Options

Approach	Secret Rotation	Reloadable	Secure-ish?	Notes
CSID w/ polling	⚠️ Maybe	⚠️ Partial	🚫 Not reliable	Depends on bug-prone polling logic
Vault Agent + env vars	❌ No	❌ No	⚠️ Fragile	Secrets visible to proc env
Vault Agent + encrypted file	✅ With effort	❌ No	✅ Better	Requires restart, HSM key possible
Vault Agent + PEM certs	✅ Yes	❌ No	✅ Strong	Requires cert reload on disk rotation
Patch CSID to load from disk	✅ Yes	✅ Manual	✅ Strong	Custom logic required

---

🎯 Final Takeaway

Kafka is not a secrets-native platform.

But with reasonable TTLs, automated rotation, scoped certs, and restart logic, you can meet your security requirements without rewriting the entire stack.

And honestly, if your client is banking, and Kafka is not handling direct payment or identity paths, using a Vault Agent to rotate a cert every 24h with a nightly restart is already better than most.

kafka-vault-cert-auth.md

GPT Guide to Kafka Connect with Vault TLS Cert Auth

This guide explains how to configure the Vault Kafka Connector to authenticate using TLS certificates via Vault's cert auth method.

---

1. Enable Vault TLS Cert Auth Method

vault auth enable cert

Create a role that maps client certs to Vault policies:

vault write auth/cert/certs/kafka-connect \
    display_name="kafka-connect" \
    policies="kafka" \
    certificate=@/etc/ssl/certs/kafka-connect.pem \
    ttl="1h"

• Ensure the certificate's CN or SAN matches what Vault expects.
• The policies field should reference a policy that allows reading the necessary secrets.

---

2. Issue Machine Certificate

Use your internal PKI or Vault’s PKI secrets engine:

vault write pki/issue/kafka-connect \
    common_name="kafka-connect.service.consul" \
    ttl="24h"

Save the following securely to the Kafka Connect node:

• certificate → /etc/kafka/secrets/kafka-connect.crt
• private_key → /etc/kafka/secrets/kafka-connect.key
• issuing_ca → /etc/kafka/secrets/ca.pem

---

3. Prepare TLS Key/Trust Stores

Convert to PKCS#12 for Kafka Connect if needed:

# Convert client cert + key to PKCS12
openssl pkcs12 -export \
  -in kafka-connect.crt \
  -inkey kafka-connect.key \
  -certfile ca.pem \
  -out kafka-connect.p12 \
  -name kafka-connect \
  -password pass:changeit

# Convert CA cert to PKCS12 truststore
keytool -importcert \
  -trustcacerts \
  -alias vault-ca \
  -file ca.pem \
  -keystore ca.p12 \
  -storepass changeit \
  -storetype PKCS12

---

4. Configure Kafka Connect Worker

Update the worker config (connect-distributed.properties or similar):

# Enable Vault provider
config.providers=vault
config.providers.vault.class=io.confluent.connect.secret.VaultSecretConfigProvider

# Vault connection
config.providers.vault.vault.uri=https://vault.yourdomain.com:8200
config.providers.vault.vault.engine.version=2
config.providers.vault.vault.auth.method=cert

# TLS settings
config.providers.vault.vault.ssl.keyStorePath=/etc/kafka/secrets/kafka-connect.p12
config.providers.vault.vault.ssl.keyStorePassword=changeit
config.providers.vault.vault.ssl.trustStorePath=/etc/kafka/secrets/ca.p12
config.providers.vault.vault.ssl.trustStorePassword=changeit

---

5. Use in Connector Config

connection.password=${vault:secret/data/myapp/db#password}

Vault will resolve and inject the secret into the connector at runtime.

---

Notes

• Ensure Vault policy for the connector allows access to the required secret paths.
• Rotate TLS certs regularly or use Vault's short-lived certs.
• You can skip PKCS#12 conversion if your Vault plugin supports PEM + key directly.