Istio Dynamic module - Proposal gist

README.md

Dynamic Modules Support for Istio TrafficExtension

Context

Add Envoy Dynamic Modules (.so shared libraries loaded via dlopen) as a third filter type in TrafficExtension, alongside WASM and Lua. Dynamic modules offer near-native performance with zero-copy data access but run unsandboxed in Envoy's process.

Go-control-plane v1.37.1 already includes DynamicModuleFilter, DynamicModuleNetworkFilter, DynamicModuleFilterPerRoute, and DynamicModuleConfig protos.

Design Decisions

• Envoy version declaration: OCI annotation io.envoyproxy.envoy.version (major.minor, e.g. 1.38)
• OCI layer format: Standard application/vnd.oci.image.layer.v1.tar+gzip containing the .so
• Failure behavior: NACK + RBAC fallback (consistent with WASM)
• SHA256: Mandatory — native code requires tamper-proof verification
• Namespace restriction: Root namespace only (istio-system) — native code loading is a cluster-admin operation
• Upgrade safety: Open question — ABI incompatibility handling needs design discussion before implementation (see Phase 5)
• OCI tag convention (recommended, not enforced): <module>:v1.0-envoy1.38 — future iteration may add envoy_version_tag_pattern field for automated resolution (e.g. oci://registry/module:v1.0-envoy{ENVOY_VERSION})

---

Phase 1: OCI Image Fetching with Multi-Architecture Support

1.1 Platform-aware fetching

File: pkg/wasm/imagefetcher.go

Add Platform to ImageFetcherOption:

type ImageFetcherOption struct {
    PullSecret []byte
    Insecure   bool
    Platform   *v1.Platform // nil = portable (WASM), non-nil = native (.so)
}

In NewImageFetcher, append remote.WithPlatform(*opt.Platform) to fetchOpts when Platform is non-nil. This makes remote.Get resolve OCI Image Index manifests to the correct architecture automatically. WASM passes nil; dynamic modules pass linux/<runtime.GOARCH>.

1.2 .so extraction

New file: pkg/wasm/sofetcher.go

extractDynamicModuleBinary(r io.Reader, moduleName string) ([]byte, error) — extracts lib<moduleName>.so (or any .so) from the last tar.gz layer. Parallel to extractWasmPluginBinary. Reuses features.MaxWasmBinarySizeBytes for size limits.

1.3 Envoy version compatibility check

File: pkg/wasm/imagefetcher.go (extend PrepareFetch)

After remote.Get, before binary extraction:

1. Read manifest annotation io.envoyproxy.envoy.version
2. Compare against the agent's Envoy major.minor (build-time constant via ldflags, same pattern as IstioVersion)
3. If missing or mismatched: return error "dynamic module requires Envoy X.Y but sidecar runs X.Z"

New constant: EnvoyMajorMinorVersion in pkg/version/version.go, set via ldflags.

1.4 Cache for .so modules

File: pkg/wasm/cache.go

Store at /var/lib/istio/data/<hash>/<checksum>.so. Add ModuleType to GetOptions to select the .so extractor. No structural change to moduleKey.

1.5 Delivery to Envoy

Agent rewrites DynamicModuleConfig.Module from Remote to Local{filename: "<cache-path>"}, same as WASM. No separate shared directory needed.

---

Phase 2: TrafficExtension API

2.1 Proto definition

Repo: istio.io/api — extensions/v1alpha1/traffic_extension.proto

message TrafficExtension {
  // ... existing fields ...
  oneof filter_config {
    WasmConfig wasm = 6;
    LuaConfig lua = 7;
    DynamicModuleConfig dynamic_module = 8;  // NEW
  }
}

// DynamicModuleConfig configures a Dynamic Module filter.
message DynamicModuleConfig {
  // URL of the dynamic module OCI image. Required.
  // Supports oci://, http://, https://, file:// schemes.
  string url = 1;

  // SHA256 checksum of the module binary. Required.
  // Dynamic modules run as native code in Envoy's address space — SHA256 is
  // mandatory to prevent loading tampered or unexpected binaries.
  string sha256 = 2;

  // Name of the filter implementation inside the dynamic module. Required.
  // Maps to Envoy's DynamicModuleFilter.filter_name.
  // Also used as the key for per-route config overrides via EnvoyFilter.
  string filter_name = 3;

  // Configuration passed to the filter. Serialized as JSON to Envoy.
  google.protobuf.Struct filter_config = 4;

  // Module type: HTTP or NETWORK.
  PluginType type = 5;

  // Failure strategy when module fetch fails or version is incompatible.
  FailStrategy fail_strategy = 6;

  // Image pull policy.
  PullPolicy image_pull_policy = 7;

  // Kubernetes secret name for image pulling (must be in same namespace).
  string image_pull_secret = 8;
}

2.2 Validation

File: pkg/config/validation/validation.go

Add validateDynamicModuleConfig():

• url: required, parseable
• sha256: required, hex-encoded, 64 chars (unlike WASM where it is optional)
• filter_name: required, max 256 chars
• filter_config: max 65KB (same as WASM)
• image_pull_secret: same rules as WASM
• type: HTTP or NETWORK (UNSPECIFIED defaults to HTTP)
• Mutual exclusivity in the oneof is already enforced by proto

2.3 Namespace restriction (root namespace only)

Dynamic modules run unsandboxed native code — a compromised .so has full access to Envoy's process. Two enforcement layers:

1. Validation-time: Reject if namespace != meshConfig.RootNamespace. This requires passing namespace context to the validator — extend ValidateTrafficExtension with a validation context (following the pattern used by other namespace-aware validators), or register a wrapper validator for dynamic module TrafficExtensions:

   "TrafficExtension with dynamic_module must be created in root namespace %q"
   

2. Runtime: In convertToTrafficExtensionWrapper(), skip non-root-namespace dynamic modules with a warning:

   if trafficExt.GetDynamicModule() != nil && plugin.Namespace != ps.Mesh.RootNamespace {
       log.Warnf("trafficextension %v/%v discarded: dynamic_module only allowed in root namespace %q",
           plugin.Namespace, plugin.Name, ps.Mesh.RootNamespace)
       return nil
   }

Double enforcement guards against webhook bypass (direct API server access, webhook failure mode).

2.4 Model layer

File: pilot/pkg/model/extensions.go

• Update MatchType() to handle dynamic modules (check type field -> FilterChainType)
• Add BuildHTTPDynamicModuleFilter():

  &dynamicmoduleshttpv3.DynamicModuleFilter{
      DynamicModuleConfig: &dynamicmodulesv3.DynamicModuleConfig{
          Module:          buildDataSource(u, dm.Url, dm.Sha256),
          NackOnCacheMiss: true,
      },
      FilterName:   dm.FilterName,
      FilterConfig: marshalFilterConfig(dm.FilterConfig),
  }

• Add BuildNetworkDynamicModuleFilter() (same pattern)
• Add dynamic_module branch in convertToTrafficExtensionWrapper() with URL parsing and secret normalization

2.5 Extension filter processing

File: pilot/pkg/networking/core/extension/extensionfilter.go

toEnvoyHTTPTrafficExtension() — add dynamic module branch:

} else if filter.GetDynamicModule() != nil {
    return &hcm.HttpFilter{
        Name: filter.ResourceName,
        ConfigType: &hcm.HttpFilter_ConfigDiscovery{
            ConfigDiscovery: &core.ExtensionConfigSource{
                ConfigSource: defaultConfigSource,
                TypeUrls: []string{
                    xds.DynamicModuleHTTPFilterType,
                    xds.RBACHTTPFilterType,
                },
            },
        },
    }
}

toEnvoyNetworkTrafficExtension() — same pattern with network filter type URLs.

InsertedTrafficExtensionConfigurations() — add dynamic module case:

if filter.GetDynamicModule() != nil {
    switch filter.GetDynamicModule().Type {
    case extensions.PluginType_NETWORK:
        // BuildNetworkDynamicModuleFilter()
    default:
        // BuildHTTPDynamicModuleFilter()
    }
}

New file: pilot/pkg/networking/core/extension/dynamicmodule.go (parallel to lua.go)

2.6 Type URL constants

File: pkg/config/xds/filters.go

DynamicModuleHTTPFilterType    = pm.APITypePrefix + "envoy.extensions.filters.http.dynamic_modules.v3.DynamicModuleFilter"
DynamicModuleNetworkFilterType = pm.APITypePrefix + "envoy.extensions.filters.network.dynamic_modules.v3.DynamicModuleNetworkFilter"

2.7 Phase-based loading

No changes needed. Dynamic modules use TrafficExtension's existing Phase field (AUTHN/AUTHZ/STATS/UNSPECIFIED). PopAppendHTTPTrafficExtension / PopAppendNetworkTrafficExtension handle ordering generically. UNSPECIFIED inserts last (after STATS).

2.8 Secret transport

WASM piggybacks pull secrets on VM env vars (ISTIO_META_*); dynamic modules have no VM layer. The agent recognizes dynamic module ECDS resources by TypeUrl, extracts the pull secret from a well-known annotation in the TypedExtensionConfig, fetches the image using those credentials, rewrites Module from Remote to Local, and strips secret data before forwarding to Envoy. This extends the existing rewriteAndForward in pkg/istio-agent/xds_proxy.go.

---

Phase 3: Agent-Side ECDS Interception

3.1 Extend ECDS conversion

File: pkg/wasm/convert.go

Add DynamicModuleHTTPFilterType and DynamicModuleNetworkFilterType cases to tryUnmarshal. When detected:

1. Extract DynamicModuleConfig.Module.Remote (OCI URL)
2. Validate Envoy version from OCI annotations
3. Fetch .so via cache.Get() with platform-aware options (runtime.GOARCH, runtime.GOOS)
4. Rewrite Module from Remote to Local{filename: "<path>"}
5. Forward to Envoy

On failure (fetch error or version mismatch): NACK + RBAC fallback per FailStrategy. Reuse existing createHTTPDefaultFilter / createNetworkDefaultFilter.

3.2 Feature flag

File: pilot/pkg/features/pilot.go

DynamicModuleRemoteLoadConversion (default: true), parallel to WasmRemoteLoadConversion.

---

Phase 4: Module Reload

When dynamic_module.url or dynamic_module.sha256 changes on a TrafficExtension:

1. istiod: KRT fires event -> initTrafficExtensions() re-processes -> EcdsGenerator.Generate() fires (because kind.TrafficExtension is in ConfigsUpdated) -> ECDS push with updated DynamicModuleConfig.Module = Remote{uri: new URL, sha256: new hash}
2. Agent: rewriteAndForward() calls conversion function -> cache Get() with new checksum -> cache miss -> downloads new .so -> writes to new path (/var/lib/istio/data/<hash>/<new-checksum>.so)
3. Envoy: New path/inode triggers reload:
   • dlopen() on new .so
   • envoy_dynamic_module_on_program_init() (ABI check)
   • envoy_dynamic_module_on_http_filter_config_new() with new config
   • New streams use new module filters
   • Existing streams continue on old module filters until completion
   • After drain: envoy_dynamic_module_on_http_filter_config_destroy() -> dlclose() on old .so (unless do_not_close is set)

Cache correctness: SHA256 is mandatory, so same checksum = same binary (no reload). Different checksum = cache miss (triggers reload). Correct by construction.

Cleanup: LocalFileCache purges expired .so files (default 24h). Envoy holds old .so via dlopen fd, so OS keeps it usable even after unlink — safe for in-flight streams.

---

Phase 5: Upgrade Safety and ABI Compatibility (OPEN — needs design discussion)

This phase defines the problem and constraints. The solution needs design discussion before implementation. Phases 1–4 and 6 can proceed independently.

5.1 The problem

Dynamic modules guarantee ABI forward compatibility for one minor version only:

• 1.38 -> 1.39: works (forward compat guaranteed)
• 1.38 -> 1.40: may crash Envoy at dlopen

When an admin upgrades Istio from v1.X (Envoy 1.A) to v1.Y (Envoy 1.B), existing dynamic module .so files may be ABI-incompatible. Unlike WASM (stable Proxy-Wasm ABI across versions), dynamic modules can break across Envoy minor versions.

5.2 Constraints

• ABI incompatibility must not block Istio upgrades or crash Envoy
• Envoy itself has a final safety net: on_program_init returns null for truly incompatible modules, rejecting the .so at dlopen time
• The version check infrastructure exists: OCI annotation io.envoyproxy.envoy.version (Phase 1.3) and EnvoyMajorMinorVersion build-time constant provide the data needed
• The NACK + RBAC fallback mechanism (Phase 3) already handles fetch/load failures generically

5.3 Open questions

Q1: Where should the version check live?

• Agent-side only (in ECDS interception, Phase 3): agent compares OCI annotation vs running Envoy, NACKs on mismatch. Simple, but istiod has no visibility.
• istiod-side (in convertToTrafficExtensionWrapper()): istiod knows its Envoy version at build time, can check without fetching the image. But istiod doesn't fetch OCI images today — it would need the annotation propagated differently.
• Both: istiod does a best-effort check (if annotation is available in config), agent does the authoritative check at fetch time.

Q2: How should incompatibility be reported to the user?

• Status condition on the TrafficExtension CR (e.g. ModuleCompatible=False)? Requires a status controller (follows Gateway API pattern in pilot/pkg/config/kube/gateway/).
• Event on the CR? Simpler but less discoverable.
• Log-only? Lowest effort but hardest to notice.
• How does the agent communicate the mismatch back to istiod for status updates? Via NACK error detail? Separate channel?

Q3: Should there be a bypass flag?

• A feature flag like PILOT_DYNAMIC_MODULE_ENFORCE_ABI_VERSION=false would let admins skip the check for known-compatible modules or testing. Is this needed for the initial implementation, or can it be added later?

Q4: What is the admin's upgrade workflow?

• Must the admin update each TrafficExtension's url/sha256 manually to point to a new module build?
• Should Istio support an automated resolution pattern (e.g. envoy_version_tag_pattern field that substitutes {ENVOY_VERSION} in the OCI tag)?
• Can module authors publish multi-version OCI images (Image Index with Envoy-version-specific manifests via annotations)?
• What happens during the gap between Istio upgrade and module update — is RBAC fallback acceptable, or should the old module keep running until explicitly replaced?

Q5: What is the version matching granularity?

• Major.minor only (e.g. 1.38)? Envoy guarantees forward compat for N -> N+1 minor.
• Should we allow range matching (e.g. >=1.38,<1.40)?
• What about patch versions?

5.4 Possible approaches (not decided)

These are starting points for discussion, not decisions:

Approach A — Minimal (agent-side only): Agent checks version at fetch time, NACKs + RBAC fallback on mismatch, logs a warning. No status condition, no bypass flag. Admin sees the NACK in proxy logs and updates the TrafficExtension manually.

Approach B — Status-aware: Same as A, plus istiod sets a ModuleCompatible status condition on the CR. Requires a status controller. Admin can kubectl get trafficextension to see which modules are incompatible.

Approach C — Full lifecycle: Same as B, plus bypass flag, automated tag resolution, and pre-upgrade compatibility check (e.g. istioctl analyze warns about incompatible modules before upgrade).

---

Phase 6: EnvoyFilter Per-Route Override

filter_name in the TrafficExtension becomes the Envoy filter name for TypedPerFilterConfig. Users override per-route config via EnvoyFilter with HTTP_ROUTE + MERGE.

TrafficExtension

apiVersion: extensions.istio.io/v1alpha1
kind: TrafficExtension
metadata:
  name: my-auth-module
  namespace: istio-system
spec:
  targetRefs:
    - kind: Service
      group: ""
      name: my-service
  phase: AUTHZ
  dynamic_module:
    url: oci://registry.example.com/my-auth-module:v1
    sha256: "abc123..."
    filter_name: "auth_checker"
    filter_config:
      log_level: "info"
      default_action: "allow"
    type: HTTP
    fail_strategy: FAIL_CLOSE
    image_pull_policy: IfNotPresent
    image_pull_secret: my-registry-secret

Per-route override via EnvoyFilter

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: auth-module-admin-override
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: my-service
  configPatches:
    - applyTo: HTTP_ROUTE
      match:
        context: SIDECAR_INBOUND
        routeConfiguration:
          vhost:
            route:
              name: "default"
      patch:
        operation: MERGE
        value:
          typed_per_filter_config:
            # Key = ECDS resource name: extensions.istio.io/trafficextension/<ns>.<name>
            "extensions.istio.io/trafficextension/istio-system.my-auth-module":
              "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_modules.v3.DynamicModuleFilterPerRoute
              filter_name: "auth_checker"
              filter_config:
                "@type": type.googleapis.com/google.protobuf.Struct
                value:
                  log_level: "debug"
                  default_action: "deny"
                  admin_bypass: true

DynamicModuleFilterPerRoute is already in go-control-plane. Envoy passes per-route config to on_http_filter_per_route_config_new.

---

OCI Image Specification

FROM scratch
COPY libmy_module.so /lib/my_module.so
LABEL io.envoyproxy.envoy.version="1.38"

docker buildx build --platform linux/amd64,linux/arm64 \
  -t registry.example.com/my-module:v1.0-envoy1.38 --push .

Requirements:

• Annotation io.envoyproxy.envoy.version with major.minor
• Standard tar.gz layer with .so at lib/<name>.so
• Multi-arch: OCI Image Index with per-platform manifests

---

Files to Modify

File	Change
istio.io/api: extensions/v1alpha1/traffic_extension.proto	DynamicModuleConfig message + oneof field 8
pkg/config/validation/validation.go	validateDynamicModuleConfig(), root namespace check
pilot/pkg/model/extensions.go	Builders, convertToTrafficExtensionWrapper() (namespace guard), MatchType()
pilot/pkg/networking/core/extension/extensionfilter.go	Dynamic module branches in HTTP/Network/ECDS functions
pilot/pkg/networking/core/extension/dynamicmodule.go	New builder file (parallel to lua.go)
pkg/config/xds/filters.go	DynamicModuleHTTPFilterType, DynamicModuleNetworkFilterType
pkg/wasm/imagefetcher.go	Platform in options, version annotation check
pkg/wasm/sofetcher.go	New: .so extraction from tar.gz
pkg/wasm/cache.go	.so file extension, ModuleType in GetOptions
pkg/wasm/convert.go	Dynamic module ECDS interception, RBAC fallback
pkg/istio-agent/xds_proxy.go	Extend rewriteAndForward for dynamic module TypeUrls
pilot/pkg/features/pilot.go	DynamicModuleRemoteLoadConversion flag
pkg/version/version.go	EnvoyMajorMinorVersion constant
Phase 5 (TBD)	Files depend on chosen approach — may include pilot/pkg/features/pilot.go, pilot/pkg/status/, pilot/pkg/model/extensions.go

Verification

1. Unit: pkg/wasm/imagefetcher_test.go — multi-arch resolution, .so extraction, version annotation check. pilot/pkg/model/extensions_test.go — wrapper conversion, builders.
2. Validation: pkg/config/validation/validation_test.go — SHA256 required, filter_name required, root namespace enforcement.
3. Namespace: Non-root-namespace dynamic module rejected at validation and skipped at runtime.
4. ECDS: pilot/pkg/xds/ecds_test.go — dynamic module ECDS generation.
5. Reload: URL/SHA256 change triggers ECDS push, agent re-fetches, Envoy loads new .so.
6. ABI version mismatch tests: TBD — depends on Phase 5 design decisions.
7. Integration: tests/integration/ — deploy TrafficExtension with dynamic module, verify filter loading and phase ordering.
8. Per-route override: EnvoyFilter HTTP_ROUTE + MERGE overrides dynamic module config.