Per namespace debounce in Istio

goodCitizens-badCitizens.md

Feature Request: Adaptive per-namespace event throttling in Pilot to protect against config churn

Problem

In large multicluster mesh deployments, a single noisy namespace can destabilize the entire control plane. A buggy controller reconciling a DestinationRule in a loop, short-lived CronJobs churning endpoints, or a misconfigured operator flapping a ServiceEntry — any of these produce a sustained stream of config events that forces Pilot into continuous initPushContext() recomputations and xDS pushes to every connected proxy.

Proposed solution: Adaptive per-namespace event throttling with backoff

Add a pre-debounce throttling layer at DiscoveryServer.ConfigUpdate() that tracks event rates per namespace and applies exponential backoff delay to namespaces exceeding a configurable threshold. Well-behaved namespaces pass through with zero added latency.

How it works

Every config event already flows through ConfigUpdate() as a PushRequest containing ConfigsUpdated sets.Set[ConfigKey], where each ConfigKey has a Namespace field. The throttler uses this existing data — no new watches, informers, or controllers needed.

ConfigUpdate(PushRequest) called
    │
    ├─ Forced / global push (no ConfigsUpdated)? → pass through unchanged
    ├─ System namespace (istio-system, kube-system)? → pass through unchanged
    │
    ├─ Track event rate per namespace using sliding window
    │    if rate < threshold → pass through (zero penalty)
    │    if rate >= threshold:
    │       → buffer event in per-namespace staging area
    │       → start/reset flush timer = baseDelay * 2^(backoff_level)
    │       → on timer fire: merge all buffered events, forward ONE PushRequest
    │       → increment backoff_level (capped at max)
    │
    ├─ When events stop arriving for a namespace → backoff decays to zero
    │
    └─ pushChannel → existing debounce() → Push()  [UNCHANGED]

Key design properties

• Zero impact on well-behaved namespaces: Events below the threshold pass straight through with no added latency. The common case has zero overhead.
• Proportional response via exponential backoff: Occasional bursts (deployment rollouts) may trigger the base delay (1s) briefly, then reset. Sustained churn sees escalating delays (1s → 2s → 4s → 8s capped), collapsing hundreds of events into a handful of merged pushes.
• No config removal: Unlike discoverySelectors, throttled namespaces retain their last-pushed config in proxies. Traffic continues to flow. When churn stops and backoff decays, the next push converges proxies to the actual current state (informer cache always has the latest).
• Existing debounce is untouched: This is a pre-filter before pushChannel, not a modification to the debounce loop.

Why backoff makes VIP/exclusion lists unnecessary

The gap between legitimate bursts and pathological churn is large:

Scenario	Event rate	Duration	Throttler behavior
200-pod rollout	~3/sec burst	~60s, self-terminating	May briefly trigger base delay (1s). Backoff resets within seconds of rollout completing.
HPA scale-up (50 pods)	~1.5/sec burst	~30s	Stays below threshold. No impact.
Applying 10 Istio CRDs	10 events in 1s	Instantaneous	Below sustained threshold. No impact.
Buggy controller loop	10+/sec sustained	Indefinite	Backoff escalates to max. Collapses 600 events/min into ~8 merged pushes/min.
CronJob churn	5-10/sec sustained	Hours	Same as above — sustained rate triggers escalating backoff.

Legitimate operations burst and stop — backoff resets immediately. Pathological churn is sustained — backoff escalates and stays high. The mechanism is self-calibrating.

Configuration

PILOT_ENABLE_NAMESPACE_THROTTLE    (bool, default: false)
PILOT_NAMESPACE_THROTTLE_THRESHOLD (int, default: 100 events per window)
PILOT_NAMESPACE_THROTTLE_WINDOW    (duration, default: 30s)
PILOT_NAMESPACE_THROTTLE_BASE_DELAY (duration, default: 1s)
PILOT_NAMESPACE_THROTTLE_MAX_DELAY  (duration, default: 10s)

Disabled by default. Zero overhead when disabled.

Observability

• pilot_namespace_throttle_events_total{namespace} — counter of events that were buffered
• pilot_namespace_throttle_active{namespace} — gauge indicating active throttling
• Log line when throttling activates/deactivates for a namespace, including event rate

Implementation scope

• One new file: pilot/pkg/xds/namespace_throttler.go (~150-200 lines)
• Small modification to ConfigUpdate() in pilot/pkg/xds/discovery.go (~15 lines)
• Feature flag registration in pilot/pkg/features/tuning.go
• Metrics registration in pilot/pkg/xds/monitoring.go
• Unit tests (~150 lines)

Describe alternatives you've considered

1. discoverySelectors: Works for excluding namespaces but fires DELETE events, actively removing clusters/listeners/routes from proxies. This breaks traffic — we want to freeze config, not remove it.

2. Increasing global PILOT_DEBOUNCE_AFTER/PILOT_DEBOUNCE_MAX: Penalizes all namespaces equally. A 30s debounce to handle churn means 30s config propagation delay for legitimate changes everywhere.

3. Per-namespace debounce queues: Would require restructuring the debounce loop into parallel per-namespace queues, each independently feeding into Push(). Much larger change, higher risk, harder to review.

4. External admission webhook rate limiter: Could rate-limit writes to the API server, but this is outside Istio's control, doesn't cover endpoint/pod events, and adds latency to all API operations.

5. Overload Manager (Envoy-side): Protects individual proxies from being overwhelmed by xDS updates, but doesn't reduce Pilot's processing load — the expensive initPushContext() and push generation still happens.