00-README.md

Envoy RING_HASH + EDS: 503 on late endpoint arrival

This file contains example repro script output for issue envoyproxy/envoy#45344

Scenario 1

envoy.reloadable_features.coalesce_lb_rebuilds_on_batch_update: true

$ ./repro-envoy-ring-hash-503.sh coalesce-enabled
Working directory: /tmp/repro-ring-hash-503.Mzj6Ud

=== Envoy RING_HASH + EDS: 503 on late endpoint arrival ===

This test checks whether Envoy correctly handles the case where an EDS
cluster starts with zero endpoints and receives its first endpoints shortly
after Envoy is already running.

Expected behavior: once endpoints are pushed via EDS and become healthy,
requests should be routed successfully (HTTP 200).

Bug (envoyproxy/envoy#45055): with coalesce_lb_rebuilds_on_batch_update
enabled (default in Envoy 1.38), worker threads can snapshot the empty
ring state. When endpoints arrive, the ring is rebuilt on the main thread
but workers are never re-notified. Result: permanent 503 (no healthy
upstream) even though endpoints are healthy.

Config: ENVOY_IMAGE=envoyproxy/envoy:v1.38.0
        coalesce_lb_rebuilds_on_batch_update=ENABLED

--- Step 1: Starting 1 backend server ---
  backend-1: 172.20.0.2:8080

--- Step 2: Start Envoy with RING_HASH cluster (0 endpoints) ---
  Waiting for Envoy to start... ready.

--- Step 3: Send request → expect 503 (no endpoints yet) ---
  Result: HTTP 503

--- Step 4: Push endpoint via EDS update ---
  Waiting for endpoint to become healthy... ok.

--- Step 5: Send 10 requests → expect 200 (endpoint is healthy) ---

  Result:  503 503 503 503 503 503 503 503 503 503
  (200s: 0, 503s: 10)

  Envoy cluster stats:
    app::172.20.0.2:8080::rq_error::0
    app::172.20.0.2:8080::rq_total::0
    app::172.20.0.2:8080::health_flags::healthy

  (Debug dumps saved to: /tmp/repro-ring-hash-503.Mzj6Ud/)

FAIL: Got 10 x 503 even though the endpoint is healthy.
      Workers are permanently stuck with the empty ring from step 2.
      The 503 will never self-heal until another unrelated EDS change arrives.

Scenario 2

envoy.reloadable_features.coalesce_lb_rebuilds_on_batch_update: false

$ ./repro-envoy-ring-hash-503.sh coalesce-disabled
Working directory: /tmp/repro-ring-hash-503.APxrL4

=== Envoy RING_HASH + EDS: 503 on late endpoint arrival ===

This test checks whether Envoy correctly handles the case where an EDS
cluster starts with zero endpoints and receives its first endpoints shortly
after Envoy is already running.

Expected behavior: once endpoints are pushed via EDS and become healthy,
requests should be routed successfully (HTTP 200).

Bug (envoyproxy/envoy#45055): with coalesce_lb_rebuilds_on_batch_update
enabled (default in Envoy 1.38), worker threads can snapshot the empty
ring state. When endpoints arrive, the ring is rebuilt on the main thread
but workers are never re-notified. Result: permanent 503 (no healthy
upstream) even though endpoints are healthy.

Config: ENVOY_IMAGE=envoyproxy/envoy:v1.38.0
        coalesce_lb_rebuilds_on_batch_update=DISABLED

--- Step 1: Starting 1 backend server ---
  backend-1: 172.20.0.2:8080

--- Step 2: Start Envoy with RING_HASH cluster (0 endpoints) ---
  Waiting for Envoy to start... ready.

--- Step 3: Send request → expect 503 (no endpoints yet) ---
  Result: HTTP 503

--- Step 4: Push endpoint via EDS update ---
  Waiting for endpoint to become healthy... ok.

--- Step 5: Send 10 requests → expect 200 (endpoint is healthy) ---

  Result:  200 200 200 200 200 200 200 200 200 200
  (200s: 10, 503s: 0)

  Envoy cluster stats:
    app::172.20.0.2:8080::rq_error::0
    app::172.20.0.2:8080::rq_total::10
    app::172.20.0.2:8080::health_flags::healthy

  (Debug dumps saved to: /tmp/repro-ring-hash-503.APxrL4/)

PASS: All requests returned 200.
      The ring was correctly built when the endpoint arrived.

repro-envoy-ring-hash-503.sh

#!/usr/bin/env bash
# Reproduction: RING_HASH 503 when EDS cluster starts empty and endpoints arrive later.
#
# Requirements: docker, jq, curl

set -euo pipefail

ENVOY_IMAGE=${ENVOY_IMAGE:-envoyproxy/envoy:v1.38.0}

describe() {
  cat <<'EOF'
=== Envoy RING_HASH + EDS: 503 on late endpoint arrival ===

This test checks whether Envoy correctly handles the case where an EDS
cluster starts with zero endpoints and receives its first endpoints shortly
after Envoy is already running.

Expected behavior: once endpoints are pushed via EDS and become healthy,
requests should be routed successfully (HTTP 200).

Bug (envoyproxy/envoy#45055): with coalesce_lb_rebuilds_on_batch_update
enabled (default in Envoy 1.38), worker threads can snapshot the empty
ring state. When endpoints arrive, the ring is rebuilt on the main thread
but workers are never re-notified. Result: permanent 503 (no healthy
upstream) even though endpoints are healthy.

EOF
}

usage() {
  describe
  cat <<EOF
Usage: $0 <command>

Commands:
  coalesce-enabled    Run with coalesce_lb_rebuilds_on_batch_update ENABLED
                      (default in Envoy 1.38). Expected result: FAIL (503s).

  coalesce-disabled   Run with coalesce_lb_rebuilds_on_batch_update DISABLED.
                      Expected result: PASS (200s).

Environment variables:
  ENVOY_IMAGE   Envoy container image (default: envoyproxy/envoy:v1.38.0)
EOF
}

WORKDIR=""
NETWORK=envoy-eds-503-repro

cleanup() {
  docker rm -f envoy-repro backend-1 >/dev/null 2>&1 || :
  docker network rm "$NETWORK" >/dev/null 2>&1 || :
}
trap cleanup EXIT

run_test() {
  local disable_coalesce=$1
  WORKDIR=$(mktemp -d /tmp/repro-ring-hash-503.XXXXXX)
  echo "Working directory: $WORKDIR"
  echo ""

  describe
  echo "Config: ENVOY_IMAGE=$ENVOY_IMAGE"
  echo "        coalesce_lb_rebuilds_on_batch_update=$([ "$disable_coalesce" = true ] && echo DISABLED || echo ENABLED)"
  echo ""

  # --- Network and backend ---
  echo "--- Step 1: Starting 1 backend server ---"
  docker network create "$NETWORK" >/dev/null 2>&1 || :
  docker run -d --name backend-1 --network "$NETWORK" -e ENV_INSTANCE=backend-1 ghcr.io/tsaarni/echoserver:latest >/dev/null
  sleep 2

  local BACKEND1_IP
  BACKEND1_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' backend-1)
  echo "  backend-1: $BACKEND1_IP:8080"
  echo ""

  # --- xDS files ---
  mkdir -p "$WORKDIR/xds"

  cat > "$WORKDIR/xds/cds.yaml" <<EOF
resources:
- "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster
  name: app
  connect_timeout: 2s
  type: EDS
  lb_policy: RING_HASH
  eds_cluster_config:
    eds_config:
      path_config_source:
        path: /etc/envoy/xds/eds.yaml
    service_name: app
  common_lb_config:
    healthy_panic_threshold:
      value: 0
EOF

  # EDS: start EMPTY (no endpoints — simulates control plane not yet ready)
  cat > "$WORKDIR/xds/eds.yaml" <<EOF
resources:
- "@type": type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment
  cluster_name: app
  endpoints:
  - lb_endpoints: []
EOF

  cat > "$WORKDIR/xds/lds.yaml" <<EOF
resources:
- "@type": type.googleapis.com/envoy.config.listener.v3.Listener
  name: ingress
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 10000
  filter_chains:
  - filters:
    - name: envoy.filters.network.http_connection_manager
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        stat_prefix: ingress
        route_config:
          name: local
          virtual_hosts:
          - name: app
            domains: ["*"]
            routes:
            - match:
                prefix: "/"
              route:
                cluster: app
                hash_policy:
                - cookie:
                    name: X-Session-Affinity
                    ttl: 0s
                    path: /
        http_filters:
        - name: envoy.filters.http.router
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
EOF

  local RUNTIME_LAYER=""
  if [ "$disable_coalesce" = "true" ]; then
    RUNTIME_LAYER='
layered_runtime:
  layers:
  - name: disable_coalesce
    static_layer:
      envoy.reloadable_features.coalesce_lb_rebuilds_on_batch_update: false'
  fi

  cat > "$WORKDIR/envoy.yaml" <<EOF
node:
  id: test
  cluster: test
dynamic_resources:
  cds_config:
    path_config_source:
      path: /etc/envoy/xds/cds.yaml
  lds_config:
    path_config_source:
      path: /etc/envoy/xds/lds.yaml
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901
$RUNTIME_LAYER
EOF

  # --- Start Envoy with empty cluster ---
  echo "--- Step 2: Start Envoy with RING_HASH cluster (0 endpoints) ---"
  docker run -d --name envoy-repro --network "$NETWORK" -p 10000:10000 -p 9901:9901 \
    -v "$WORKDIR/envoy.yaml:/etc/envoy/envoy.yaml:ro" \
    -v "$WORKDIR/xds:/etc/envoy/xds" \
    "$ENVOY_IMAGE" envoy -c /etc/envoy/envoy.yaml --concurrency 4 >/dev/null

  echo -n "  Waiting for Envoy to start..."
  for i in $(seq 1 30); do curl -s http://localhost:9901/ready 2>/dev/null | grep -q LIVE && break; sleep 1; done
  echo " ready."
  echo ""

  # --- Send request to empty cluster ---
  echo "--- Step 3: Send request → expect 503 (no endpoints yet) ---"
  local code
  code=$(curl -s -o /dev/null -w '%{http_code}' --http1.1 http://localhost:10000/)
  echo "  Result: HTTP $code"
  echo ""

  # Let Envoy's workers snapshot the empty ring
  sleep 2

  # --- Push endpoint via EDS ---
  echo "--- Step 4: Push endpoint via EDS update ---"
  cat > "$WORKDIR/xds/eds.yaml.tmp" <<EOF
resources:
- "@type": type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment
  cluster_name: app
  endpoints:
  - lb_endpoints:
    - endpoint:
        address:
          socket_address:
            address: $BACKEND1_IP
            port_value: 8080
EOF
  mv "$WORKDIR/xds/eds.yaml.tmp" "$WORKDIR/xds/eds.yaml"

  echo -n "  Waiting for endpoint to become healthy..."
  for i in $(seq 1 30); do
    local count
    count=$(curl -s http://localhost:9901/clusters | grep -c "app::.*::health_flags::healthy" || :)
    [ "$count" -ge 1 ] && break
    sleep 1
  done
  if [ "${count:-0}" -lt 1 ]; then
    echo " FAILED (Envoy did not pick up the EDS change)"
    exit 1
  fi
  echo " ok."
  sleep 2
  echo ""

  # --- Send requests after endpoint is healthy ---
  echo "--- Step 5: Send 10 requests → expect 200 (endpoint is healthy) ---"
  echo ""

  local status_codes=""
  local num_503=0
  local num_200=0
  for i in $(seq 1 10); do
    code=$(curl -s -o /dev/null -w '%{http_code}' --http1.1 http://localhost:10000/)
    status_codes="$status_codes $code"
    if [ "$code" = "503" ]; then
      num_503=$((num_503 + 1))
    elif [ "$code" = "200" ]; then
      num_200=$((num_200 + 1))
    fi
  done

  echo "  Result: $status_codes"
  echo "  (200s: $num_200, 503s: $num_503)"
  echo ""
  echo "  Envoy cluster stats:"
  curl -s http://localhost:9901/clusters | grep "^app::" | grep -E "rq_total|rq_error|health_flags" | sed 's/^/    /'

  curl -s http://localhost:9901/config_dump > "$WORKDIR/config_dump.json" 2>/dev/null
  curl -s http://localhost:9901/clusters > "$WORKDIR/clusters.txt" 2>/dev/null
  echo ""
  echo "  (Debug dumps saved to: $WORKDIR/)"

  echo ""
  if [ "$num_503" -gt 0 ]; then
    echo "FAIL: Got $num_503 x 503 even though the endpoint is healthy."
    echo "      Workers are permanently stuck with the empty ring from step 2."
    echo "      The 503 will never self-heal until another unrelated EDS change arrives."
    exit 1
  else
    echo "PASS: All requests returned 200."
    echo "      The ring was correctly built when the endpoint arrived."
  fi
}

case "${1:-}" in
  coalesce-enabled)
    run_test false
    ;;
  coalesce-disabled)
    run_test true
    ;;
  *)
    usage
    exit 0
    ;;
esac