Public IP Address Mapping in Multi-Rack Kubernetes Clusters

A Practitioner's Guide to IP → Instance → Node → ToR Switch Resolution

The Mapping Problem
Conceptual Model
Kubernetes-Native Primitives
Cilium Primitives
Implementation Tiers
Tier 1: LB IPAM + BGP Service Advertisement
Tier 2: Interface IP Advertisement
Tier 3: Multi-Pool IPAM + BGP CiliumPodIPPool
Tier 4: L2 Announcements
Tier 5: Custom Controller with Hybrid Primitives
IP Ownership and Authority
BGP Topology Design Patterns
Observability and Verification
Lifecycle Management
Anti-Patterns and Footguns
Decision Matrix
Reference Manifests
Glossary

The Mapping Problem

In a multi-rack bare-metal Kubernetes cluster, every workload instance that holds a public IP address must be locatable. Not "reachable through a proxy" or "accessible via NAT" — locatable. The network fabric must know exactly which physical Top-of-Rack (ToR) switch connects to the node that hosts the instance that owns a given public IP.

This is a four-layer resolution chain:

  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
  │  Public IP   │────▶│  Instance   │────▶│    Node     │────▶│ ToR Switch  │
  │ 203.0.113.17 │     │   vm-web-1  │     │  node-a-04  │     │  tor-rack-a │
  └─────────────┘     └─────────────┘     └─────────────┘     └─────────────┘

Each layer in this chain has a source of truth. Each transition between layers requires a mechanism. The quality of an implementation is measured by how many of these mechanisms are native to Kubernetes and Cilium versus how many require custom controllers, sidecars, or out-of-band configuration.

Why This Matters

When the chain is broken — when the ToR switch does not know that 203.0.113.17 lives on node-a-04 — traffic to that IP enters the fabric and has no next hop. It is black-holed. When the chain is stale — when the instance migrates from node-a-04 to node-b-02 but the ToR still points at rack A — traffic arrives at the wrong node. It is misrouted.

The chain must be:

Complete: every public IP has a resolvable path to a ToR switch
Accurate: the mapping reflects the current placement of the instance
Timely: changes propagate within the BGP convergence window
Authoritative: no instance can claim an IP it does not own

Conceptual Model

Physical Topology

                    ┌──────────────────┐
                    │   Spine Switch   │
                    │    (BGP peer)    │
                    └──┬──────────┬───┘
                       │          │
              ┌────────┘          └────────┐
              │                            │
     ┌────────┴────────┐         ┌─────────┴───────┐
     │  ToR Switch A   │         │  ToR Switch B    │
     │  ASN 65001      │         │  ASN 65002       │
     └──┬─────┬────┬───┘         └──┬─────┬────┬───┘
        │     │    │                │     │    │
      ┌─┴─┐ ┌┴──┐┌┴──┐          ┌──┴┐ ┌──┴┐ ┌┴──┐
      │ N1 │ │N2 ││N3 │          │ N4 │ │N5 │ │N6 │
      └─┬──┘ └┬──┘└───┘          └──┬─┘ └───┘ └───┘
        │     │                     │
     ┌──┴──┐ ┌┴────┐            ┌───┴──┐
     │VM-a │ │VM-b │            │VM-c  │
     │ .17 │ │ .18 │            │ .19  │
     └─────┘ └─────┘            └──────┘

Logical Mapping

┌──────────────────────────────────────────────────────────────┐
│                    CLUSTER STATE                             │
│                                                              │
│  IP Address       Instance     Node        ToR      ASN      │
│  ─────────────    ────────     ────────    ─────    ─────    │
│  203.0.113.17     vm-a         node-1      tor-a    65001    │
│  203.0.113.18     vm-b         node-2      tor-a    65001    │
│  203.0.113.19     vm-c         node-4      tor-b    65002    │
│                                                              │
│  BGP Advertisements:                                         │
│    tor-a receives: 203.0.113.17/32, 203.0.113.18/32          │
│    tor-b receives: 203.0.113.19/32                           │
│    tor-a does NOT receive: 203.0.113.19/32                   │
│    tor-b does NOT receive: 203.0.113.17/32, .18/32           │
└──────────────────────────────────────────────────────────────┘

The Four Sources of Truth

Layer	Source of Truth	Kubernetes Primitive
IP → Instance	Who owns this IP?	`IPAddress` API, CRD, or Service
Instance → Node	Where is this instance running?	Pod `spec.nodeName`, CRD status
Node → ToR Switch	Which switch does this node peer with?	`Node.Labels`, BGP session state
ToR → Spine	What routes does this switch announce?	BGP RIB (outside Kubernetes)

Kubernetes-Native Primitives

This section catalogs every Kubernetes-native API resource and pattern relevant to the mapping problem. These are the building blocks available before any CNI-specific features enter the picture.

IPAddress (networking.k8s.io/v1)

The IPAddress resource is a cluster-scoped singleton representing ownership of a single IP address. The object name is the IP in canonical format.

apiVersion: networking.k8s.io/v1
kind: IPAddress
metadata:
  name: 203.0.113.17
spec:
  parentRef:
    group: example.com
    resource: vminstances
    namespace: production
    name: vm-a

Key properties:

Cluster-scoped, one object per IP — name uniqueness enforces no-duplicate-IP
spec.parentRef identifies the owning resource (group, resource, namespace, name)
No nodeName field — the IPAddress object tracks ownership, not placement
No status subresource — no conditions, no readiness signal
parentRef becomes immutable from Kubernetes v1.36 alpha onward (+k8s:alpha(since: "1.36")=+k8s:immutable)
The v1 type was introduced in Kubernetes 1.33 (+k8s:prerelease-lifecycle-gen:introduced=1.33); a v1beta1 version exists in earlier releases

What this gives you: a cluster-wide registry mapping IP → Instance with uniqueness enforcement by the API server. You create an IPAddress object when an instance is assigned a public IP. If another instance tries to claim the same IP, the create fails with AlreadyExists.

What this does NOT give you: any node-level information. You must combine this with the instance's placement to get IP → Instance → Node.

ServiceCIDR (networking.k8s.io/v1)

The ServiceCIDR resource defines IP ranges from which ClusterIPs are allocated to Services. It is managed by the service-cidrs-controller.

Key properties:

Cluster-scoped, graduated to stable in Kubernetes 1.33 with the MultiCIDRServiceAllocator feature gate
Has a status subresource with conditions:
- ServiceCIDRConditionReady — the CIDR is available for allocation
- ServiceCIDRReasonTerminating — the CIDR is being deleted
The controller watches ServiceCIDR and IPAddress objects, detects overlapping CIDRs, and manages finalizers for safe deletion
Admission can be restricted via ValidatingAdmissionPolicy to prevent overlap with other networks

Relevance to the mapping problem: ServiceCIDR is strictly for Service ClusterIP allocation. It is NOT applicable to VM public IP pools. However, its controller is a reference implementation for the informer + work queue + reconciler pattern used in custom IPAM controllers, and its status conditions pattern is worth emulating.

Node Object

The Node object carries several fields relevant to the mapping:

Node.Status.Addresses

type NodeAddress struct {
    Type    NodeAddressType  // InternalIP, ExternalIP, Hostname, ...
    Address string
}

Set by kubelet or cloud controller manager. InternalIP is the node's primary cluster-facing IP — the address that the ToR switch peers with. ExternalIP has no defined semantics and may or may not exist.

This is the Node → ToR anchor: the node's InternalIP is the address the ToR switch knows as the BGP peer address. From the ToR's perspective, every /32 route it receives from that peer is "on that node."

Node.Labels

Labels encode topology. Kubernetes defines well-known labels:

kubernetes.io/hostname           → node identity
topology.kubernetes.io/zone      → failure domain / zone
topology.kubernetes.io/region    → broader geographic region

Custom labels encode rack-level topology:

network.example.com/tor-switch: tor-rack-a
network.example.com/rack-id: rack-a
network.example.com/asn: "65001"

The NodeRestriction admission plugin prevents kubelets from setting labels outside kubelet.kubernetes.io/* and node.kubernetes.io/*. It also forbids labels in the node-restriction.kubernetes.io namespace and unknown kubernetes.io/k8s.io labels. This means labels under a custom domain like network.example.com/* can only be set by an administrator or a privileged controller — they are trustworthy as the Node → ToR mapping.

Node.Spec.PodCIDRs

Allocated by the nodeipam-controller from --cluster-cidr. This is for pod networking only and does not manage public IPs. However, the allocation pattern — a bitmap allocator that writes the result to the Node spec — is the reference architecture for "allocate a range to a node."

EndpointSlice (discovery.k8s.io/v1)

EndpointSlices track Service backend endpoints. Each endpoint entry contains:

type Endpoint struct {
    Addresses           []string                 // endpoint IPs
    Conditions          EndpointConditions       // ready, serving, terminating
    Hostname            *string                  // optional DNS hostname
    TargetRef           *v1.ObjectReference       // reference to backing object
    DeprecatedTopology  map[string]string        // deprecated, use zone/nodeName
    NodeName            *string                  // hosting node
    Zone                *string                  // topology zone
    Hints               *EndpointHints           // topology routing hints (pointer, may be nil)
}

Note that Hints is a pointer (*EndpointHints), not a value type. A nil Hints means no topology hints are set, which is the default state for most endpoints. When populated, it contains:

type EndpointHints struct {
    ForZones []ForZone  // zone(s) to consume from (max 8)
    ForNodes []ForNode  // node(s) to consume from (max 8)
}

If instances are modeled as Service backends (e.g., behind a headless Service), EndpointSlices provide IP → nodeName for free. This is the most Kubernetes-native mechanism for tracking where an endpoint lives.

The ForNodes hint is available for fine-grained topology-aware routing control when it is populated by the EndpointSlice controller.

ValidatingAdmissionPolicy (admissionregistration.k8s.io/v1)

CEL-based admission policies evaluated at write time. Relevant for IP ownership verification:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: vm-ip-ownership
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: ConfigMap
  matchConstraints:
    resourceRules:
    - apiGroups: ["example.com"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["vminstances"]
  variables:
  - name: allowedCIDRs
    expression: "params.data['allowed-cidrs'].split(',')"
  validations:
  - expression: >
      variables.allowedCIDRs.exists(c, cidr(c).containsIP(object.spec.publicIP))
    message: "Public IP is not within the allowed range for this namespace"

Limitation: CEL expressions have access to object, oldObject, request, params, namespaceObject, variables, and authorizer. They cannot make API calls. You cannot check whether another IPAddress object already exists with the same IP. For cross-object uniqueness enforcement, use a webhook or a controller-side check.

Service with externalTrafficPolicy

When externalTrafficPolicy: Local is set on a Service:

kube-proxy only programs forwarding rules for endpoints local to the node
Traffic arriving at a node with no local endpoints is dropped
This is a node-locality enforcement mechanism

If an instance's public IP is modeled as the Service's loadBalancerIP, then externalTrafficPolicy: Local ensures only the node hosting the instance handles traffic for that IP. This fact — "this node has a local endpoint for this IP" — is what BGP advertisement systems use to decide whether to announce the route.

Note on Service.spec.externalIPs: This field is on a deprecation path. The AllowServiceExternalIPs feature gate (KEP 5707) controls whether kube-proxy programs rules for externalIPs. When disabled, kube-proxy will not program rules for externalIPs, effectively disabling this deprecated feature. Additionally, any user with Service create/update permission can add any IP to spec.externalIPs, making it a route hijack vector in multi-tenant clusters. The DenyServiceExternalIPs admission plugin can be used to block it entirely. New architectures should avoid relying on externalIPs.

PodTopologyLabels Admission Plugin

The PodTopologyLabels feature gate (alpha) enables an admission plugin (podtopologylabels) that copies topology.kubernetes.io/{zone,region} labels from the assigned Node onto the Pod at scheduling time via the pod/binding subresource.

Relevance: If a custom label like network.example.com/tor-switch were added to the set of propagated labels (via a similar admission plugin pattern), you could propagate the ToR switch identity to VMs at scheduling time. This gives VM → ToR without requiring a separate lookup.

Controller Pattern

The standard Kubernetes controller pattern for custom IPAM:

┌─────────────────────────────────────────────────────┐
│                   CONTROLLER                        │
│                                                     │
│  ┌──────────┐   ┌────────────┐   ┌──────────────┐  │
│  │ Informer │──▶│ Work Queue │──▶│  Reconciler  │  │
│  │ (Watch)  │   │  (Buffer)  │   │  (Act)       │  │
│  └──────────┘   └────────────┘   └──────────────┘  │
│       │                                │            │
│       │ watches:                       │ writes:    │
│       │ - VMInstance CRD               │ - IPAddress│
│       │ - Node objects                 │ - Status   │
│       │ - IPAddress objects            │ - Events   │
│                                                     │
└─────────────────────────────────────────────────────┘

The service-cidrs-controller in Kubernetes is a reference implementation that demonstrates this pattern for IP range management:

Watches ServiceCIDR and IPAddress objects via informers
Detects overlapping CIDRs
Manages finalizers for safe deletion
Reports status conditions (ServiceCIDRConditionReady, ServiceCIDRReasonTerminating)
Uses workqueue for rate-limited retry

RBAC for such a controller follows the pattern:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vm-ip-controller
rules:
- apiGroups: ["networking.k8s.io"]
  resources: ["ipaddresses"]
  verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: ["example.com"]
  resources: ["vminstances"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["example.com"]
  resources: ["vminstances/status"]
  verbs: ["patch", "update"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]

Cross-Namespace ownerReferences

Kubernetes garbage collection does not support cross-namespace ownership or namespace-scoped → cluster-scoped ownership chains. If a namespace-scoped VMInstance "owns" a cluster-scoped IPAddress, you cannot rely on ownerReferences and GC to clean up the IPAddress when the VMInstance is deleted. Use a controller with a finalizer instead.

Cilium Primitives

Cilium extends Kubernetes with CNI-level capabilities for IP management, BGP peering, and route advertisement. This section inventories every Cilium CRD and feature relevant to the mapping problem.

API Version Note: All Cilium BGP CRDs use apiVersion: cilium.io/v2 as their storage version. The cilium.io/v2alpha1 versions of these resources are deprecated (+kubebuilder:deprecatedversion). All manifests in this document use cilium.io/v2.

CRD Relationship Map

┌─────────────────────────────────────────────────────────────────┐
│                     CILIUM BGP CRD GRAPH                        │
│                                                                 │
│  ┌───────────────────────┐                                      │
│  │ CiliumBGPClusterConfig│─── nodeSelector ──▶ Node objects     │
│  │                       │                                      │
│  │  bgpInstances:        │  MaxItems: 16                        │
│  │   - name: "inst-65k"  │                                      │
│  │     localASN: 65000   │                                      │
│  │     peers:            │                                      │
│  │      - peerConfigRef ─┼──────────────────────┐               │
│  │        peerASN        │                      │               │
│  │        peerAddress    │                      ▼               │
│  └───────────┬───────────┘         ┌────────────────────────┐   │
│              │                     │  CiliumBGPPeerConfig   │   │
│              │ operator            │                        │   │
│              │ generates           │  transport:            │   │
│              ▼                     │    peerPort: 179       │   │
│  ┌───────────────────────┐         │  timers:               │   │
│  │  CiliumBGPNodeConfig  │         │    holdTime: 90        │   │
│  │  (per-node, auto)     │         │    keepAlive: 30       │   │
│  │                       │         │  families:             │   │
│  │  status:              │         │    - afi: ipv4         │   │
│  │    peers:             │         │      safi: unicast     │   │
│  │     - peeringState    │         │  advertisements: ──────┼─┐ │
│  │       routeCount      │         │    matchLabels         │ │ │
│  │       establishedTime │         └────────────────────────┘ │ │
│  └───────────────────────┘                                    │ │
│                                    ┌──────────────────────────┘ │
│  ┌───────────────────────┐         ▼                            │
│  │CiliumBGPNodeConfig-   │  ┌──────────────────────────────┐   │
│  │Override               │  │ CiliumBGPAdvertisement       │   │
│  │  (per-node, manual)   │  │                              │   │
│  │                       │  │ advertisements:               │   │
│  │  routerID override    │  │  - advertisementType:         │   │
│  │  localPort override   │  │     PodCIDR |                │   │
│  │  localASN override    │  │     CiliumPodIPPool |        │   │
│  │  peers:               │  │     Service |                │   │
│  │   - localAddress      │  │     Interface                │   │
│  │   - localPort         │  │    service:                   │   │
│  └───────────────────────┘  │      addresses: [...]         │   │
│                             │    interface:                  │   │
│                             │      name: "..."              │   │
│                             │    selector: {...}             │   │
│                             │    attributes:                 │   │
│                             │      communities:              │   │
│                             │        standard: [...]         │   │
│                             │        wellKnown: [...]        │   │
│                             │        large: [...]            │   │
│                             │      localPreference: N        │   │
│                             └──────────────────────────────┘   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Structural notes:

bgpInstances on CiliumBGPClusterConfig has a maximum of 16 entries (+kubebuilder:validation:MaxItems=16). This is relevant for designs with many BGP instances per node (e.g., multi-VRF or per-rack-ASN patterns).
peerASN on a peer entry defaults to 0. When set to 0, BGP OPEN message ASN validation is disabled and the ASN is determined from the peer's OPEN message. This is useful with the DefaultGateway auto-discovery pattern where the ToR ASN may not be known statically.
selector on BGPAdvertisement is a sibling of service: and interface:, not a child. The BGPServiceOptions struct has only three fields: addresses, aggregationLengthIPv4, aggregationLengthIPv6.
CiliumBGPNodeConfigOverride peer overrides have two fields per peer: localAddress and localPort (both optional).

Advertisement Types

The CiliumBGPAdvertisement CRD supports four advertisementType values:

// +kubebuilder:validation:Enum=PodCIDR;CiliumPodIPPool;Service;Interface
type BGPAdvertisementType string

Each has a distinct source of truth and different implications for the mapping problem.

PodCIDR

Source of truth: Node.Spec.PodCIDRs — the pod IP range allocated to each node by the node IPAM controller.

What it advertises: The node's pod CIDR as received from Kubernetes. This enables external routers to reach pods directly without encapsulation.

Relevance to the mapping problem: Low direct relevance. Pod CIDRs are per-node ranges, not per-instance public IPs.

CiliumPodIPPool

Source of truth: CIDRs allocated from CiliumPodIPPool resources to individual nodes by Cilium's Multi-Pool IPAM.

What it advertises: The per-node CIDR slices allocated from CiliumPodIPPool resources. Unlike PodCIDR which advertises the Kubernetes-native pod CIDR, this type advertises Cilium-managed pool allocations.

Relevance to the mapping problem: Medium. If instances receive IPs from a CiliumPodIPPool allocated from a public range, this advertisement type announces the per-node slice of that range. The ToR switch learns which range is reachable via which node, but does not learn individual /32 routes.

Service

Source of truth: Kubernetes Service objects — specifically their ClusterIP, ExternalIP, and LoadBalancerIP addresses.

What it advertises: Service VIPs. The reconciler watches Service objects and their endpoints. When externalTrafficPolicy: Local is set, the reconciler tracks whether local endpoints exist on the node and only advertises the route when at least one local endpoint is present. When all local endpoints disappear, the route is withdrawn.

Available address types:

LoadBalancerIP — IPs assigned by LB IPAM or a cloud provider
ExternalIP — IPs in service.spec.externalIPs (note: deprecated feature, see Kubernetes-Native Primitives section)
ClusterIP — the cluster-internal VIP

Selector: The selector field is on the BGPAdvertisement struct itself — it is a sibling of the service field, not a child. Only Services matching the selector have their IPs injected into the BGP RIB.

# CORRECT structure — selector is a sibling of service:
advertisements:
- advertisementType: "Service"
  service:
    addresses:
    - LoadBalancerIP
  selector:                    # ← sibling of service:, NOT a child
    matchLabels:
      purpose: vm-public-ip

Aggregation: aggregationLengthIPv4 and aggregationLengthIPv6 on the BGPServiceOptions struct allow aggregating multiple Service IPs into a shorter prefix. The maximum valid values are 31 (IPv4) and 127 (IPv6), enforced by kubebuilder validation:

// +kubebuilder:validation:Maximum=31
AggregationLengthIPv4 *int16

// +kubebuilder:validation:Maximum=127
AggregationLengthIPv6 *int16

To get exact /32 or /128 advertisement (the default behavior for Service IPs), omit the aggregation fields entirely. Setting them to 32 or 128 will fail admission validation. Aggregation is ignored when externalTrafficPolicy: Local is set.

Relevance to the mapping problem: High. If each instance has a corresponding Service of type LoadBalancer, and LB IPAM assigns the public IP from a CiliumLoadBalancerIPPool, then the full chain is:

CiliumLoadBalancerIPPool  →  assigns IP to Service
Service                   →  tracked by EndpointSlice
EndpointSlice             →  records nodeName
CiliumBGPAdvertisement    →  advertises /32 from the correct node
BGP session               →  delivers route to ToR switch

This is the most Kubernetes-idiomatic path because every link in the chain uses a native or Cilium-native resource with defined semantics.

Interface

Source of truth: IP addresses assigned to a named local network interface on the node, as seen by the Linux kernel.

What it advertises: Every IP address found on the named interface, as exact /32 (IPv4) or /128 (IPv6) routes. The reconciler inspects the interface via netlink.

Excluded ranges: Loopback, multicast, IPv6 link-local, and IPv4-mapped-IPv6 addresses are automatically excluded.

Interface state requirements: The interface must be administratively enabled and in operationally up or unknown state. If the interface goes admin-down, all advertisements for its IPs are withdrawn.

Matching: The interface.name field requires an exact interface name. No regex, glob, or pattern matching is supported.

What Cilium manages vs does not manage: Cilium advertises whatever IPs are present on the interface. Cilium does not assign IPs to the interface, does not validate that the IPs belong to any particular pool, and does not check ownership. The administrator (or an external DHCP server or IPAM controller) is responsible for IP assignment.

Relevance to the mapping problem: Medium-high. If the VM networking layer exposes VM IPs on a host-visible interface, Cilium can advertise them. But the IP→Instance link is not tracked by Cilium — it exists only in whatever system assigned the IP to the interface.

BGP Communities

The attributes.communities field on BGPAdvertisement is a struct with three distinct sub-fields, not a flat list:

type BGPCommunities struct {
    Standard  []BGPStandardCommunity   // e.g., "65000:100"
    WellKnown []BGPWellKnownCommunity  // e.g., "no-export"
    Large     []BGPLargeCommunity      // e.g., "65000:1:100"
}

Example in YAML:

attributes:
  communities:
    standard:
    - "65000:100"
    wellKnown:
    - "no-export"
    large:
    - "65000:1:100"
  localPreference: 100

LB IPAM

CiliumLoadBalancerIPPool defines ranges of IP addresses that Cilium can assign to Services of type LoadBalancer.

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: public-pool
spec:
  blocks:
  - cidr: "203.0.113.0/24"
  serviceSelector:
    matchLabels:
      ip-pool: public

Key behaviors:

Pools can contain any IP range — RFC1918, public routable, IPv6
serviceSelector controls which Services can draw from which pools
When a Service is deleted, its allocated IP is returned to the pool
When a Service has no backends, the behavior depends on externalTrafficPolicy and --enable-no-service-endpoints-routable
LB IPAM is allocation only — it does not advertise. Advertisement is handled by the BGP control plane or L2 announcements
Multiple pools can exist; pool selection is by label matching

Integration with BGP:

LB IPAM allocates. BGP advertises. The link between them is the Service object itself — LB IPAM writes the allocated IP to service.status.loadBalancer.ingress, and the BGP reconciler watches Services for those IPs.

┌────────────────────┐      ┌─────────────┐      ┌──────────────────┐
│CiliumLoadBalancer  │      │   Service   │      │CiliumBGP-        │
│IPPool              │─────▶│  (type: LB) │◀─────│Advertisement     │
│                    │ IP   │             │watch │ (type: Service)  │
│ 203.0.113.0/24     │alloc │ .status.lb. │      │                  │
│                    │      │  ingress[0] │      │ addresses:       │
│                    │      │  = .17      │      │  - LoadBalancerIP│
└────────────────────┘      └─────────────┘      └──────────────────┘

Multi-Pool IPAM

CiliumPodIPPool defines IP pools from which pod (and potentially instance) IPs are allocated. Unlike LB IPAM, which allocates to Services, Multi-Pool IPAM allocates to workloads directly.

apiVersion: cilium.io/v2alpha1
kind: CiliumPodIPPool
metadata:
  name: public-vms
spec:
  ipv4:
    cidrs:
    - "203.0.113.0/24"
    maskSize: 28

Key behaviors:

Allocates per-node CIDR slices from the pool (e.g., each node gets a /28)
Pre-allocation via ipam-multi-pool-pre-allocation flag
Pre-allocation formula: neededIPs = roundUp(inUseIPs + pendingIPs + preAllocIPs, preAllocIPs)
Can allocate from any IP range, including public routable
Allocated CIDRs can be announced via BGP using advertisementType: "CiliumPodIPPool" (not PodCIDR, which is for Kubernetes-native pod CIDRs only)

Trade-offs vs LB IPAM:

Multi-Pool IPAM gives each node a slice of the pool. This means:

The node "owns" a range, not individual IPs
BGP advertisement is per-CIDR, not per-IP
IP-to-instance resolution requires additional tracking within the node's slice
Pool fragmentation occurs as nodes join/leave

L2 Announcements

Cilium can announce Service IPs via ARP/NDP on a Layer 2 network instead of (or in addition to) BGP.

Key behaviors:

Leader election determines which node announces each IP
Uses CiliumL2AnnouncementPolicy CRD
Each IP is announced by exactly one node at a time
Failover occurs when the leader node becomes unavailable
Gratuitous ARP / unsolicited NDP is sent on leader change

Relevance to the mapping problem: L2 announcements solve the same "which node owns this IP" problem as BGP, but at Layer 2 instead of Layer 3. In a multi-rack topology where racks are separated by L3 boundaries (routed fabric), L2 announcements are limited to within a single L2 domain. For cross-rack reachability, BGP is required.

L2 and BGP can coexist. A common pattern is L2 within a rack and BGP across racks.

BGP Peer Auto-Discovery

The CiliumBGPClusterConfig supports automatic discovery of BGP peers:

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: auto-tor
spec:
  nodeSelector:
    matchLabels:
      bgp-enabled: "true"
  bgpInstances:
  - name: "tor-peering"
    localASN: 65100
    peers:
    - name: "tor-auto"
      peerASN: 0          # 0 = disable ASN validation, learn from peer OPEN
      autoDiscovery:
        mode: "DefaultGateway"
        defaultGateway:
          addressFamily: ipv4
      peerConfigRef:
        name: "tor-config"

DefaultGateway mode: Cilium inspects the node's default route to determine the ToR switch's IP address. This assumes the default gateway IS the ToR switch — which is true in most leaf-spine datacenter topologies.

peerASN: 0 semantics: When peerASN is set to 0 (which is the default), BGP OPEN message ASN validation is disabled. The peer's ASN is determined from the peer's OPEN message instead. This is particularly useful with auto-discovery where the ToR switch's ASN may not be known statically or may differ per rack.

What this gives you: Automatic Node → ToR peering without hardcoding switch IPs per node. Combined with nodeSelector, you can have different BGP configurations per rack.

Can coexist with manual peers: Yes. A single BGP instance can have both auto-discovered and manually specified peers.

Implementation Tiers

The following tiers are ranked from most to least idiomatic — where "idiomatic" means "uses the fewest custom components and relies most heavily on standard Kubernetes and Cilium semantics."

Tier Ranking Criteria

┌─────────────────────────────────────────────────────────────┐
│  RANKING CRITERIA (weighted)                                 │
│                                                              │
│  1. Kubernetes API conformance (35%)                         │
│     Does it use standard API resources (Service, IPAddress,  │
│     EndpointSlice) or require custom CRDs?                   │
│                                                              │
│  2. Cilium-native integration (25%)                          │
│     Does it use Cilium's built-in reconcilers or require     │
│     external controllers to bridge gaps?                     │
│                                                              │
│  3. Lifecycle automation (20%)                               │
│     How much of the IP assignment, advertisement, and        │
│     withdrawal lifecycle is automatic vs manual?             │
│                                                              │
│  4. Operational observability (10%)                          │
│     Can the mapping be verified through standard tooling     │
│     (kubectl, cilium-dbg) without custom dashboards?         │
│                                                              │
│  5. Failure mode safety (10%)                                │
│     What happens on component failure? Black hole? Stale     │
│     route? Graceful withdrawal?                              │
└─────────────────────────────────────────────────────────────┘

Tier 1: LB IPAM + BGP Service Advertisement

Idiomaticity: ★★★★★ — Fully native

This is the highest-idiomaticity path. Every component is a standard Kubernetes or Cilium resource with defined lifecycle semantics.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│  ┌──────────────────┐    ┌──────────────┐    ┌───────────────┐  │
│  │CiliumLoadBalancer│    │  Service     │    │ EndpointSlice │  │
│  │IPPool            │───▶│  type: LB    │◀───│               │  │
│  │                  │ IP │              │ ep │  nodeName:     │  │
│  │ 203.0.113.0/24   │alloc│ .status.lb. │track│  node-a-04   │  │
│  │                  │    │  ingress[0]  │    │               │  │
│  └──────────────────┘    │  =203.0.    │    └───────────────┘  │
│                          │   113.17    │                        │
│                          └──────┬──────┘                        │
│                                 │                               │
│                          ┌──────┴──────┐                        │
│                          │CiliumBGP-   │                        │
│                          │Advertisement│                        │
│                          │ type:Service│                        │
│                          │ addresses:  │                        │
│                          │  -LB IP     │                        │
│                          │             │                        │
│                          │ selector:   │ ← sibling of service  │
│                          │  purpose:   │                        │
│                          │   public    │                        │
│                          └──────┬──────┘                        │
│                                 │                               │
│                          ┌──────┴──────┐                        │
│                          │  BGP RIB    │                        │
│                          │             │                        │
│                          │ 203.0.113.  │                        │
│                          │ 17/32 → N1  │                        │
│                          └──────┬──────┘                        │
│                                 │                               │
│                          ┌──────┴──────┐                        │
│                          │  ToR Switch │                        │
│                          └─────────────┘                        │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Data Flow

1. Admin creates CiliumLoadBalancerIPPool with public CIDR
2. Instance is created → a Service of type LoadBalancer is created
3. LB IPAM assigns an IP from the pool → writes to Service.status
4. EndpointSlice controller records the backend pod/instance and its node
5. CiliumBGPAdvertisement (type: Service) watches the Service
6. Cilium agent on the node with local endpoints injects /32 into BGP RIB
7. GoBGP announces the route to the ToR switch
8. Instance is destroyed → Service is deleted → IP returns to pool
9. Cilium withdraws the /32 route

Mapping Resolution

IP → Instance:    Service.metadata.name (1:1 Service per instance)
Instance → Node:  EndpointSlice.endpoints[].nodeName
Node → ToR:       BGP session (CiliumBGPNodeConfig.status.peers)

When to Use

WHEN instances can be modeled as Service backends (pods, or external endpoints)
WHEN you want fully automatic IP lifecycle (allocate on create, free on delete)
WHEN the number of instances is manageable as individual Services (hundreds, not tens of thousands)
WHEN you need externalTrafficPolicy: Local semantics for node-sticky advertisement

When NOT to Use

WHEN instances are not representable as Service backends
WHEN you need tens of thousands of individual /32 routes (Service-per-instance does not scale beyond ~5,000-10,000 Services in most clusters)
WHEN the IP assignment is managed externally (e.g., by DHCP) and cannot be modeled as LB IPAM allocation

Manifests

---
# 1. IP Pool — defines the public IP range available for allocation
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: public-vm-pool
spec:
  blocks:
  - cidr: "203.0.113.0/24"
  serviceSelector:
    matchLabels:
      purpose: vm-public-ip

---
# 2. BGP Cluster Config — establishes peering with ToR switches
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: tor-peering
spec:
  nodeSelector:
    matchLabels:
      bgp-enabled: "true"
  bgpInstances:
  - name: "rack-peering"
    localASN: 65100
    peers:
    - name: "tor-switch"
      peerASN: 0       # learn ASN from peer OPEN message
      autoDiscovery:
        mode: "DefaultGateway"
        defaultGateway:
          addressFamily: ipv4
      peerConfigRef:
        name: "tor-peer-config"

---
# 3. Peer Config — timers, address families, advertisement reference
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: tor-peer-config
spec:
  transport:
    peerPort: 179
  timers:
    holdTimeSeconds: 90
    keepAliveTimeSeconds: 30
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: public-vms

---
# 4. Advertisement — what to inject into BGP RIB
#    NOTE: selector is a SIBLING of service:, not a child
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: vm-service-advertisement
  labels:
    advertise: public-vms
spec:
  advertisements:
  - advertisementType: "Service"
    service:
      addresses:
      - LoadBalancerIP
      # aggregation fields OMITTED to get default /32 advertisement
      # Maximum valid values: aggregationLengthIPv4=31, aggregationLengthIPv6=127
    selector:                   # ← sibling of service:, NOT nested inside it
      matchLabels:
        purpose: vm-public-ip

---
# 5. Per-instance Service (one per VM)
apiVersion: v1
kind: Service
metadata:
  name: vm-web-1
  labels:
    purpose: vm-public-ip
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    vm.example.com/instance: vm-web-1
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP

Tier 2: Interface IP Advertisement

Idiomaticity: ★★★★☆ — Cilium-native BGP, external IP assignment

This tier uses Cilium's ability to advertise IPs found on a local interface. The IP assignment happens outside Cilium (via DHCP, static config, or a custom controller), and Cilium simply announces whatever it finds.

Architecture

┌─────────────────────────────────────────────────────────┐
│  NODE                                                   │
│                                                         │
│  ┌───────────────────────┐                              │
│  │  Network Interface    │                              │
│  │  name: "vm-public-0"  │                              │
│  │                       │                              │
│  │  inet 203.0.113.17/32 │ ◀── assigned externally      │
│  │  inet 203.0.113.19/32 │ ◀── assigned externally      │
│  └───────────┬───────────┘                              │
│              │ netlink                                   │
│              ▼                                           │
│  ┌───────────────────────┐                              │
│  │  Cilium Agent         │                              │
│  │  BGP Reconciler       │                              │
│  │                       │                              │
│  │  "Interface vm-public │                              │
│  │   -0 has .17 and .19" │                              │
│  │  → inject /32 routes  │                              │
│  └───────────┬───────────┘                              │
│              │ BGP UPDATE                                │
│              ▼                                           │
│  ┌───────────────────────┐                              │
│  │  ToR Switch           │                              │
│  │  receives:            │                              │
│  │   203.0.113.17/32     │                              │
│  │   203.0.113.19/32     │                              │
│  └───────────────────────┘                              │
│                                                         │
└─────────────────────────────────────────────────────────┘

What Cilium Manages

BGP session establishment and maintenance
Detecting IPs on the named interface via netlink
Injecting /32 routes into the BGP RIB
Withdrawing routes when IPs are removed or the interface goes down

What Cilium Does NOT Manage

IP assignment to the interface
Validation that the IP belongs to a legitimate pool
Tracking which instance owns which IP
Any mapping between the IP and a Kubernetes resource

Mapping Resolution

IP → Instance:    NOT tracked by Cilium. Requires external system.
Instance → Node:  Implicit (the IP is on this node's interface)
Node → ToR:       BGP session

Manifest

---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: interface-advertisement
  labels:
    advertise: vm-interfaces
spec:
  advertisements:
  - advertisementType: "Interface"
    interface:
      name: "vm-public-0"

When to Use

WHEN IPs are assigned by an external system (DHCP, cloud-init, manual config)
WHEN the VM networking layer exposes VM IPs on a host-visible interface
WHEN you need a simple, low-ceremony BGP announcement mechanism
WHEN the IP→Instance mapping is tracked elsewhere (external CMDB, DHCP lease table)

When NOT to Use

WHEN the VM networking layer does NOT expose IPs to the host kernel
WHEN you need Kubernetes-API-level tracking of IP ownership
WHEN you need automatic IP lifecycle management
WHEN the interface name varies across nodes (no pattern matching support)

Footguns

The interface name must be exact. If node A uses one name and node B uses another, you need separate CiliumBGPAdvertisement resources or standardize the interface name across all nodes.
Cilium advertises ALL IPs on the interface. There is no CIDR filter. If the interface has management IPs, DHCP relay IPs, or other non-VM IPs, they will all be advertised as /32 routes.
No ownership verification. If a VM configures a rogue IP on the interface, Cilium will happily advertise it. IP authority must be enforced at the assignment layer.

Tier 3: Multi-Pool IPAM + BGP CiliumPodIPPool

Idiomaticity: ★★★☆☆ — Native IPAM, per-CIDR (not per-IP) advertisement

This tier uses Cilium's Multi-Pool IPAM to allocate a slice of a public IP range to each node, then advertises the per-node CIDR via BGP using the CiliumPodIPPool advertisement type.

Architecture

┌──────────────────────────────────────────────────────────────┐
│                                                              │
│  CiliumPodIPPool: 203.0.113.0/24, maskSize: /28             │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐       │
│  │   Node A     │  │   Node B     │  │   Node C     │       │
│  │              │  │              │  │              │       │
│  │ allocated:   │  │ allocated:   │  │ allocated:   │       │
│  │ .0/28        │  │ .16/28       │  │ .32/28       │       │
│  │              │  │              │  │              │       │
│  │ instances:   │  │ instances:   │  │ instances:   │       │
│  │  .1 (vm-a)   │  │  .17 (vm-c)  │  │  (none)      │       │
│  │  .2 (vm-b)   │  │              │  │              │       │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘       │
│         │ BGP              │ BGP              │ BGP          │
│         ▼                  ▼                  ▼              │
│  ToR receives:       ToR receives:      ToR receives:       │
│  203.0.113.0/28      203.0.113.16/28    203.0.113.32/28     │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Advertisement type: Use advertisementType: "CiliumPodIPPool", not "PodCIDR". The PodCIDR type advertises Kubernetes-native pod CIDRs from Node.Spec.PodCIDRs. The CiliumPodIPPool type advertises Cilium-managed pool allocations.

Trade-offs

Advantage: Fewer BGP routes. Instead of one /32 per instance, you advertise one CIDR per node. At 100 nodes with a /28 per node, that is 100 BGP routes instead of (potentially) 1,400.

Disadvantage: The ToR switch knows the CIDR is on a node, but does not know which specific IP within that CIDR is in use. If the range is sparsely populated, the ToR still routes all IPs in the /28 to that node, even if most are unassigned. This wastes address space and can cause confusion in troubleshooting.

Disadvantage: Instance migration across nodes means the IP changes (because each node has a different slice of the pool). This breaks any system that depends on IP stability across migrations.

When to Use

WHEN instances do not need stable IPs across node migrations
WHEN the IP range is large enough to partition into per-node slices
WHEN minimizing BGP route table size is a priority
WHEN all instances on a node should be reachable via the same prefix

When NOT to Use

WHEN instances require a stable public IP regardless of which node they run on
WHEN the IP range is small and cannot be efficiently partitioned
WHEN per-IP-level route advertisement is required

Tier 4: L2 Announcements

Idiomaticity: ★★★☆☆ — Native Cilium, but L2-scoped

L2 announcements use ARP/NDP to claim IPs on a Layer 2 network segment. This is a viable approach within a single rack (single L2 domain) but does not cross L3 boundaries.

Architecture

┌──────────────────────────────────────────────────────────┐
│  SINGLE L2 DOMAIN (one rack)                             │
│                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐               │
│  │  Node A  │  │  Node B  │  │  Node C  │               │
│  │  leader  │  │          │  │          │               │
│  │  for .17 │  │  leader  │  │          │               │
│  │          │  │  for .18 │  │          │               │
│  └────┬─────┘  └────┬─────┘  └──────────┘               │
│       │ GARP         │ GARP                               │
│       ▼              ▼                                    │
│  ═══════════════════════════════════ L2 switch ═══════    │
│                                                          │
│  ARP table:                                              │
│    203.0.113.17 → MAC(Node A)                            │
│    203.0.113.18 → MAC(Node B)                            │
│                                                          │
└──────────────────────────────────────────────────────────┘

Key Difference from BGP

With BGP, the ToR switch learns /32 → next-hop from BGP UPDATE messages. With L2, the switch learns IP → MAC from ARP. The switch does not know which IP is on which node at the routing level — it knows at the MAC table level. This distinction matters for multi-rack: a spine switch cannot forward based on MAC entries from a different rack's L2 domain.

When to Use

WHEN all nodes sharing public IPs are on the same L2 segment
WHEN BGP is not available or not desired
WHEN the deployment is single-rack or uses a flat L2 fabric

When NOT to Use

WHEN the topology is multi-rack with L3 boundaries between racks
WHEN the ToR switch needs a BGP route entry per IP
WHEN integration with a spine/leaf BGP fabric is required

Tier 5: Custom Controller with Hybrid Primitives

Idiomaticity: ★★☆☆☆ — Significant custom code required

When no single tier above fully fits, a custom controller can bridge the gaps by combining Kubernetes and Cilium primitives.

Pattern

┌──────────────────────────────────────────────────────────────┐
│  CUSTOM CONTROLLER                                           │
│                                                              │
│  Watches:                                                    │
│  ┌──────────────────────────────────────────────────────┐    │
│  │ VMInstance CRD   │ Node objects  │ IPAddress objects  │    │
│  └──────────────────────────────────────────────────────┘    │
│                                                              │
│  Reconciles:                                                 │
│  ┌──────────────────────────────────────────────────────┐    │
│  │ 1. VMInstance created with .spec.publicIP             │    │
│  │ 2. Controller creates IPAddress object (uniqueness)   │    │
│  │ 3. Controller creates/updates Service (for BGP adv)   │    │
│  │ 4. Controller sets VMInstance.status.node              │    │
│  │ 5. On instance migration: updates Service endpoints   │    │
│  │ 6. On instance deletion: deletes IPAddress + Service  │    │
│  └──────────────────────────────────────────────────────┘    │
│                                                              │
│  Relies on Cilium for:                                       │
│  ┌──────────────────────────────────────────────────────┐    │
│  │ BGP session management                                │    │
│  │ Service IP advertisement                              │    │
│  │ Route injection/withdrawal                            │    │
│  └──────────────────────────────────────────────────────┘    │
│                                                              │
└──────────────────────────────────────────────────────────────┘

When to Use

WHEN the IP assignment source is external (DHCP) AND you need Kubernetes-API-level tracking
WHEN instance lifecycle does not map cleanly to Service lifecycle
WHEN admission-time validation of IP ownership is required
WHEN the VM management layer does not integrate with Kubernetes Services

IP Ownership and Authority

The mapping is only as trustworthy as the IP assignment mechanism. A system that advertises IPs it does not authoritatively own is a route hijack vector.

Authority Models

┌────────────────────────────────────────────────────────────────┐
│  MODEL 1: Cilium LB IPAM as Authority                         │
│                                                                │
│  CiliumLoadBalancerIPPool defines the range                    │
│  LB IPAM assigns IPs to Services                              │
│  → Authority is the Cilium operator                            │
│  → Spoofing requires Service create/update permission          │
│  → Mitigated by RBAC + namespace isolation                     │
│                                                                │
│  TRUST CHAIN:                                                  │
│  Pool (admin-created) → IPAM (operator) → Service (RBAC) → BGP│
├────────────────────────────────────────────────────────────────┤
│  MODEL 2: External DHCP as Authority                           │
│                                                                │
│  DHCP server assigns IP to MAC                                 │
│  VM receives IP via DHCP                                       │
│  IP appears on host interface                                  │
│  Cilium advertises whatever is on the interface                │
│  → Authority is the DHCP server                                │
│  → Spoofing requires ARP spoofing or static IP override        │
│  → Mitigated by DHCP snooping, port security, MAC filtering    │
│                                                                │
│  TRUST CHAIN:                                                  │
│  DHCP config (admin) → DHCP lease → VM interface → host        │
│  interface → Cilium → BGP                                      │
├────────────────────────────────────────────────────────────────┤
│  MODEL 3: Kubernetes IPAddress API as Authority                │
│                                                                │
│  Controller creates IPAddress objects                           │
│  Name = IP (uniqueness enforced by API server)                 │
│  parentRef = owning VMInstance                                  │
│  → Authority is the controller + API server uniqueness          │
│  → Spoofing requires IPAddress create permission               │
│  → Mitigated by RBAC + ValidatingAdmissionPolicy               │
│  → parentRef is immutable from v1.36 alpha onward              │
│                                                                │
│  TRUST CHAIN:                                                  │
│  Controller (RBAC) → IPAddress (API server) → parentRef → CRD  │
└────────────────────────────────────────────────────────────────┘

Admission-Time Validation

IP ownership can be enforced at three levels:

LEVEL 1: RBAC
  WHO can create Services / IPAddress objects / CRDs?
  Coarse-grained. Prevents unauthorized actors from claiming IPs,
  but does not prevent authorized actors from claiming WRONG IPs.

LEVEL 2: ValidatingAdmissionPolicy (CEL)
  WHAT IP ranges are allowed for a given namespace / label / owner?
  Medium-grained. Can enforce "this namespace can only use IPs from
  this CIDR." Cannot cross-reference other objects (CEL has no API
  call capability — only object, oldObject, request, params,
  namespaceObject, variables, and authorizer are available).

LEVEL 3: ValidatingAdmissionWebhook
  IS this specific IP already assigned to another instance?
  Fine-grained. Can query the API server to check for existing
  IPAddress objects with the same name. Adds availability dependency.

BGP Topology Design Patterns

Pattern 1: Per-Rack ASN with Auto-Discovery

Each rack has its own ASN. Nodes auto-discover their ToR switch via default gateway. This is the most common leaf-spine BGP design.

                    ┌────────────────┐
                    │  Spine Switch  │
                    │  ASN 65000     │
                    └──┬──────────┬──┘
                       │ eBGP     │ eBGP
              ┌────────┘          └────────┐
              │                            │
     ┌────────┴────────┐         ┌─────────┴───────┐
     │  ToR A          │         │  ToR B           │
     │  ASN 65001      │         │  ASN 65002       │
     └──┬─────┬────┬───┘         └──┬─────┬────┬───┘
        │     │    │                │     │    │
      Nodes with               Nodes with
      localASN: 65001          localASN: 65002
      auto-discover            auto-discover
      ToR A as peer            ToR B as peer

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: rack-a-peering
spec:
  nodeSelector:
    matchLabels:
      network.example.com/rack: rack-a
  bgpInstances:
  - name: "rack-a"
    localASN: 65001
    peers:
    - name: "tor"
      peerASN: 65001    # iBGP within the rack
      autoDiscovery:
        mode: "DefaultGateway"
        defaultGateway:
          addressFamily: ipv4
      peerConfigRef:
        name: "standard-peer"

Pattern 2: Shared ASN with NodeSelector Partitioning

All nodes share an ASN, but nodeSelector on CiliumBGPClusterConfig partitions which nodes peer with which ToR switches.

# One CiliumBGPClusterConfig per rack
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: rack-a
spec:
  nodeSelector:
    matchLabels:
      network.example.com/rack: rack-a
  bgpInstances:
  - name: "peering"
    localASN: 65100     # same ASN everywhere
    peers:
    - name: "tor-a"
      peerASN: 65000
      peerAddress: "10.0.1.1"     # ToR A management IP
      peerConfigRef:
        name: "standard-peer"
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: rack-b
spec:
  nodeSelector:
    matchLabels:
      network.example.com/rack: rack-b
  bgpInstances:
  - name: "peering"
    localASN: 65100
    peers:
    - name: "tor-b"
      peerASN: 65000
      peerAddress: "10.0.2.1"     # ToR B management IP
      peerConfigRef:
        name: "standard-peer"

Pattern 3: Per-Node Override for Multi-Homed Nodes

Some nodes connect to two ToR switches (dual-homed for redundancy). Use CiliumBGPNodeConfigOverride for per-node peer address customization.

apiVersion: cilium.io/v2
kind: CiliumBGPNodeConfigOverride
metadata:
  name: node-a-04    # MUST match the node name
spec:
  bgpInstances:
  - name: "peering"  # MUST match the instance name in ClusterConfig
    routerID: "10.0.1.104"
    peers:
    - name: "tor-primary"
      localAddress: "10.0.1.104"
      localPort: 179              # optional per-peer port override
    - name: "tor-secondary"
      localAddress: "10.0.2.104"
      localPort: 179

Observability and Verification

CiliumBGPNodeConfig Status

Every node with active BGP sessions has a CiliumBGPNodeConfig resource auto-generated by the Cilium operator. Its .status field is the authoritative source for BGP session state:

$ kubectl describe ciliumbgpnodeconfig node-a-04

Status:
  Bgp Instances:
    Local ASN:  65001
    Name:       rack-a
    Peers:
      Established Time:  2026-05-14T10:23:45Z
      Name:              tor
      Peer ASN:          65001
      Peer Address:      10.0.1.1
      Peering State:     established
      Route Count:
        Advertised:  3
        Afi:         ipv4
        Received:    12
        Safi:        unicast

This tells you:

Peering State: is the BGP session up?
Route Count (Advertised): how many /32 routes is this node announcing?
Route Count (Received): how many routes has the ToR sent to this node?
Established Time: when did the session come up?

cilium-dbg Commands

From inside the Cilium agent pod:

# Show BGP peer sessions
cilium-dbg bgp peers

# Show advertised and received routes
cilium-dbg bgp routes

# Show GoBGP route policies (export filters, communities)
cilium-dbg bgp route-policies

Verifying the Mapping

To verify the full IP → Instance → Node → ToR chain:

# 1. Find which Service owns the IP
kubectl get svc -A -o wide | grep "203.0.113.17"

# 2. Find which node has the endpoint
kubectl get endpointslices -A -o json | \
  jq '.items[] | select(.endpoints[].addresses[] == "203.0.113.17") |
      {service: .metadata.labels["kubernetes.io/service-name"],
       node: .endpoints[].nodeName}'

# 3. Find which rack the node is in
kubectl get node node-a-04 \
  -o jsonpath='{.metadata.labels.network\.example\.com/rack}'

# 4. Verify BGP advertisement from that node
kubectl exec -n kube-system ds/cilium -c cilium-agent -- \
  cilium-dbg bgp routes advertised ipv4 unicast | grep "203.0.113.17"

# 5. Check BGP session state
kubectl get ciliumbgpnodeconfig node-a-04 -o yaml | \
  yq '.status.bgpInstances[].peers[]'

Status Conditions on CiliumBGPClusterConfig

The CiliumBGPClusterConfig reports conditions:

cilium.io/NoMatchingNode — nodeSelector does not match any nodes
cilium.io/ConflictingClusterConfig — two configs select the same node

Check for these to catch misconfigurations before they cause silent failures.

Lifecycle Management

Instance Creation

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│ Instance │    │  Service  │    │ LB IPAM  │    │   BGP    │
│ Created  │───▶│ Created   │───▶│ Assigns  │───▶│Advertises│
│          │    │ type: LB  │    │  .17     │    │ .17/32   │
└──────────┘    └──────────┘    └──────────┘    └──────────┘
     t=0s            t=0s           t~1s           t~2-5s

Instance Migration

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│ Instance │    │ Endpoint │    │ Old Node │    │ New Node │
│ Migrates │───▶│  Moves   │───▶│Withdraws │───▶│Advertises│
│ A → B    │    │ A → B    │    │ .17/32   │    │ .17/32   │
└──────────┘    └──────────┘    └──────────┘    └──────────┘
     t=0           t~1s           t~2-5s          t~2-5s

The convergence window between withdrawal and re-advertisement is the danger zone. During this window, traffic may be black-holed (if the old route is withdrawn before the new one is advertised) or duplicated (if both nodes briefly advertise the same /32).

Instance Deletion

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│ Instance │    │  Service  │    │ LB IPAM  │    │   BGP    │
│ Deleted  │───▶│ Deleted   │───▶│ Frees    │───▶│Withdraws │
│          │    │           │    │  .17     │    │ .17/32   │
└──────────┘    └──────────┘    └──────────┘    └──────────┘
     t=0s            t=0s           t~1s           t~2-5s

Node Drain

Draining a node requires careful ordering to avoid black-holing:

CORRECT ORDER:
  1. Cordon the node (no new scheduling)
  2. Evict instances (they migrate to other nodes)
  3. Wait for BGP routes to reconverge on new nodes
  4. Remove BGP peering label (shuts down BGP sessions)
  5. Wait for ToR to remove routes (hold timer expiry)
  6. Shut down the node

INCORRECT ORDER (causes black-holing):
  1. Shut down the node immediately
     → BGP session drops
     → ToR holds routes until hold timer expires (90s default)
     → Traffic arrives at dead node for up to 90 seconds

Graceful Restart

BGP Graceful Restart allows the ToR switch to retain routes from a node during a Cilium agent restart. This prevents route withdrawal during upgrades. Configure in CiliumBGPPeerConfig:

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: tor-peer-config
spec:
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120

Cilium Operator Down

When the Cilium operator is down:

Existing BGP sessions continue (they run in the Cilium agent, not the operator)
Existing advertisements continue
New CiliumBGPNodeConfig resources are NOT generated
LB IPAM allocations stop
Changes to CiliumBGPClusterConfig are not reconciled

This means existing mappings survive operator outage, but new instances cannot be provisioned.

Anti-Patterns and Footguns

NEVER: Mix BGPv1 and BGPv2 CRDs

CiliumBGPPeeringPolicy (v1) and CiliumBGPClusterConfig (v2) must not coexist on the same cluster. If both match a node, CiliumBGPPeeringPolicy takes precedence — silently overriding the v2 config. This causes impossible-to-debug routing inconsistencies.

NEVER: Use `cilium.io/v2alpha1` for BGP CRDs

The v2alpha1 API versions of all Cilium BGP CRDs are deprecated. Use cilium.io/v2 for CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, CiliumBGPNodeConfig, and CiliumBGPNodeConfigOverride.

NEVER: Use ServiceCIDR for VM Public IPs

ServiceCIDR is strictly for Service ClusterIP allocation. Using it for VM public IPs will conflict with the Service allocator and produce unpredictable behavior.

NEVER: Rely on Node.Status.Addresses[ExternalIP] for VM IPs

ExternalIP is a node-level field with undefined semantics. It may be a NAT'd IP, a cloud provider IP, or absent entirely. It is not designed to track per-VM IPs.

NEVER: Build New Architecture on Service.spec.externalIPs

Service.spec.externalIPs is on a deprecation path (KEP 5707, AllowServiceExternalIPs feature gate). Any user with Service create/update permission can add any IP, making it a route hijack vector. The DenyServiceExternalIPs admission plugin can block it entirely. Use LoadBalancer Services with LB IPAM instead.

NEVER: Set aggregationLengthIPv4 to 32 or aggregationLengthIPv6 to 128

The maximum valid values are 31 and 127 respectively (kubebuilder validation). Setting 32 or 128 will fail admission. To get /32 or /128 advertisement (which is the default), omit the aggregation fields entirely.

NEVER: Nest `selector` Inside `service:` on CiliumBGPAdvertisement

The selector field is on the BGPAdvertisement struct, not on BGPServiceOptions. It is a sibling of service:, not a child. Nesting it inside service: will cause the selector to be silently ignored (unknown fields are dropped by the API server with strict validation, or ignored with lenient validation).

NEVER: Assume Interface IP Advertisement Filters by CIDR

Cilium advertises ALL non-excluded IPs on the named interface. If the interface has management IPs, loopback IPs (other than the excluded ranges), or other non-VM IPs, they will be advertised as /32 routes.

ALWAYS: Enable bgpControlPlane.enabled=true

The BGP control plane is disabled by default. Without this Helm flag, all CiliumBGPClusterConfig, CiliumBGPAdvertisement, and related CRDs are no-ops.

ALWAYS: Label Nodes for Rack/ToR Identity

Without a label mapping nodes to their ToR switch, there is no Kubernetes-API-level mechanism to determine which switch a node peers with. The BGP session exists at the network level, but the cluster has no metadata about it.

kubectl label node node-a-01 network.example.com/rack=rack-a
kubectl label node node-a-01 network.example.com/tor-switch=tor-a
kubectl label node node-a-01 bgp-enabled=true

ALWAYS: Set externalTrafficPolicy: Local on Per-Instance Services

Without externalTrafficPolicy: Local, ALL nodes advertise the Service IP (when using BGP Service advertisement). This means the ToR switch receives the same /32 from every node and cannot determine the correct next hop.

With Local, only the node hosting the instance advertises the /32.

WATCH: bgpInstances MaxItems=16

CiliumBGPClusterConfig.spec.bgpInstances has a maximum of 16 entries. Designs requiring more than 16 BGP instances per node (e.g., many VRFs or per-tenant ASNs) will hit this limit.

WATCH: Overlapping CiliumBGPAdvertisement Selectors

When two CiliumBGPAdvertisement resources match the same Service:

Communities: union of all matches
Local Preference: conflict — last match wins (undefined ordering)
This can cause non-deterministic BGP attributes

WATCH: CiliumBGPNodeConfigOverride Name Must Match Node Name

The CiliumBGPNodeConfigOverride resource name must exactly match the Kubernetes node name. If they differ, the override is silently ignored. Similarly, the bgpInstances[].name and peers[].name must match the names in CiliumBGPClusterConfig.

WATCH: Hold Timer vs Instance Migration Time

If instance migration takes longer than the BGP hold timer:

Old node withdraws the route
ToR removes the route
Traffic is black-holed
New node eventually advertises the route
Traffic resumes

Ensure the hold timer (default 90s) exceeds the maximum expected migration time, or use Graceful Restart to hold routes during transitions.

WATCH: IPAddress ParentRef Immutability (v1.36+)

Starting with Kubernetes v1.36, IPAddress.spec.parentRef is immutable. To reassign an IP to a different instance, you must delete and recreate the IPAddress object. This is a deliberate safety mechanism to prevent accidental IP reassignment.

WATCH: peerASN Defaults to 0

If peerASN is omitted, it defaults to 0, which disables ASN validation in the BGP OPEN message. This is intentional for auto-discovery scenarios but can cause unexpected behavior if a specific ASN was intended. Always set peerASN explicitly when using manually specified peer addresses.

Decision Matrix

┌─────────────────────────────────────────────────────────────────────┐
│                        DECISION TREE                                │
│                                                                     │
│  Can instances be modeled as Service backends?                      │
│  ├── YES ──▶ Can you use LB IPAM for IP assignment?                │
│  │           ├── YES ──▶ TIER 1 (LB IPAM + Service BGP)            │
│  │           └── NO ───▶ TIER 5 (Custom controller + Service BGP)  │
│  │                                                                  │
│  └── NO ───▶ Are VM IPs visible on a host interface?               │
│              ├── YES ──▶ Do you need Kubernetes-level IP tracking? │
│              │           ├── YES ──▶ TIER 5 (Custom + Interface)   │
│              │           └── NO ───▶ TIER 2 (Interface BGP)        │
│              │                                                      │
│              └── NO ───▶ Can you allocate per-node CIDR slices?    │
│                          ├── YES ──▶ TIER 3 (Multi-Pool +          │
│                          │           CiliumPodIPPool advert)       │
│                          └── NO ───▶ Are all nodes on same L2?     │
│                                      ├── YES ──▶ TIER 4 (L2)      │
│                                      └── NO ───▶ TIER 5 (Custom)  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Feature Comparison

┌──────────────────┬────────┬────────┬────────┬────────┬────────┐
│ Feature          │ Tier 1 │ Tier 2 │ Tier 3 │ Tier 4 │ Tier 5 │
├──────────────────┼────────┼────────┼────────┼────────┼────────┤
│ Per-IP /32 BGP   │  ✓     │  ✓     │  ✗     │  N/A   │  ✓     │
│ Auto IP assign   │  ✓     │  ✗     │  ✓     │  ✓     │  ~     │
│ Auto advertise   │  ✓     │  ✓     │  ✓     │  ✓     │  ✓     │
│ Auto withdraw    │  ✓     │  ✓     │  ✓     │  ✓     │  ✓     │
│ IP→Instance API  │  ✓     │  ✗     │  ✗     │  ✗     │  ✓     │
│ Node-local only  │  ✓     │  ✓     │  ~     │  ✓     │  ✓     │
│ Multi-rack       │  ✓     │  ✓     │  ✓     │  ✗     │  ✓     │
│ Stable IP on     │  ✓     │  ~     │  ✗     │  ✓     │  ✓     │
│   migration      │        │        │        │        │        │
│ No custom code   │  ✓     │  ✓     │  ✓     │  ✓     │  ✗     │
│ Scale (10k IPs)  │  ~     │  ✓     │  ✓     │  ~     │  ~     │
├──────────────────┼────────┼────────┼────────┼────────┼────────┤
│ Custom code      │ none   │ none   │ none   │ none   │ ctrl   │
│ required         │        │        │        │        │        │
└──────────────────┴────────┴────────┴────────┴────────┴────────┘

  ✓ = fully supported    ✗ = not supported    ~ = partial / depends

Reference Manifests

Complete Tier 1 Stack

The following manifests constitute a complete, deployable Tier 1 configuration for a two-rack cluster.

---
# === IP POOL ===
# Defines the public IP range available for VM Services
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: public-vm-pool
spec:
  blocks:
  - cidr: "203.0.113.0/24"
  serviceSelector:
    matchLabels:
      purpose: vm-public-ip

---
# === BGP PEER CONFIG ===
# Shared peer configuration for all ToR switch sessions
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
  name: tor-peer
spec:
  transport:
    peerPort: 179
  timers:
    holdTimeSeconds: 90
    keepAliveTimeSeconds: 30
  gracefulRestart:
    enabled: true
    restartTimeSeconds: 120
  families:
  - afi: ipv4
    safi: unicast
    advertisements:
      matchLabels:
        advertise: vm-public-ips

---
# === BGP ADVERTISEMENT ===
# Defines what gets injected into the BGP RIB
# NOTE: selector is a SIBLING of service:, not nested inside it
# NOTE: aggregation fields omitted → default /32 advertisement
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
  name: vm-public-ip-advertisement
  labels:
    advertise: vm-public-ips
spec:
  advertisements:
  - advertisementType: "Service"
    service:
      addresses:
      - LoadBalancerIP
    selector:
      matchLabels:
        purpose: vm-public-ip
    attributes:
      communities:
        standard:
        - "65000:100"

---
# === RACK A BGP CONFIG ===
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: rack-a-bgp
spec:
  nodeSelector:
    matchLabels:
      network.example.com/rack: rack-a
  bgpInstances:             # MaxItems: 16
  - name: "rack-a-tor"
    localASN: 65001
    peers:
    - name: "tor-a"
      peerASN: 0            # learn ASN from peer OPEN message
      autoDiscovery:
        mode: "DefaultGateway"
        defaultGateway:
          addressFamily: ipv4
      peerConfigRef:
        name: "tor-peer"

---
# === RACK B BGP CONFIG ===
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: rack-b-bgp
spec:
  nodeSelector:
    matchLabels:
      network.example.com/rack: rack-b
  bgpInstances:
  - name: "rack-b-tor"
    localASN: 65002
    peers:
    - name: "tor-b"
      peerASN: 0
      autoDiscovery:
        mode: "DefaultGateway"
        defaultGateway:
          addressFamily: ipv4
      peerConfigRef:
        name: "tor-peer"

---
# === EXAMPLE: Per-Instance Service ===
apiVersion: v1
kind: Service
metadata:
  name: vm-web-1
  labels:
    purpose: vm-public-ip
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    vm.example.com/instance: vm-web-1
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP

Node Labeling

# Rack A nodes
for node in node-a-{01..06}; do
  kubectl label node $node \
    network.example.com/rack=rack-a \
    network.example.com/tor-switch=tor-a \
    bgp-enabled=true
done

# Rack B nodes
for node in node-b-{01..06}; do
  kubectl label node $node \
    network.example.com/rack=rack-b \
    network.example.com/tor-switch=tor-b \
    bgp-enabled=true
done

Helm Values for Cilium

bgpControlPlane:
  enabled: true

# Required for LB IPAM
kubeProxyReplacement: true

# If using native routing instead of encapsulation
routingMode: native
ipv4NativeRoutingCIDR: "10.0.0.0/8"

# Operator high availability (recommended for BGP)
operator:
  replicas: 3

Glossary

Term	Definition
ToR	Top-of-Rack switch. The first L3 hop for nodes in a rack.
ASN	Autonomous System Number. Identifies a BGP routing domain.
eBGP	External BGP. Sessions between different ASNs.
iBGP	Internal BGP. Sessions within the same ASN.
RIB	Routing Information Base. The BGP route table.
GoBGP	The BGP implementation embedded in the Cilium agent.
LB IPAM	LoadBalancer IP Address Management. Cilium's built-in IP allocator for Services.
/32	A single IPv4 host route. The most specific prefix possible.
/128	A single IPv6 host route.
GARP	Gratuitous ARP. An unsolicited ARP reply used to claim an IP on L2.
NDP	Neighbor Discovery Protocol. IPv6 equivalent of ARP.
Leaf-Spine	Datacenter network topology with leaf switches (ToR) connecting to spine switches.
Hold Timer	BGP parameter. If no keepalive is received within this time, the session is declared dead and routes are withdrawn. Default 90 seconds.
Graceful Restart	BGP feature that allows routes to be retained during a peer restart, preventing unnecessary traffic disruption.
PodCIDR	The IP range allocated to a node for pod networking.
EndpointSlice	Kubernetes API resource tracking Service backend endpoints and their node placement.
IPAddress	Kubernetes networking.k8s.io/v1 resource. A cluster-scoped singleton representing ownership of a single IP address. v1 introduced in Kubernetes 1.33.
ServiceCIDR	Kubernetes networking.k8s.io/v1 resource. Defines IP ranges for Service ClusterIP allocation. Has status conditions (`Ready`, `Terminating`).
CiliumBGPClusterConfig	Cilium CRD (`cilium.io/v2`) defining cluster-wide BGP topology. Selects nodes and defines BGP instances (max 16).
CiliumBGPPeerConfig	Cilium CRD (`cilium.io/v2`) defining shared BGP peer settings (timers, auth, address families).
CiliumBGPAdvertisement	Cilium CRD (`cilium.io/v2`) defining what prefixes are injected into the BGP RIB. Supports four types: PodCIDR, CiliumPodIPPool, Service, Interface.
CiliumBGPNodeConfig	Cilium CRD (`cilium.io/v2`) auto-generated per node by the operator. Source of truth for a node's BGP state.
CiliumBGPNodeConfigOverride	Cilium CRD (`cilium.io/v2`) for per-node BGP overrides (router ID, local port, peer addresses, peer local ports).
CiliumLoadBalancerIPPool	Cilium CRD defining IP pools for Service LoadBalancer allocation.
CiliumPodIPPool	Cilium CRD defining IP pools for direct workload IP allocation (Multi-Pool IPAM).
CiliumL2AnnouncementPolicy	Cilium CRD defining L2 (ARP/NDP) IP announcement policies.
NodeRestriction	Kubernetes admission plugin that limits which labels kubelets can set on their own Node objects.
ValidatingAdmissionPolicy	Kubernetes admission resource using CEL expressions for declarative validation at write time.
externalTrafficPolicy: Local	Service setting that restricts traffic handling to nodes with local endpoints only.
BGPCommunities	Cilium struct with three sub-fields: `standard` (RFC 1997 numeric), `wellKnown` (RFC 1997 string aliases), `large` (RFC 8092).

usrbinkat/README.md

Public IP Address Mapping in Multi-Rack Kubernetes Clusters

A Practitioner's Guide to IP → Instance → Node → ToR Switch Resolution

Table of Contents

The Mapping Problem

Why This Matters

Conceptual Model

Physical Topology

Logical Mapping

The Four Sources of Truth

Kubernetes-Native Primitives

IPAddress (networking.k8s.io/v1)

ServiceCIDR (networking.k8s.io/v1)

Node Object

Node.Status.Addresses

Node.Labels

Node.Spec.PodCIDRs

EndpointSlice (discovery.k8s.io/v1)

ValidatingAdmissionPolicy (admissionregistration.k8s.io/v1)

Service with externalTrafficPolicy

PodTopologyLabels Admission Plugin

Controller Pattern

Cross-Namespace ownerReferences

Cilium Primitives

CRD Relationship Map

Advertisement Types

PodCIDR

CiliumPodIPPool

Service

Interface

BGP Communities

LB IPAM

Multi-Pool IPAM

L2 Announcements

BGP Peer Auto-Discovery

Implementation Tiers

Tier Ranking Criteria

Tier 1: LB IPAM + BGP Service Advertisement

Architecture

Data Flow

Mapping Resolution

When to Use

When NOT to Use

Manifests

Tier 2: Interface IP Advertisement

Architecture

What Cilium Manages

What Cilium Does NOT Manage

Mapping Resolution

Manifest

When to Use

When NOT to Use

Footguns

Tier 3: Multi-Pool IPAM + BGP CiliumPodIPPool

Architecture

Trade-offs

When to Use

When NOT to Use

Tier 4: L2 Announcements

Architecture

Key Difference from BGP

When to Use

When NOT to Use

Tier 5: Custom Controller with Hybrid Primitives

Pattern

When to Use

IP Ownership and Authority

Authority Models

Admission-Time Validation

BGP Topology Design Patterns

Pattern 1: Per-Rack ASN with Auto-Discovery

Pattern 2: Shared ASN with NodeSelector Partitioning

Pattern 3: Per-Node Override for Multi-Homed Nodes

Observability and Verification

CiliumBGPNodeConfig Status

cilium-dbg Commands

Verifying the Mapping

Status Conditions on CiliumBGPClusterConfig

Lifecycle Management

Instance Creation

NEVER: Use `cilium.io/v2alpha1` for BGP CRDs

NEVER: Nest `selector` Inside `service:` on CiliumBGPAdvertisement