Skip to content

Instantly share code, notes, and snippets.

@denislemire

denislemire/CALICO_CLOUD_SERVICE_GRAPH_GIST.md

Last active June 11, 2026 03:38
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save denislemire/72591aa927c1e1dcee544a4d26953ef6 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save denislemire/72591aa927c1e1dcee544a4d26953ef6 to your computer and use it in GitHub Desktop.
Download ZIP
Calico Cloud Service Graph empty after Calico 3.32 upgrade — EhWS incident report
Raw
CALICO_CLOUD_SERVICE_GRAPH_GIST.md

Bug report: Calico Cloud Service Graph empty after Calico OSS 3.32 upgrade

Product: Calico OSS + Tigera Operator + Calico Cloud (managed cluster connection)
Versions: Calico 3.32.0, Tigera Operator v1.42.0
Related issues: tigera/operator#4804, tigera/operator#4913

Summary

After upgrading from Calico 3.31.x to 3.32.0, a cluster with an established Calico Cloud connection (ManagementClusterConnection) regressed from working Service Graph / flow telemetry to:

  • Calico Cloud UI: cluster Connected, but no Service Graph or flow data
  • Goldmane flow emitter TLS errors when posting to Guardian
  • Whisker unreachable via cluster ingress (separate policy gap)

The regression is reproducible on-cluster. It is not caused by Calico Cloud SaaS outage or stale connection credentials. Re-applying the ManagementClusterConnection manifest does not restore flow uploads.

Two distinct operator gaps explain the behavior:

  1. Calico 3.32 calico-system policy tier — OSS operator network policies for Guardian, Goldmane, and Whisker are incomplete; the tier’s default deny blocks required DNS, Kubernetes API, and inter-service paths.
  2. Goldmane emitter CA trust — the operator mounts only the local cluster tigera-operator-signer on Goldmane. The Calico Cloud flow-upload path presents certificates signed by a different tigera-operator-signer from tigera-voltron-linseed-certs-public (identical CN, different key). Goldmane cannot verify the TLS server certificate until both signers are trusted.

Environment (representative)

Item Value
Kubernetes Multi-node cluster, Calico as CNI
Calico OSS 3.32.0, Tigera Operator v1.42.0, eBPF dataplane
Calico Cloud Managed connection via ManagementClusterConnection
Observability CRs Goldmane/default, Whisker/default
Connection material tigera-managed-cluster-connection, tigera-voltron-linseed-certs-public (in tigera-operator namespace)

Regression path

  1. Upgrade Calico to v3.32.0 via Tigera Operator (documented upgrade path).
  2. Observe Guardian lose Voltron egress under the new calico-system policy tier → cluster may show Disconnected until supplemental egress is added.
  3. After restoring Guardian egress, Calico Cloud shows Connected again.
  4. Service Graph remains empty; Goldmane logs show flow emitter failures (DNS/API timeouts and/or TLS verify errors depending on which policy gaps remain).
  5. After supplemental network policies and merging the Cloud CA into goldmane-ca-bundle, Goldmane logs show sustained successful flow uploads.

Symptoms

Calico Cloud UI

  • Managed cluster status: Connected
  • Service Graph: empty
  • Connection diagnostics: setup appears correct

Goldmane logs (TLS failure — primary blocker after policy egress is fixed)

Error emitting flows to https://guardian.calico-system.svc.cluster.local:443/api/v1/flows/bulk:
tls: failed to verify certificate: x509: certificate signed by unknown authority
(possibly because of "crypto/rsa: verification error" while trying to verify
candidate authority certificate "tigera-operator-signer")

Goldmane logs (network policy — before kubernetes API egress)

Get "https://10.43.0.1:443/api": dial tcp 10.43.0.1:443: i/o timeout

(Goldmane cannot reach the Kubernetes API to read flow-emitter-state ConfigMap.)

Goldmane logs (after workaround — expected healthy state)

Successfully emitted flows after retries. bucket=emitter.bucketKey{...} retries=4

Components that continued to work

  • Felix → Goldmane gRPC ingest (Connection from client on all nodes)
  • Guardian → Voltron management tunnel (TLS dial ... succeeded)
  • Workload networking via Calico CNI
  • TigeraStatus for management-cluster-connection: Available: True (does not reflect emitter health)

Root cause 1: calico-system policy tier (OSS operator network policies)

Calico 3.32 introduces a calico-system policy tier (defaultAction: Deny) and operator-managed policies. On the OSS code path, several observability components ship incomplete rules:

Component Operator policy Missing rules
guardian calico-system.guardian-access (ingress only) Egress: DNS, Kubernetes API, Goldmane :7443, Voltron/management :443
goldmane calico-system.goldmane (ingress :7443 only) Egress: DNS, Kubernetes API, Guardian Service
whisker calico-system.whisker (egress only) Ingress: ingress controller in kube-system → :8081

The Enterprise Guardian render path calls AppendDNSEgressRules(); the OSS ossNetworkPolicy() path does not (guardian.go v1.42.0).

Community report: tigera/operator#4804 describes the same OSS egress gap.

Workaround: supplemental NetworkPolicy resources in the calico-system tier allowing the missing egress/ingress paths listed above.

Root cause 2: Goldmane emitter CA trust (Calico Cloud PKI)

Operator behavior (v1.42.0)

When ManagementClusterConnection is configured, pkg/render/goldmane/component.go:

  • Sets PUSH_URL=https://guardian.calico-system.svc.cluster.local:443/api/v1/flows/bulk
  • Sets CA_CERT_PATH to the operator TrustedCertBundle (local goldmane-ca-bundle ConfigMap)
  • Does not include the Calico Cloud tigera-operator-signer from tigera-voltron-linseed-certs-public

Goldmane uses the same CA_CERT_PATH for both its gRPC server and the HTTP flow emitter (goldmane/pkg/daemon/daemon.go v3.32.0).

TLS path for flow uploads

The Guardian Service maps port 443 → pod port 8080 (Voltron tunnel proxy). Flow bulk POSTs terminate TLS with the Cloud tigera-linseed leaf certificate (issuer: Cloud tigera-operator-signer), not the local guardian-key-pair serving material.

CA material comparison

Material Subject CN SHA-256 fingerprint (prefix)
Local goldmane-ca-bundle tigera-operator-signer 03:AE:C3:19:…
Cloud CA in tigera-voltron-linseed-certs-public (2nd PEM in tls.crt) tigera-operator-signer 5B:58:B7:69:…
management-cluster.crt (Voltron tunnel identity) tigera-voltron separate PKI

Observations:

  • Same subject CN, different keys — Goldmane cannot verify Cloud-presented certs using only the local CA.
  • The tigera-linseed leaf verifies against the Cloud tigera-operator-signer, not the local one.
  • A combined bundle (local + Cloud operator-signer) verifies both local mTLS paths and the emitter upload path.

Approaches tested (no sustained fix)

Approach Result
Re-apply ManagementClusterConnection manifest No change
Set SERVER_NAME=guardian.calico-system.svc.cluster.local on Goldmane TLS verify errors may pause briefly, then EOF on POST; no sustained uploads; operator reverts Deployment changes on reconcile
Mount client cert/key from tigera-managed-cluster-connection on Goldmane Did not restore uploads
Merge Cloud CA into goldmane-ca-bundle → tigera-ca-bundle.crt Successfully emitted flows in Goldmane logs

Suggested operator fix

When ManagementClusterConnection is present, populate Goldmane’s trusted CA material with a combined bundle: local tigera-operator-signer (for Felix→Goldmane gRPC) plus the Cloud tigera-operator-signer from tigera-voltron-linseed-certs-public. Tracked in tigera/operator#4913.

Workaround (until operator fix)

1. Supplemental calico-system tier network policies

Allow the egress/ingress paths listed in Root cause 1. Minimum for Goldmane emitter:

  • DNS to kube-dns / CoreDNS in kube-system
  • TCP to the kubernetes Service in default (emitter state ConfigMap)
  • TCP to the guardian Service in calico-system

2. Combined CA bundle on Goldmane

The operator may revert goldmane-ca-bundle on reconcile. A small reconciler (Deployment + RBAC) can:

  1. Read the local CA from ConfigMap goldmane-ca-bundle / key tigera-ca-bundle.crt
  2. Extract the Cloud tigera-operator-signer (second certificate in Secret tigera-voltron-linseed-certs-public, key tls.crt, tigera-operator namespace)
  3. Patch the ConfigMap to contain both PEM blocks when only one is present
  4. Re-run on an interval (e.g. 45s)

The reconciler pod needs calico-system tier egress to the Kubernetes API (same default-deny tier as Goldmane).

Verification

# Emitter health
kubectl logs -n calico-system deployment/goldmane --tail=50 \
  | grep -E 'Successfully emitted|Error emitting'

# Combined CA present (expect 2 certificates)
kubectl get configmap goldmane-ca-bundle -n calico-system \
  -o jsonpath='{.data.tigera-ca-bundle\.crt}' \
  | awk '/BEGIN CERTIFICATE/{n++} END{print n " certs in bundle"}'

# Optional: verify Cloud leaf against merged bundle
kubectl get secret tigera-voltron-linseed-certs-public -n tigera-operator \
  -o jsonpath='{.data.tls\.crt}' | base64 -d \
  | awk '/BEGIN CERTIFICATE/{n++} n==1,/END CERTIFICATE/{print}' > /tmp/linseed-leaf.pem
kubectl get configmap goldmane-ca-bundle -n calico-system \
  -o jsonpath='{.data.tigera-ca-bundle\.crt}' > /tmp/bundle.pem
openssl verify -CAfile /tmp/bundle.pem /tmp/linseed-leaf.pem
# Expect: OK

Documentation gaps

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment