ca-zonal-stop

ca-zonal-stop.md

Here's what Cluster Autoscaler will do in your scenario, end‑to‑end. The behavior is entirely driven by the fact that the Kubernetes Node objects still exist (kubelet stops, but the API server retains them) and the Azure cloud provider does not surface VM power state to CA.

How CA classifies the 33 stopped nodes

CA buckets every node every loop in clusterstate.go (updateReadinessStats). The buckets are mutually exclusive:

Bucket	Definition
Ready	K8s Node + NodeReady=True
Unready	K8s Node + NodeReady != True, older than MaxNodeStartupTime (15m)
NotStarted	Same, but inside the 15m startup window
Unregistered	Cloud‑provider instance with no matching K8s Node
LongUnregistered	Unregistered for longer than MaxNodeProvisionTime (15m)

Your 33 stopped VMs land in Unready — not Unregistered/LongUnregistered — because:

• The K8s Node objects are still in etcd (just NotReady).
• Azure's Nodes() keeps returning them as InstanceRunning. See azure_scale_set_instance_cache.go: the provider only consults VM power state when ProvisioningState == Failed and enableFastDeleteOnFailedProvisioning is set. A user‑stopped VM stays ProvisioningState=Succeeded → default: branch → InstanceRunning. Azure literally does not tell CA that those VMs are stopped.

This means the "old unregistered node" cleanup path (static_autoscaler.go removeOldUnregisteredNodes) is a no‑op for these nodes. So is deleteCreatedNodesWithErrors (no ErrorInfo) and fixNodeGroupSize (target still equals registered).

Cluster health check — does CA freeze?

Two health checks gate the loop. With 33/99 unready:

Cluster‑wide (clusterstate.go IsClusterHealthy):

totalUnready > OkTotalUnreadyCount(3)         AND
totalUnready > MaxTotalUnreadyPercentage(45%) * totalNodes
33 > 3  ✓     AND     33 > 0.45 * 99 = 44.55  ✗   →  HEALTHY

Per‑nodegroup (clusterstate.go IsNodeGroupHealthy): same 45% / 3 thresholds → also healthy.

Result: CA continues operating normally. The fail‑safe at static_autoscaler.go (which would return nil and skip both scale‑up and scale‑down) does not trigger.

If you killed roughly 12 more nodes (≥45%), CA would flip to "ClusterUnhealthy" and freeze the loop entirely — neither scaling up replacements nor deleting the stopped nodes.

What CA actually does, on a timeline

Assuming all defaults:

Time	Event
t=0	You stop 33 VMs via Azure. Pods still running, Nodes still Ready.
t≈40s	kube‑controller‑manager flips NodeReady to Unknown (--node-monitor-grace-period). Next CA loop (every --scan-interval, default 10s) buckets the 33 as Unready. CA continues — no immediate action.
t≈5m	kube‑controller‑manager evicts pods due to node.kubernetes.io/unreachable taint with default tolerationSeconds=300. Pods become Pending/Unschedulable.
t≈5m + scheduling latency	StaticAutoscaler.RunOnce finds unschedulable pods and calls scaleUpOrchestrator.ScaleUp(...) at static_autoscaler.go. For Azure VMSS this becomes ScaleSet.IncreaseSize() → ARM BeginCreateOrUpdate to bump VMSS capacity. Azure's own VMSS zone‑balancing decides where the new VMs land — CA doesn't pin them per‑AZ. Typically they fill the two healthy zones.
t≈25m	The 33 stopped/Unready nodes have now been "unneeded" for ≥ --scale-down-unready-time (default 20m). The unneeded tracker uses a per‑node threshold based on readiness (nodes.go):
Ready → ScaleDownUnneededTime (10m)
Unready → ScaleDownUnreadyTime (20m)
The actuator taints them with ToBeDeletedByClusterAutoscaler, then calls nodeGroup.DeleteNodes() → Azure ScaleSet.DeleteNodes → BeginDeleteInstances (azure_scale_set.go). Azure removes the stopped instances regardless of power state. Subject to MinSize, --max-scale-down-parallelism (10), --max-drain-parallelism (1).

Summary of the answer to "would CA do anything?"

Yes, two things — but only via the Unready code path, never via the Unregistered path:

1. Replace the lost capacity (~5 min in): as soon as the kubelet eviction makes pods Pending, CA scales the VMSS up to restore them. New VMs land per Azure's VMSS zone allocator.
2. Delete the stopped VMs (~20 min after they go Unready): governed by --scale-down-unready-time (default 20m) and --scale-down-unready-enabled (default true). CA issues BeginDeleteInstances and Azure removes them.

Notable edge cases:

• If you set --scale-down-unready-enabled=false, CA will never delete those 33 nodes — they'll stay Unready forever (Azure won't tell CA they're broken). Replacement scale‑up still works.
• If you stopped enough VMs to push the unready ratio above 45% (--max-total-unready-percentage), CA stops everything (logs ClusterUnhealthy, emits an event, returns from the loop). Neither replacement nor cleanup happens until you either restart VMs or manually kubectl delete node (which would reclassify them as Unregistered and let removeOldUnregisteredNodes clean up after --max-node-provision-time, default 15m).
• The --force-delete-unregistered-nodes flag and Azure's azure_force_delete_scale_set.go only apply to LongUnregistered nodes, so they're irrelevant here.
• CA never tries to "restart" a stopped VM — there is no such operation in the cloud‑provider interface. It only knows how to grow/shrink a node group.