kstatus - conditions and generations check

This document describes how the checker tool (CLI and library) can perform Kubernetes object status conditions and generations check as per kstatus.

Before running the tests, the checker is provided with some context about the controller:

The status conditions it supports with their polarity and priority of the conditions.
The scheme of the controller managed object.

Configuration

List of status conditions in the order of their priority.

# List of supported conditions.
conditions:
  negativePolarity:
    - FetchFailed
    - ArtifactOutdated
    - Stalled
    - Reconciling
  positivePolarity:
    - RemoteAvailable
    - SecretsAvailable

In the conditions above, among the negative polarity conditions, the first condition (FetchFailed) has the highest priority and the last condition (Reconciling) has the lowest priority. In case of multiple negative polarity conditions being present, the values of the condition with the highest polarity is also reflected in the Ready condition value.

Usage

The checker is executed whenever there's a need to verify if the current object's status is valid.

For tests that execute terminal commands, for example in smoke tests, the checker could be run after every execution of a change, like create an object, update an object, etc.

$ controller-check --config cc.yaml --name targetobj --namespace targetns --kind Foo

Similarly, for go tests, the checker could be run after every change by calling the checker function.

conditions := cc.Conditions{NegativePolarity: []string{C1, C2, C3}}
checker := cc.NewChecker(client, scheme, conditions)
checker.CheckErr(ctx, obj)

Status condition check

Given an object, ensure that:

There are no negative polarity conditions set when Ready condition is True. Suggest that the negative polarity condition can be removed. (WARN0001)
There are no True negative polarity conditions when Ready condition is True. (FAIL0001)
There is Ready condition set always with value True or False based on the situation. (FAIL0002)
When there are multiple negative polarity conditions with True value present, the Ready condition value is equal to the value of the negative polarity condition with the highest priority. (WARN0003)

If the controller supports Reconciling condition, ensure that:

Ready condition is False when Reconciling condition is True. (FAIL0003)
Reconciling condition is not present when its value is False. (WARN0003)

If the controller supports Stalled condition, ensure that:

Ready condition is False when Stalled condition is True. (FAIL0004)
Stalled condition is not present when its value is False. (WARN0004)

If the controller supports both Reconciling condition and Stalled condition, ensure that:

Reconciling condition and Stalled condition are mutually exclusive to one another. (FAIL0005)
In presence of Reconciling condition or Stalled condition, Ready condition is always set to False. (FAIL0004, FAIL0005)

NOTE: Reconciling condition and Stalled condition conditions need not be present whenever Ready condition is False. There could be situation where an object is not ready and not reconciling, like a transient error due to unavailable of some resource.

Status observed generation and conditions check

Given an object, ensure that:

The status root ObservedGeneration and the status conditions ObservedGeneration are always less than or equal to the object Generation. (FAIL0006)
When the status root ObservedGeneration or the status conditions ObservedGeneration is less than the object Generation, the object's Ready condition status is False. It indicates that the desired state and the actual state aren't equal. (FAIL0007, FAIL0008)
When the Ready condition is True, all the status conditions' ObservedGeneration equal to the object status root ObservedGeneration. (FAIL0009)
All the status conditions have ObservedGeneration defined. (WARN0005)
When the Reconciling condition is True, the Reconciling condition ObservedGeneration is greater than the status root ObservedGeneration, if present. (FAIL0010)

NOTE: All the status conditions' ObservedGeneration, root ObservedGeneration and object Generation can be equal when Ready condition is True or Stalled condition is True.

Warning and Failure codes

Warnings:

Code	Message
WARN0001	Negative polarity condition present when `Ready` condition is True
WARN0002	`Ready` condition should have the value of the negative polarity conditon that's present with the highest priority
WARN0003	`Reconciling` condition can be removed when its value is False
WARN0004	`Stalled` condition can be removed when its value is False
WARN0005	Missing ObservedGeneration from status condition

Failures:

Code	Message
FAIL0001	Negative polarity condition cannot be True when `Ready` condition is True
FAIL0002	`Ready` condition must always be present
FAIL0003	`Ready` condition must be False when `Reconciling` condition is True
FAIL0004	`Ready` condition must be False when `Stalled` condition is True
FAIL0005	Only one of `Reconciling` condition or `Stalled` condition must be present at a time
FAIL0006	The ObservedGeneration must be less than or equal to the object Generation
FAIL0007	`Ready` condition must be False when the ObservedGeneration is less than the object Generation
FAIL0008	`Ready` condition must be False when any of the status condition's ObservedGeneration is less than the object Generation
FAIL0009	The status conditions' ObservedGenerations must be equal to the root ObservedGeneration when `Ready` condition is True
FAIL0010	The root ObservedGeneration must be less than the `Reconciling` condition ObservedGeneration when `Reconciling` condition is True

Go package implementation: https://pkg.go.dev/github.com/darkowlzz/controller-check@main/status