Skip to content

Instantly share code, notes, and snippets.

@stevendborrelli

stevendborrelli/crossplane-1.14.md

Last active January 8, 2024 08:29
Show Gist options
  • Save stevendborrelli/f94d796076490010b4f03dede9dd886d to your computer and use it in GitHub Desktop.
Save stevendborrelli/f94d796076490010b4f03dede9dd886d to your computer and use it in GitHub Desktop.
Download ZIP
Crossplane 1.14 Overview
Raw
crossplane-1.14.md

A Tour of Crossplane 1.14

Crossplane is an Open-Source project that enables management of almost anything in the Cloud via the Kubernetes API.

Crossplane version 1.14 released November 1, 2023, is one of the most consequential releases in the project's history.

While Composition Functions and Provider performance improvements are headline features, there are substantial updates across the project: from a focus on developer and operator experience, Upbound's code donation to the CNCF, to API maturation.

This Document was updated November 7, 2023.

Upbound's Code Donation to the CNCF Crossplane Project and Expanded Charter

One of the biggest issues in the Crossplane community was confusion between the CNCF Crossplane project and Upbound. For example, there is a community AWS Provider that was part of the Crossplane Organization and an Official AWS Provider supported by Upbound.

On September 19th, Upbound announced the donation of the Upjet provider generation tool and providers for AWS, Azure and Google Cloud. Upbound will also migrate some functionality from the up CLI to Crossplane.

The Crossplane Charter has been expanded to include Providers and Developer Experience.

As explained in the Crossplane Blog:

"This move makes Crossplane the only complete cloud automation solution governed entirely under a foundation."

The effect of all these changes will strengthen the Crossplane project with richer cloud support and better development experience. It also guarantees users that they won't have to worry about a vendor changing the license to the software they depend on.

A Note about Crossplane API Versioning

Crossplane APIs are versioned following Kubernetes API Conventions. For example, in 1.14 Functions are moving from v1alpha1 to v1beta1.

Alpha APIs like v1alpha1 are for experimentation. They need to be explicitly enabled using a feature flag and there is no guarantee that the API will not change or even remain part of Crossplane.

Beta APIs indicate the API is mature, but the implementation needs to mature. Beta APIs are enabled in Crossplane by default.

Finally, stable APIs are versioned v1, v2, etc. after the beta code has matured.

Composition Functions

API Status: v1beta1 Enabled by default in 1.14.

Kubernetes is fundamentally a declarative system. Users apply a desired state to the cluster and Controllers are responsible for implementing the desired state. In Kubernetes this desired state is a manifest usually created in YAML or JSON that doesn't support any variables or logic.

Crossplane has worked around these limitations with Object References and Composition Patch and Transforms. However, it's difficult to represent more complex logic like conditions or loops in Composition, and the project maintainers are adamant about not creating another Domain Specific Language(DSL) out of Composition.

Composition Functions were first released in 1.11.0 as an v1alpha1. Feedback on the v1alpha1 design led to the v1beta design coming in 1.14.

Major design changes between the v1alpha1 and v1beta1 design include:

  • Migrating Function execution from a Crossplane-based runner to a Kubernetes Deployment.
  • Function installation and configuration has been moved out of the Composition to a Function type configured via a DeploymentRuntimeConfig
  • The FunctionIO struct has been replaced with RunFunctionRequest/RunFunctionResponse protocol buffer messages sent to the Function's service endpoint.
  • Crossplane manages The Mutual TLS (mTLS) between the Crossplane pod and Function pod.
  • Patch and Transform is disabled when Function Pipeline mode is enabled.

Crossplane Function Architecture

Crossplane functions run in a pipeline of steps. In each pipeline step the function mutates a desired state and passes it on the the next function.

The initial input to the first function is the observed state, with the desired state only holding the information from the Claim's desired state.

Function Pipeline

After the final step is run, the desired state is converted to Managed Resources and applied to the Kubernetes API server. Providers watching for Managed Resource types then provision the resources.

Installing and Running Functions

Functions are packaged as an OCI (Docker) image. To install, apply a Function object to your cluster that points to your function image.

The Crossplane package manager will create the Kubernetes Deployment, ServiceAccounts, Service, and TLS Secret for your function.

apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: crossplane-contrib-function-patch-and-transform
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.1.4

To list installed functions, run kubectl get function:

# kubectl get function
NAME                                              INSTALLED   HEALTHY   PACKAGE                                                                  AGE
crossplane-contrib-function-patch-and-transform   True        True      xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.1.4   3d22h

Logs are available from the function's pod:

# kubectl logs -n crossplane-system crossplane-contrib-function-patch-and-transform-218a156adelgg7b  | head -3
{"level":"info","ts":1696585022.3244684,"caller":"fn/fn.go:31","msg":"Running Function","tag":""}
{"level":"info","ts":1696585022.341672,"caller":"fn/fn.go:31","msg":"Running Function","tag":""}
{"level":"info","ts":1696585082.3578289,"caller":"fn/fn.go:31","msg":"Running Function","tag":""}

function-patch-and-transform

function-patch-and-transform is the most notable new function. It takes the built-in Patch & Transform engine and runs it as a function in a Composition pipeline.

While in the past updates to the built-in Patch and Transform engine could take months to implement, with the engine broken out into a function advanced users can now fork and use their own implementations.

Existing Compositions only need minor changes to run as a function, and if resources share the same name the patch-and-transform function can modify Objects defined earlier in the pipeline.

In the example below, we can see it defined as a step in the pipeline, with all the resources and patchSets now under the function's input:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: dynamo-with-bucket
spec:
  compositeTypeRef:
    apiVersion: database.example.com/v1alpha1
    kind: NoSQL
  mode: Pipeline
  pipeline:
  - step: patch-and-transform
    functionRef:
      name: function-patch-and-transform
    input:
      apiVersion: pt.fn.crossplane.io/v1beta1
      kind: Resources
      resources:
      - name: s3Bucket
        base:
          apiVersion: s3.aws.upbound.io/v1beta1
          kind: Bucket
          metadata:
            name: crossplane-quickstart-bucket
          spec:
            forProvider:
              region: us-east-2
        patches:
        - type: FromCompositeFieldPath
          fromFieldPath: "location"
          toFieldPath: "spec.forProvider.region"
          transforms:
          - type: map
            map: 
              EU: "eu-north-1"
              US: "us-east-2"
      - name: dynamoDB
        base:
          apiVersion: dynamodb.aws.upbound.io/v1beta1
          kind: Table
          metadata:
            name: crossplane-quickstart-database
          spec:
            forProvider:
              region: "us-east-2"
              writeCapacity: 1
              readCapacity: 1
              attribute:
              - name: S3ID
                type: S
              hashKey: S3ID
        patches:
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.location"
          toFieldPath: "spec.forProvider.region"
          transforms:
          - type: map
            map: 
              EU: "eu-north-1"
              US: "us-east-2"

Updating Compositions to use function-patch-and-transform

There are several steps to migrate existing Compositions to a step in a pipeline using function-patch-and-transform:

  1. Set up the function in the Composition
    1. Enable mode: Pipeline. This will disable the built-in Patch and Transform engine for this Composition.
    2. Create the pipeline step for patch-and-transform
    3. Set the functionRef to the name of the function
  2. Indent existing resources & patchsets under Input
  3. Update transforms to have their type specified
  4. Update any ConnectionDetails to have type and name set
  5. All resources must have a name. This was suggested but not enforced in previous versions of Crossplane.
Built-in Composition Using Functions 
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xeks.aws.platformref.upbound.io
spec:
  compositeTypeRef:
    apiVersion: aws.platformref.upbound.io/v1alpha1
    kind: XCluster
  writeConnectionSecretsToNamespace: upbound-system
  patchSets: []
  resources: []

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xeks.aws.platformref.upbound.io
spec:
  compositeTypeRef:
    apiVersion: aws.platformref.upbound.io/v1alpha1
    kind: XCluster
  writeConnectionSecretsToNamespace: upbound-system
  mode: Pipeline   #This disables existing patch & transform
  pipeline:
  - step: patch-and-transform
    functionRef:
      name: crossplane-contrib-function-patch-and-transform
    # Move existing patchSets and resources here
    input:   
      apiVersion: pt.fn.crossplane.io/v1beta1
      kind: Resources
      patchSets: []
      resources: []
 
transforms:
- type: string
    string:
    fmt: "%s-eks" #type: was optional previously
 
transforms:
- type: string
    string:
    type: Format
      fmt: "%s-eks"
 
connectionDetails:
 # name and type were optional
 - fromConnectionSecretKey: kubeconfig
 
connectionDetails:
 - fromConnectionSecretKey: kubeconfig
   name: kubeconfig
   type: FromConnectionSecretKey

Migrating Compositions to Pipelines using the crossplane-migrator tool

crossplane-migrator has support for converting existing Compositions to use the new function-patch-and-transform:

crossplane-migrator new-pipeline-composition -i composition.yaml -o composition-pipeline.yaml

By default, the functionNameRef is set to function-patch-and-transform. If installing a function via a Crossplane Configuration package, the package organization will be added (i.e. crossplane-contrib-function-patch-and-transform).

crossplane-migrator new-pipeline-composition --function-name crossplane-contrib-function-patch-and-transform -i composition.yaml

function-template-go

function-template-go allows users to create a Crossplane desired state using Go's text/template engine.

For example, we can define a GCP bucket that skips patch-and-transform to set the fields directly, and allows the optional configuration of Bucket Versioning via a user setting spec.parameters.versioningEnabled:

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xstoragebuckets.platform.upbound.io
  labels:
    provider: gcp
    type: generic
spec:
  writeConnectionSecretsToNamespace: upbound-system
  compositeTypeRef:
    apiVersion: platform.upbound.io/v1alpha1
    kind: XStorageBucket
  mode: Pipeline
  pipeline:
    - step: render-templates
      functionRef:
        name: function-go-templating
      input:
        apiVersion: gotemplate.fn.crossplane.io/v1beta1
        kind: GoTemplate
        source: Inline
        inline:
          template: |
            ---
            {{- define "ownerAndProjectLabels" }}
              labels:
                owner: {{ .observed.composite.resource.spec.parameters.owner }}
                service: {{ .observed.composite.resource.spec.parameters.service }}
            {{- end }}

            apiVersion: storage.gcp.upbound.io/v1beta1
            kind: Bucket
            metadata:
              name: {{ .observed.composite.resource.metadata.name }}
              annotations:
                gotemplating.fn.crossplane.io/composition-resource-name: {{ .observed.composite.resource.metadata.name }}
              {{ template "ownerAndProjectLabels" . }}
            spec:
              forProvider:
                location: {{ .observed.composite.resource.spec.parameters.location }}
                storageClass: {{ .observed.composite.resource.spec.parameters.storageClass }}
                {{- if .observed.composite.resource.spec.parameters.versioningEnabled }}
                versioning:
                  - enabled: true
                {{- end }}
              providerConfigRef:
                  name: {{ .observed.composite.resource.spec.parameters.environment }}
    - step: ready
      functionRef:
        name: function-auto-ready

More Crossplane Function Resources

Other Composition Improvements

Patching to an Object Key

A pain point for many users was trying to patch the key of a map object. This was often needed when attaching tags to a cloud object, most notably with AWS EKS network and load balancer integrations.

In 1.14 a new patch transform was added that allows conversion of JSON strings to Objects. In the example below, an AWS tag will be created of kubernetes.io/cluster/<cluster-name>: "true", with the cluster-name being a dynamic value based on spec.clusterName

    patches:
    - type: FromCompositeFieldPath
      fromFieldPath: spec.clusterName
      toFieldPath: spec.forProvider.tags
      transforms:
      - type: string
        string:
          type: Format
          fmt: '{"kubernetes.io/cluster/%s": "true"}'
      - type: convert
        convert:
          toType: object
          format: json

Fixing SHA Hash Calculations

A bug was found in that hash values were being calculated incorrectly on certain text inputs. This was addressed in PR 4445 and affects transforms such as SHA, SHA256 and SHA512.

The fix was backported to Crossplane versions 1.11 and 1.12 and will be included in the next point release for each.

DeploymentRuntimeConfig Replaces ControllerConfig

API Status: v1beta1 Enabled by default in 1.14.

ControllerConfig allows Crossplane operators to finely tune how Providers run on their clusters. When a Provider Package is installed, Crossplane creates a Kubernetes ServiceAccount and Deployment. Users who want to tune the behavior such as setting memory limits could create a ControllerConfig and then set up their Provider with a controllerConfigRef.

Unfortunately, users who created a ControllerConfig often got confused about deprecation warnings, as there has been long-time goal of Crossplane has been to replace ControllerConfig with an alternative that could support alternative runtimes and have a cleaner implementation.

These goals are reflected in the design of the Package Runtime Config and implemented in Pull Request 4744.

Starting in 1.14, Crossplane can be configured what to do when a Provider package is installed via the --package-runtime flag. There are currently two options:

  • Deployment, (default) which configures a Kubernetes deployment, and is manged by a DeploymentRuntimeConfig
  • External, install the Provider's CRDs and assume an external system will manage the Provider deployment.

When Deployment is selected, a new deploymentruntimeconfigs.pkg.crossplane.io Custom Resource Definition is installed in the cluster, and a default DeploymentRuntimeConfig is created. Unless configured otherwise, all Providers and Functions will use the default configuration.

$ kubectl get DeploymentRuntimeConfig
NAME                AGE
default             9h

Creating a DeploymentRuntimeConfig

To create a new DeploymentRuntimeConfig apply a manifest to the cluster. An example is below:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: func-env
spec:
  serviceAccountTemplate:
    metadata:
      name: dev-sa # this service account will be used in the deployment
      labels:
        foo: bar  # service account will be created with labels and annotations
  serviceTemplate: {}
  deploymentTemplate:
    spec:
      replicas: 2  # functions support replicas, providers should only run one pod
      selector: {}
      template:
        metadata:
          labels:
            some-pod-labels: cool-label
          annotations:
            some-pod-annotations: cool-annotation
        spec:
          containers:
            # The container used to run the Provider or Function must be named
            # 'package-runtime'. The package manager will overlay the package's
            # runtime image, pull policy, etc into this container.
            - # must be named package-runtime, the De
              resources:
                limits:
                  cpu: 2
                  memory: 2Gi
                requests:
                  cpu: 100m
                  memory: 128Mi
              volumeMounts:
                - name: shared-volume
                  mountPath: /shared
          volumes:
            - name: shared-volume
              emptyDir: {}

Both providers and v1beta1 Functions will support a runtimeConfigRef.

For example, to have a Function use the RuntimeDeploymentConfig above:

apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: function-patch-and-transform
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.1.4
  runtimeConfigRef:
    apiVersion: pkg.crossplane.io/v1beta1   # currently apiVersion and kind are optional
    kind: DeploymentRuntimeConfig
    name: func-env

Migrating From ControllerConfig to DeploymentRuntimeConfig

Unlike the custom fields of the ControllerConfig, the DeploymentRuntimeConfig embeds a Kubernetes deployment spec.

For example, in ControllerConfig any arguments passed to the Provider are in spec.args:

apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: aws-config
  labels:
    app: crossplane-provider-aws
spec:
  args:
    - -d
    - --enable-external-secret-stores

In DeploymentRuntimeConfig args are nested in the deploymentTemplate for the package-runtime container:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: enable-ess
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        spec:
          containers:
            - name: package-runtime
              args:
                - -d
                - --enable-external-secret-stores

Using the crossplane-migrator utility

To accelerate the migration process, I've created a utility crossplane-migrator to assist in creating DeploymentRuntimeConfigs from existing ControllerConfigs.

To write out a DeploymentRuntimeConfig file from a ControllerConfig manifest:

crossplane-migrator new-deployment-runtime  -i examples/enable-flags.yaml -o my-drconfig.yaml

Create a new DeploymentRuntimeConfig via Stdout

crossplane-migrator new-deployment-runtime -i cc.yaml | grep -v creationTimestamp | kubectl apply -f -

The migrator tool is in early development, so double-check the generated files.

Provider Improvements

Massive Performance Improvements in Upjet Providers

The Crossplane Special Interest Group #sig-upjet-provider-efficiency has been focused on the performance and CPU/Memory requirements for upjet-based Crossplane providers.

Upjet is a framework that can generate Crossplane providers from existing Terraform providers. While this allowed the creation of official providers with vast API coverage, a drawback of the initial upjet architecture was the use of the Terraform binary.

Using the Terraform Binary is resource-intensive (CPU and Memory) and had significant bottlenecks when provisioning thousands of resources.

Alongside the 1.14 release, the upjet engine has been updated to eliminate the Terraform process and connect to the drivers directly.

The previous terraform architecture vs. no-fork

Initial experiments in provisioning IAM roles show significant improvements across the board, with the ability to easily mange thousands of resources vis vastly lower CPU and Memory requirements. Testing was performed with thousands of managed resources. With the new architecture the cloud API began to become the limiting factor.

Metric Old Architecture New Architecture Improvement
Average Time To Ready State 19.55 min 1.012 sec 1159x speedup
Peak Time To Ready State 95.45 min 4.00 sec 1432x speedup
Average Memory Utilization 1.23 GiB 405 MiB 60% reduction
Peak Memory 1.79 GiB 421 MiB 77% reduction
Average CPU 95.77% 2.91% 92.86% difference
Peak CPU 98.17% 3.46% 94.71% difference

Testing the no-fork provider Release Candidates

The release candidate images for the AWS provider are version v0.44.0-rc.1. This version is API compatible to previous official providers.

The AWS provider is expected to have a production release later in November. Updated GCP and Azure providers are on the roadmap for release.

Granular Management Policies promoted to v1beta1

API Status: v1beta1 Enabled by default in 1.14.

Granular Management Policy was released as an alpha feature in 1.13. It allows platform engineers to fine tune what actions a Provider can perform on a Managed Resource, allowing Observe-Only resources and Deletion prevention.

For example, to prevent the Provider from deleting a resource if the Kubernetes Managed Resource Object is deleted, we'd remove the Delete policy from the resource:

spec:
  managementPolicies: ["Create", "Update", "Observe", "LateInitialize"]

Similarly, to Observe a resource without having Crossplane Manage it:

 managementPolicies: ["Observe"]

Graduation to v1beta1 is tracked via 4126.

v1alpha1 ManagementPolicies were implemented by Providers by setting the --enable-management-policies flag:

spec:
  args:
   - --enable-management-policies

With graduation to v1beta1, the following provider releases include ManagementPolicies turned on by default and do not require the feature flag:

Uptest

Uptest, the end-to-end testing framework was updated to v0.6.0, with a bugfix release at v0.6.1.

Most notably this release included import and resource update testing.

Developer Experience

Developer experience has been a concern of the Crossplane community, with users often struggling to develop compositions and debug Crossplane.

To address these issues the Crossplane Developer Experience Special Interest Group was formed in September 2023. Crossplane 1.14 contains a number of improvements from this effort. A few notable changes includes:

Crossplane CLI Improvements

kubectl crossplane has been renamed to crossplane

Previously, the Crossplane CLI has packaged as a plugin for the kubectl command. Starting in 1.14 it will be broken out into a command called crossplane that will be a single binary for developers and operators.

In addition, the CLI has been updated with several new features, including the ability to render compositions locally and display resources in a cluster.

To get the new crossplane binary, follow the installation instructions.

crossplane beta render

Render. A benefit of Composition Functions, users can now render Compositions locally and test functions on your machine. This promises to accelerate the Composition development cycle.

To render a composition, provide a Composite Resource, a Composition and the functions that will run in the composition (see examples).

crossplane beta render examples/xr.yaml examples/composition.yaml examples/functions.yaml

---
apiVersion: nopexample.org/v1
kind: XBucket
metadata:
  name: test-xrender
status:
  bucketRegion: us-east-2
---
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/composition-resource-name: my-bucket
  generateName: test-xrender-
  labels:
    crossplane.io/composite: test-xrender
  ownerReferences:
  - apiVersion: nopexample.org/v1
    blockOwnerDeletion: true
    controller: true
    kind: XBucket
    name: test-xrender
    uid: ""
spec:
  forProvider:
    region: us-east-2

crossplane beta trace

crossplane beta trace. Adds the ability to print out a visualization of a Composite resource into the terminal or DOT format for Graphviz.

$ crossplane beta trace xnetwork ref-aws-network       
NAME                                                 READY   SYNCED   STATUS      
XNetwork/ref-aws-network                             True    True     Available   
├─ VPC/ref-aws-network-9h7qf                         True    True     Available   
├─ InternetGateway/ref-aws-network-gnxfv             True    True     Available   
├─ Subnet/ref-aws-network-tlns5                      True    True     Available   
├─ Subnet/ref-aws-network-4zwtn                      True    True     Available   
├─ Subnet/ref-aws-network-q7k7g                      True    True     Available   
├─ Subnet/ref-aws-network-zqnc7                      True    True     Available   
├─ RouteTable/ref-aws-network-8vj5d                  True    True     Available   
├─ Route/ref-aws-network-8fjt5                       True    True     Available   
├─ MainRouteTableAssociation/ref-aws-network-8v7wd   True    True     Available   
├─ RouteTableAssociation/ref-aws-network-6kh4v       True    True     Available   
├─ RouteTableAssociation/ref-aws-network-kvqt8       True    True     Available   
├─ RouteTableAssociation/ref-aws-network-x66l8       True    True     Available   
├─ RouteTableAssociation/ref-aws-network-nl6wd       True    True     Available   
├─ SecurityGroup/ref-aws-network-lt6r4               True    True     Available   
├─ SecurityGroupRule/ref-aws-network-7x5dv           True    True     Available   
└─ SecurityGroupRule/ref-aws-network-848f5           True    True     Available

trace can also output DOT using --output=dot, showing a visual representation of Crossplane Resources.

crossplane beta trace ObjectStorage test-resource -n my-ns -o dot | dot -Tpng -o output.png

Generates an image like:

graphviz

Alpha/Beta commands

Alpha/Beta commands have been added to the CLI so that developers can add experimental features to the CLI.

In 1.14 the following beta commands are available. There are currently no alpha commands:

Commands:
  beta render       Render a composite resource (XR).
  beta trace        Trace a Crossplane resource to get a detailed output of its
                    relationships, helpful for troubleshooting.
  beta xpkg init    Initialize a new package from a template.

xpkg build

Move up xpkg build to Crossplane. Consolidate package build code to the crossplane CLI.

Build Packages from tar files

Build packages from tar files. Now you can use tools like podman to build Crossplane Provider packages:

podman build .
      --file ci/docker/Dockerfile
      --tag $IMAGE_NAME:latest
podman save -o local_image.tar $IMAGE_NAME:latest
./crossplane xpkg build
      --package-root package
      --embed-runtime-image-tarball local_image.tar
      --output my_provider.xpkg

crossplane-migrator

crossplane-migrator migrates YAML manifests to newer APIs. The following migrations are supported:

  • ControllerConfig to DeploymentRuntimeConfig
  • Compositions to use Function-based Patch&Transform instead of the built-in engine.

For example, to convert a Composition to run via function-patch-and-transform in a pipeline:

crossplane-migrator new-pipeline-composition -i composition.yaml -o composition-pipeline.yaml

crossplane xpkg beta init function

Add xpkg beta init function to create skeleton code for functions.

For example

crossplane beta xpkg init function-example function-template-go

Will create a function template based on the function-go-sdk in the current directory.

Tracking CLI improvements

There are a lot more improvements coming to the crossplane CLI including dry run support and an language server to interact with code editors like Visual Studio Code.

CLI Improvements can be tracked via the crossplane-cli label.

Issue 4648 tracks CLI improvements in 1.14.

Documentation Improvements

Crossplane Documentation has been greatly expanded to include more concepts and code examples.

Ordered Deletion via Usages

API Status: Alpha Enable via --enable-usages in Crossplane.

Kubernetes is an eventually-consistent system that leaves dependency management to the controllers that manage objects in the Cluster.

This affects Crossplane resources that depend on resources managed with a different provider. For example, if a Crossplane-provisioned Helm Release and Kubernetes Cluster are deleted at the same time, the Helm provider will be stuck as it is unable to communicate with the now-deleting Cluster. The release will remain in a stuck state, requiring manual cleanup.

The issue is well-documented in 3393.

In 1.14, a Usage type was introduced as a v1alpha1. Design, Implementation.

In the following Composition example, we define a Usage of a Cluster by a Helm Release:

resources:
    - name: cluster
      base:
        apiVersion: container.gcp.upbound.io/v1beta1
        kind: Cluster
        ...
    - name: release
      base:
        apiVersion: helm.crossplane.io/v1beta1
        kind: Release
        ...
  - name: release-uses-cluster
    base:
      apiVersion: apiextensions.crossplane.io/v1alpha1
      kind: Usage
      spec:
        of:
          apiVersion: container.gcp.upbound.io/v1beta1
          kind: Cluster
          resourceSelector:
            matchControllerRef: true
        by:
          apiVersion: helm.crossplane.io/v1beta1
          kind: Release
          resourceSelector:
            matchControllerRef: true

Crossplane Performance Improvements

Realtime Compositor

API Status: Alpha, enable via --enable-realtime-compositions in Crossplane.

In the past, Composite Resources (XRs) were updated on a polling basis, polling every 60 seconds by default (a setting that could be changed via the --poll-interval flag).

This meant changes in the Composition or in the Manged Resources could take up to a minute to take effect.

With the Realtime Compositor in 1.14, XRs are updated immediately if a new Composition Revision is installed or one of the Manged Resources in the XR is updated.

The Realtime Compositor was implemented in two parts: Part 1: CompositionRevisions and Part 2: Manged Resources.

A benefit of this approach is the Crossplane pod will not longer waste CPU cycles rendering unchanged XRs, allowing the ability to handle more Compositions in the same CPU/Memory footprint. This also reduces the number of calls made to Composition functions.

Early testing with a complex Composition reduced provisioning time from 4 minutes to around 2.

Fix Out of Memory issues on Clusters with thousands of Secrets

4626 addresses an issue found on clusters that host tens of thousands of Kubernetes secrets.

Operator Improvements

Squashing object has been modified warnings

A longstanding issue has often confused users with constant events being emitted indicating the object has been modified.

 Warning  ConfigureCompositeResource  19s (x2 over 20s)  offered/compositeresourcedefinition.apiextensions.crossplane.io  cannot patch object: Operation cannot be fulfilled on compositenetworks.gcp.platformref.crossplane.io "test-network-d6xjh": the object has been modified; please apply your changes to the latest version and try again`

This usually happened because there were attempted writes to an Object that had been updated by another part of Crossplane's code.

In Pull Request 4758 this issue is resolved by re-queueing the update if there is a conflict.

Adding readiness and liveness probes

In certain cases crossplane was reporting as ready before all the components were ready.

Pull Request 4748 adds these probes.

Adding extraObjects to the Crossplane Helm Chart

Pull Request 4664 adds an extraObjects: [] array in the Crossplane helm chart.

This enables users to install additional Kubernetes objects during Crossplane installations, such as ControllerConfig, Providers, or PodMonitors.

Log Events

Pull Request 4754 makes it easier to correlate log events.

Eliminate Duplicated Events in the RBAC Controller

Crossplane's RBAC Controller dynamically updates Kubernetes RBAC settings.

In 4307 suppresses duplicated events where no change is made to a Role or RoleBinding.

Added Ability to Pause Reconciliation of Provider/Configuration/Function Instances

The ability to pause Managed Resources via an the crossplane.io/paused annotation was released in Crossplane 1.10.

In 1.14, pausing via the crossplane.io/paused annotation has been extended to Core types:

  • Claims
  • Composites
  • Configurations, Functions, and Providers

Paused resources cannot be deleted until the pause annotation is removed.

A use case is pausing a Configuration during a maintenance event so that the Compositions installed by the Configuration can be updated manually.

Fixing Externally Managed Kubernetes Service Accounts

Crossplane Providers run with a Kubernetes service account. Crossplane will create these service accounts automatically, but users could override this setting and use existing service accounts. There were some gaps in the current implementation where multiple providers shared the same service account or where Crossplane would take ownership of a service account that it didn't create. PR 4925, backported to 1.14, fixes this issue.

Migration of Crossplane Helm Images to xpkg.upbound.io

Before 1.14, The Crossplane Helm release pointed to images in index.docker.io. These images were affected by rate-limiting. Crossplane's CI systems have been publishing images to both dockerhub and xpkg.upbound.io since January 2023.

In 1.14 the Helm charts will pull images from xpkg.upbound.io instead of dockerhub by default

  • 3568: publish images to xpkg.upbound.io
  • 4831: update Helm charts

TLS Certificate Generation

To support Webhooks and functions, Crossplane will generate TLS Certificates automatically.

Looking Forward to 1.15

Crossplane 1.14 reflects the culmination of months of listening to end users and addressing their most desired features. The next release of Crossplane scheduled for January 2024 is likely to have a stronger focus on operations and API maturity.

The Observability SIG (slack link) launched in October 2023 will influence the high-level metrics that Crossplane and Providers export.

Another likely area of focus is to finalize the design for pulling data into compositions, as seen in the issue Add Data from Arbitrary K8s Object to Composition Environment, as this has kept EnvironmentConfig at v1alpha1.

You can track the roadmap to 1.15 via the Crossplane project roadmap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment