_observe-only-support-native.md

Adding Support for Management Policies (a.k.a Observe Only) in a Native Provider

(Re)generating a provider with Management Policies

Check out the provider repo, e.g., crossplane-contrib/provider-gcp, and go to the project directory on your local machine.

1. Generate with management policy and update crossplane-runtime dependency:

   # Consume the latest crossplane-tools:
   go get github.com/crossplane/crossplane-tools@master
   go mod tidy
   # Generate getters/setters for management policy
   make generate
   
   # Consume the latest crossplane-runtime:
   go get github.com/crossplane/crossplane-runtime@master
   go mod tidy

2. Introduce a feature flag for Management Policies.

   Add the feature flag definition into the internal/features/features.go file.

   diff --git a/internal/features/features.go b/internal/features/features.go
   index 9c6b1fc8..de261ca4 100644
   --- a/internal/features/features.go
   +++ b/internal/features/features.go
   @@ -12,4 +12,9 @@ const (
           // External Secret Stores. See the below design for more details.
           // https://github.com/crossplane/crossplane/blob/390ddd/design/design-doc-external-secret-stores.md
           EnableAlphaExternalSecretStores feature.Flag = "EnableAlphaExternalSecretStores"
   +
   +       // EnableAlphaManagementPolicies enables alpha support for
   +       // Management Policies. See the below design for more details.
   +       // https://github.com/crossplane/crossplane/pull/3531
   +       EnableAlphaManagementPolicies feature.Flag = "EnableAlphaManagementPolicies"
    )

   Add the actual flag in cmd/provider/main.go file.

   diff --git a/cmd/provider/main.go b/cmd/provider/main.go
   index 669b01f9..a60df983 100644
   --- a/cmd/provider/main.go
   +++ b/cmd/provider/main.go
   @@ -48,6 +48,7 @@ func main() {
   
                   namespace                  = app.Flag("namespace", "Namespace used to set as default scope in default secret store config.").Default("crossplane-system").Envar("POD_NAMESPACE").String()
                   enableExternalSecretStores = app.Flag("enable-external-secret-stores", "Enable support for ExternalSecretStores.").Default("false").Envar("ENABLE_EXTERNAL_SECRET_STORES").Bool()
   +               enableManagementPolicies   = app.Flag("enable-management-policies", "Enable support for Management Policies.").Default("false").Envar("ENABLE_MANAGEMENT_POLICIES").Bool()
           )
   
           kingpin.MustParse(app.Parse(os.Args[1:]))
   @@ -122,6 +123,11 @@ func main() {
                   })), "cannot create default store config")
           }
   
   +       if *enableManagementPolicies {
   +               o.Features.Enable(features.EnableAlphaManagementPolicies)
   +               log.Info("Alpha feature enabled", "flag", features.EnableAlphaManagementPolicies)
   +       }
   +
           kingpin.FatalIfError(controller.Setup(mgr, o), "Cannot setup AWS controllers")
           kingpin.FatalIfError(mgr.Start(ctrl.SetupSignalHandler()), "Cannot start controller manager")
    }

3. Update the schema of the resource such that status.atProvider is a superset of spec.forProvider.

Testing: Locally Running the Provider with Management Policies Enabled

1. Create a fresh Kubernetes cluster.

2. Apply all providers CRDs with kubectl apply -f package/crds.

3. Run the provider with --enable-management-policies.

   You can update the run target in the Makefile as below

   diff --git a/Makefile b/Makefile
   index d529a0d6..84411669 100644
   --- a/Makefile
   +++ b/Makefile
   @@ -111,7 +111,7 @@ submodules:
    run: go.build
           @$(INFO) Running Crossplane locally out-of-cluster . . .
           @# To see other arguments that can be provided, run the command with --help instead
   -       UPBOUND_CONTEXT="local" $(GO_OUT_DIR)/provider --debug
   +       UPBOUND_CONTEXT="local" $(GO_OUT_DIR)/provider --debug --enable-management-policies
   
    # NOTE(hasheddan): we ensure up is installed prior to running platform-specific
    # build steps in parallel to avoid encountering an installation race condition.

   and run with:

   make run

4. Create some resources in the provider's management console and try observing them by creating a managed resource with managementPolicy: ObserveOnly.

   For example:

   apiVersion: rds.aws.upbound.io/v1beta1
   kind: Instance
   metadata:
     name: an-existing-dbinstance
   spec:
     managementPolicy: ObserveOnly
     forProvider:
       region: us-west-1

   You should see the managed resource is ready & synced:

   NAME                              READY   SYNCED   EXTERNAL-NAME                     AGE
   an-existing-dbinstance            True    True     an-existing-dbinstance            3m

   and the status.atProvider is updated with the actual state of the resource:

   kubectl get instance.rds.aws.upbound.io an-existing-dbinstance -o yaml

Please note: You would need the terraform executable installed on your local machine.

observe-only-preview.md

Testing Observe Only with Provider AWS

Install the provider with Observe Only feature support using one of the two images depending on your arch (generated from this PR):

• turkenh/provider-aws:v0.31.0-rc0.18.gcd9a184a-amd64
• turkenh/provider-aws:v0.31.0-rc0.18.gcd9a184a-arm64

# Change this based on your arch, either arm64 or amd64
ARCH=arm64

echo "apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: controller-config
spec:
  args:
    - --enable-management-policies
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: turkenh/provider-aws:v0.31.0-rc0.18.gcd9a184a-${ARCH}
  controllerConfigRef:
    name: controller-config" | kubectl apply -f - 

Verify that provider is installed and healthy:

kubectl get providers.pkg.crossplane.io
NAME           INSTALLED   HEALTHY   PACKAGE                                               AGE
provider-aws   True        True      turkenh/provider-aws:v0.31.0-rc0.18.gcd9a184a-arm64   12m

Verify that management policies available in the schema of CRDs:

kubectl get crd clusters.eks.aws.upbound.io -o yaml | grep managementPolicy
              managementPolicy:
              rule: self.managementPolicy == 'ObserveOnly' || has(self.forProvider.vpcConfig)

Scenario 1: Observing some managed resources

After configuring your ProviderConfig as usual, create the following resources with ids/regions set for some existing resources:

A VPC

apiVersion: ec2.aws.upbound.io/v1beta1
kind: VPC
metadata:
  name: observed-vpc
  annotations:
    crossplane.io/external-name: vpc-0e21cb4864bef1928
spec:
  managementPolicy: ObserveOnly
  forProvider:
    region: us-west-1

kubectl get vpc.ec2
NAME           READY   SYNCED   EXTERNAL-NAME           AGE
observed-vpc   True    True     vpc-0e21cb4864bef1928   2m14s

An EC2 Instance

apiVersion: ec2.aws.upbound.io/v1beta1
kind: Instance
metadata:
  annotations:
    crossplane.io/external-name: i-0a695e2f3b4b24b04
  name: observed-instance
spec:
  managementPolicy: ObserveOnly
  forProvider:
    region: us-west-2

kubectl get instance.ec2
NAME              READY   SYNCED   EXTERNAL-NAME         AGE
sample-instance   True    True     i-0a695e2f3b4b24b04   96s

An EKS Cluster

apiVersion: eks.aws.upbound.io/v1beta1
kind: Cluster
metadata:
  annotations:
    crossplane.io/external-name: eks-cluster-argocd-7sz2t-nxp5n
  name: observed-eks-cluster
spec:
  managementPolicy: ObserveOnly
  forProvider:
    region: us-west-2

kubectl get cluster.eks
NAME                   READY   SYNCED   EXTERNAL-NAME                    AGE
observed-eks-cluster   True    True     eks-cluster-argocd-7sz2t-nxp5n   34s

Check status.atProvider reflects the whole observed state for the resource:

kubectl get cluster.eks observed-eks-cluster -o yaml

Scenario 2: CEL validations in action (requires k8s v1.25+)

We have created an Observe Only EKS cluster without any spec (except the region which is an identifier field). Try creating the same with default management policy (FullControl).

apiVersion: eks.aws.upbound.io/v1beta1
kind: Cluster
metadata:
  annotations:
  name: some-eks-cluster
spec:
  forProvider:
    region: us-west-2

kubectl apply -f eks-cluster.ymal

The Cluster "some-eks-cluster" is invalid: spec: Invalid value: "object": vpcConfig is a required parameter

Scenario 3: Building composition relying on Observe Only resources.

Example A: Refer to on an existing MR

You have VPCs which are managed my Terraform. You want the existing VPCs to be used in your compositions, however:

• You don't want to managed them with Crossplane (yet)
• You don't want to hardcode or distribute VPC IDs

As a platform engineer, you prepare the following MRs in you control plane and refer them in your compositions just like any other XP resource.

apiVersion: ec2.aws.upbound.io/v1beta1
kind: VPC
metadata:
  name: platform-vpc
  annotations:
    crossplane.io/external-name: vpc-03d4a84d912b57fef
spec:
  managementPolicy: ObserveOnly
  forProvider:
    region: us-west-2

apiVersion: ec2.aws.upbound.io/v1beta1
kind: VPC
metadata:
  name: internal-vpc
  annotations:
    crossplane.io/external-name: vpc-e704829f
spec:
  managementPolicy: ObserveOnly
  forProvider:
    region: us-west-2

Example B: Use some observed data from your Observe Only MR

Assume you need the OIDC issuer of an existing EKS cluster in your composition. You can observe the EKS cluster inside your composition and patch the value from the status of it.

kubectl get cluster.eks observed-eks-cluster -o yaml | grep -A 2 identity
    identity:
    - oidc:
      - issuer: https://oidc.eks.us-west-2.amazonaws.com/id/FC07E4BCB4A57B78A9E99C1639079CED

Scenario 4: Migrating to Crossplane

You want to start managing an existing database with Crossplane. However, you don't feel comfortable with directly importing it not to override any existing configuration. Follow the (new) import procedure here and give control to Crossplane after observing it as read-only first.

observe-only-support-in-upjet-based-providers.md

Adding Support for Management Policies in an Upjet Based Provider

(Re)generating a provider with Management Policies

1. Prepare your (upjet based) provider.

   Go to the repo of the provider, e.g. upbound/provider-aws, on your local.

   1. Generate with management policy and update crossplane-runtime dependency:

   # Consume the latest crossplane-tools:
   go get github.com/crossplane/crossplane-tools@master
   go mod tidy
   # Generate getters/setters for management policy
   make generate
   
   # Consume the latest crossplane-runtime:
   go get github.com/crossplane/crossplane-runtime@master
   go mod tidy

   2. Generate with the latest upjet and management policies:

      # Bump to the latest upjet
      go get github.com/upbound/upjet@main
      go mod tidy

      Enable management policies in the generator by adding config.WithFeaturesPackage option:

      diff --git a/config/provider.go b/config/provider.go
      index 964883670..1c06a53e2 100644
      --- a/config/provider.go
      +++ b/config/provider.go
      @@ -141,6 +141,7 @@ func GetProvider() *config.Provider {
                      config.WithReferenceInjectors([]config.ReferenceInjector{reference.NewInjector(modulePath)}),
                      config.WithSkipList(skipList),
                      config.WithBasePackages(BasePackages),
      +               config.WithFeaturesPackage("internal/features"),
                      config.WithDefaultResourceOptions(
                              GroupKindOverrides(),
                              KindOverrides(),

      Generate:

      make generate

   3. Introduce a feature flag for Management Policies.

      1. Add the feature flag definition into the internal/features/features.go file.

      diff --git a/internal/features/features.go b/internal/features/features.go
      index 9c6b1fc8..de261ca4 100644
      --- a/internal/features/features.go
      +++ b/internal/features/features.go
      @@ -12,4 +12,9 @@ const (
              // External Secret Stores. See the below design for more details.
              // https://github.com/crossplane/crossplane/blob/390ddd/design/design-doc-external-secret-stores.md
              EnableAlphaExternalSecretStores feature.Flag = "EnableAlphaExternalSecretStores"
      +
      +       // EnableAlphaManagementPolicies enables alpha support for
      +       // Management Policies. See the below design for more details.
      +       // https://github.com/crossplane/crossplane/pull/3531
      +       EnableAlphaManagementPolicies feature.Flag = "EnableAlphaManagementPolicies"
       )

      2. Add the actual flag in cmd/provider/main.go file.

      diff --git a/cmd/provider/main.go b/cmd/provider/main.go
      index 669b01f9..a60df983 100644
      --- a/cmd/provider/main.go
      +++ b/cmd/provider/main.go
      @@ -48,6 +48,7 @@ func main() {
      
                      namespace                  = app.Flag("namespace", "Namespace used to set as default scope in default secret store config.").Default("crossplane-system").Envar("POD_NAMESPACE").String()
                      enableExternalSecretStores = app.Flag("enable-external-secret-stores", "Enable support for ExternalSecretStores.").Default("false").Envar("ENABLE_EXTERNAL_SECRET_STORES").Bool()
      +               enableManagementPolicies   = app.Flag("enable-management-policies", "Enable support for Management Policies.").Default("false").Envar("ENABLE_MANAGEMENT_POLICIES").Bool()
              )
      
              kingpin.MustParse(app.Parse(os.Args[1:]))
      @@ -122,6 +123,11 @@ func main() {
                      })), "cannot create default store config")
              }
      
      +       if *enableManagementPolicies {
      +               o.Features.Enable(features.EnableAlphaManagementPolicies)
      +               log.Info("Alpha feature enabled", "flag", features.EnableAlphaManagementPolicies)
      +       }
      +
              kingpin.FatalIfError(controller.Setup(mgr, o), "Cannot setup AWS controllers")
              kingpin.FatalIfError(mgr.Start(ctrl.SetupSignalHandler()), "Cannot start controller manager")
       }

   4. Generate the provider with Upjet changes.

      make generate

Running the Provider with Management Policies Enabled

1. Create a fresh K8s cluster

2. Apply CRDs

3. Run the provider with --enable-management-policies

   You can update the run target in the Makefile as below

   diff --git a/Makefile b/Makefile
   index d529a0d6..84411669 100644
   --- a/Makefile
   +++ b/Makefile
   @@ -111,7 +111,7 @@ submodules:
    run: go.build
           @$(INFO) Running Crossplane locally out-of-cluster . . .
           @# To see other arguments that can be provided, run the command with --help instead
   -       UPBOUND_CONTEXT="local" $(GO_OUT_DIR)/provider --debug
   +       UPBOUND_CONTEXT="local" $(GO_OUT_DIR)/provider --debug --enable-management-policies
   
    # NOTE(hasheddan): we ensure up is installed prior to running platform-specific
    # build steps in parallel to avoid encountering an installation race condition.

   and run with

   make run

4. Create some resources in the AWS console and try observing them by creating a managed resource with managementPolicy: ObserveOnly

Please note: You would need the terraform executable installed on your local machine.

preparing-a-local-build.md

Preparing a local build

There are a couple of open PRs that need to be brought together to test the feature end to end with upjet generated providers:

Crossplane Tools:

• crossplane/crossplane-tools#51

Crossplane Runtime:

• crossplane/crossplane-runtime#384

Upjet:

• crossplane/upjet#164
• crossplane/upjet#166
• crossplane/upjet#168

(Re)generating a provider with Management Policies

1. On your local, prepare the Upjet repository to include all changes from the open PRs on top of latest main.

   Get the latest main and create a new branch from there:

   git fetch --all
   git reset --hard upstream/main
   git checkout -b o_o-test

   Bring the upjet changes together:

   # Find the commit ids from open upjet PRs and list them all below
   git cherry-pick <commit-id-1> <commit-id-2> <commit-id-3> ...

2. Prepare your (upjet generated) provider.

   Go to the repo of the provider, e.g. upbound/provider-aws, on your local.

   1. Get the latest main and create a new branch from there:

   git fetch --all
   git reset --hard upstream/main
   git checkout -b o_o-test

   2. Generate with management policy and update crossplane-runtime dependency:

   # Consume open PR in the crossplane-tools:
   go mod edit -replace=github.com/crossplane/crossplane-tools=github.com/turkenh/crossplane-tools@o_o
   go mod tidy
   # Generate getters/setters for management policy
   make generate
   
   # Consume open PR in the crossplane-runtime:
   go mod edit -replace=github.com/crossplane/crossplane-runtime=github.com/turkenh/crossplane-runtime@o_o
   go mod tidy

   3. Use local upjet as a dependency:

   # replace "../upjet" with the relative/absolute path for your local upjet repository
   go mod edit -replace=github.com/upbound/upjet=../upjet
   go mod tidy

   4. Introduce a feature flag for Management Policies.

      1. Add the feature flag definition into the internal/features/features.go file.

      diff --git a/internal/features/features.go b/internal/features/features.go
      index 9c6b1fc8..de261ca4 100644
      --- a/internal/features/features.go
      +++ b/internal/features/features.go
      @@ -12,4 +12,9 @@ const (
              // External Secret Stores. See the below design for more details.
              // https://github.com/crossplane/crossplane/blob/390ddd/design/design-doc-external-secret-stores.md
              EnableAlphaExternalSecretStores feature.Flag = "EnableAlphaExternalSecretStores"
      +
      +       // EnableAlphaManagementPolicies enables alpha support for
      +       // Management Policies. See the below design for more details.
      +       // https://github.com/crossplane/crossplane/pull/3531
      +       EnableAlphaManagementPolicies feature.Flag = "EnableAlphaManagementPolicies"
       )

      2. Add the actual flag in cmd/provider/main.go file.

      diff --git a/cmd/provider/main.go b/cmd/provider/main.go
      index 669b01f9..a60df983 100644
      --- a/cmd/provider/main.go
      +++ b/cmd/provider/main.go
      @@ -48,6 +48,7 @@ func main() {
      
                      namespace                  = app.Flag("namespace", "Namespace used to set as default scope in default secret store config.").Default("crossplane-system").Envar("POD_NAMESPACE").String()
                      enableExternalSecretStores = app.Flag("enable-external-secret-stores", "Enable support for ExternalSecretStores.").Default("false").Envar("ENABLE_EXTERNAL_SECRET_STORES").Bool()
      +               enableManagementPolicies   = app.Flag("enable-management-policies", "Enable support for Management Policies.").Default("false").Envar("ENABLE_MANAGEMENT_POLICIES").Bool()
              )
      
              kingpin.MustParse(app.Parse(os.Args[1:]))
      @@ -122,6 +123,11 @@ func main() {
                      })), "cannot create default store config")
              }
      
      +       if *enableManagementPolicies {
      +               o.Features.Enable(features.EnableAlphaManagementPolicies)
      +               log.Info("Alpha feature enabled", "flag", features.EnableAlphaManagementPolicies)
      +       }
      +
              kingpin.FatalIfError(controller.Setup(mgr, o), "Cannot setup AWS controllers")
              kingpin.FatalIfError(mgr.Start(ctrl.SetupSignalHandler()), "Cannot start controller manager")
       }

   5. Generate the provider with Upjet changes.

      make generate

Running the Provider with Management Policies Enabled

1. Create a fresh K8s cluster

2. Apply CRDs

3. Run the provider with --enable-management-policies

   You can update the run target in the Makefile as below

   diff --git a/Makefile b/Makefile
   index d529a0d6..84411669 100644
   --- a/Makefile
   +++ b/Makefile
   @@ -111,7 +111,7 @@ submodules:
    run: go.build
           @$(INFO) Running Crossplane locally out-of-cluster . . .
           @# To see other arguments that can be provided, run the command with --help instead
   -       UPBOUND_CONTEXT="local" $(GO_OUT_DIR)/provider --debug
   +       UPBOUND_CONTEXT="local" $(GO_OUT_DIR)/provider --debug --enable-management-policies
   
    # NOTE(hasheddan): we ensure up is installed prior to running platform-specific
    # build steps in parallel to avoid encountering an installation race condition.

   and run with

   make run

4. Create some resources in the AWS console and try observing them by creating a managed resource with managementPolicy: ObserveOnly

Please note: You would need the terraform executable installed on your local machine.