Skip to content

Instantly share code, notes, and snippets.

@deads2k
Created March 27, 2024 18:11
Show Gist options
  • Save deads2k/94210d6f4a80c4a25bb5b7e32df59b02 to your computer and use it in GitHub Desktop.
Save deads2k/94210d6f4a80c4a25bb5b7e32df59b02 to your computer and use it in GitHub Desktop.
how do we ensure only a resource is managed by at most one operator install.
naivest approach: **This one is subpar**
1. in-process controller determines list of all resources to be created
2. check each resource in the controller to see if it is owned by something else
(probably a list across all namespaces of something (extensions?) )
3. if there is another owner, the contorller refuses to create the resource
When I install an app I get a namespaced extension resource
1. extension resource exists in the namespace I installed the app in
2. the helmchart may create items in many different namespaces: could it write to openshift-config-managed
3. every groupresource based on the list of resources that it templated out to.
If I really wanted to use hooks, this list would be incomplete
gaps
1. race between creation
2. assumes its the only owner
3. there is no external visibility
4. no other actor modifying these contents is aware they are separate owned
5. this is probably part of the reason why hooks aren't evaluated
6. locating all the things that an operator create requires listing every resource type in every namespace by label
Alternative option
what if when we installed a helm chart, we put a side-car container with a kube-apiserver proxy in it.
The proxy would add two labels indicating extensionNamespace and extensionName.
If we did this, then without modification a helm chart being installed this way could run hooks and all its resources would
get the labels we expect.
This is done by overriding env var KUBERNETES_SERVICE to localhost
If oldSelf has label["appID"] and newSelf has label["appID"] they need to be the same
If oldSelf does NOT hav1 label["appID"] and newSelf has label["appID"]: allow (adoption)
If oldSelf has label["appID"] and newSelf does NOT have label["appID"]: allow (orphan)
If oldSelf does NOT have label["appID"] and newSelf does NOT have label["appID"]: allow (not )
what is the shape of an appID
Given an appID can I tell that it is a ClusterExtension? Could also be a different label type?
If this answer to this is true, can I write a validating admission policy that prevents using non ClusterExtension for any cluster scoped resource.
Looks like we can write this rule.
We need to be sure that an Extension and a ClusterExtension both modifying a namespace resource, only have one owner of that resource.
supporting helm hooks
1. What if the hook tries to modify a CRD that is owned by a ClusterExtension?
2. what if the hook tries to modify a job that is owned by a different operator?
ClusterExtension
spec
WhatAmI - CertManager
Namespace - where stuff is run
ServiceAccount - in the namespace
Effectively this creates the-stuff-from-the-bundle in ns/.spec.namespace and runs the helm chart
1. in the case of helm, one could imagine: create pod and just actually run helm
this requires the localhost proxy intercept approach to place the appID label on any created resources
2. in the case of helm + kapp, kapp runs in THE OLM pod and gets SA token and then the controller in THE OLM pod uses that SA and is very careful with its clientConfig.
this could just place the appID label.
If we enforced appID labelling using sidecar,
1. first instance would create CRD, side-car would label CRD as appID=first -- should work
2. second instance would attempt to update CRD, side-car would label CRD as appID=second -- should fail
2. third instance would attempt to create CRD, already exists -- should fail
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment