-
-
Save bassam/c2a5a00134768e0201533ac0ee3a57d0 to your computer and use it in GitHub Desktop.
# A composite resource definition aggregates one or more child | |
# resources into a single higher level abstraction. The abstraction | |
# is defined by a CRD and can be consumed as a standard resource. Child | |
# resources can themselves by composite enabling a hierarchy. | |
apiVersion: core.crossplane.io/v1alpha1 | |
kind: CompositeResourceDefinition | |
metadata: | |
name: private-mysql-server | |
spec: | |
# Each CompositeResourceDefinition references exactly one CRD that | |
# represents the abstract resource definition. When an CR of this abstract | |
# CRD is created, the child resources in this definition are created. | |
# The abstractResourceDefinition can be cluster scoped or namespace scoped, | |
# see rules below. | |
abstractResourceDefinition: | |
apiVersion: database.example.org/v1alpha1 | |
kind: MySQLInstance | |
# The scope of *all* child resources in this composite resource definition. | |
# Child resources can not be of different scopes, either all namespace | |
# scoped or all cluster-scoped. | |
scope: Namespaced | |
# Supported scopes combinations | |
# | |
# Child Resources Scope = Namespaced, Abstract CRD Scope = Namespaced | |
# The abstract CR will have owner references to all its composed child resources. | |
# Child Resources Scope = Cluster, Abstract CRD Scope = Cluster | |
# The abstract CR will have owner references to all its composed child resources. | |
# Child Resources Scope = Cluster, Abstract CRD Scope = Namespaced | |
# The namespaced scoped abstract CR will have binding relationships to all | |
# its cluster-scoped resources. | |
# The array of composed child resources that will be dynamically | |
# provisioned to satisfy MySQLInstance claims that use this class. The base must | |
# be a valid resource. The patch specifies how the abstract resource's fields may be | |
# used to extend or override the base's fields. | |
resources: | |
- base: | |
apiVersion: azure.crossplane.io/v1alpha3 | |
kind: ResourceGroup | |
metadata: | |
spec: | |
location: West US | |
providerRef: | |
name: example | |
reclaimPolicy: Delete | |
patch: | |
- fromFieldPath: "spec.region" | |
toFiledPath: "spec.forProvider.location" | |
transforms: | |
- type: map | |
map: | |
us-west: "West US" | |
us-east: "East US" | |
- base: | |
apiVersion: database.azure.crossplane.io/v1beta1 | |
kind: MySQLServer | |
spec: | |
forProvider: | |
administratorLogin: myadmin | |
# A resource selector allows a resource dynamically provisioned using | |
# a particular composite resource class to reference other resources | |
# provisioned by the same class. In this case MySQLServer resources | |
# provisioned using this class will select the ResourceGroup | |
# provisioned by the same class and claim. | |
resourceGroupNameSelector: | |
matchComposite: true | |
location: West US | |
sslEnforcement: Disabled | |
version: "5.6" | |
sku: | |
tier: Basic | |
capacity: 1 | |
family: Gen5 | |
storageProfile: | |
storageMB: 20480 | |
writeConnectionSecretToRef: | |
namespace: crossplane-system | |
providerRef: | |
name: example | |
reclaimPolicy: Delete | |
patch: | |
- fromFieldPath: ".metadata.uid" | |
toFiledPath: "spec.writeConnectionSecretToRef.name" | |
- fromFieldPath: "spec.engineVersion" | |
toFiledPath: "spec.forProvider.version" | |
- fromFieldPath: "spec.storageGB" | |
toFiledPath: "spec.forProvider.storageMB" | |
transforms: | |
- type: math | |
math: | |
multiply: 1024 | |
- fromFieldPath: "spec.region" | |
toFiledPath: "spec.forProvider.location" | |
transforms: | |
- type: map | |
map: | |
us-west: "West US" | |
us-east: "East US" | |
- base: | |
apiVersion: database.azure.crossplane.io/v1alpha3 | |
kind: MySQLServerVirtualNetworkRule | |
spec: | |
name: my-cool-vnet-rule | |
serverNameSelector: | |
matchComposite: true | |
resourceGroupNameSelector: | |
matchComposite: true | |
properties: | |
virtualNetworkSubnetIdRef: | |
name: sample-subnet | |
reclaimPolicy: Delete | |
providerRef: | |
name: azure-provider | |
Here's a variant of this I've been playing with.
First, an example of something like today's "resource claim dynamic provisioning" pattern. In this example an application operator creates a namespaced resource (a MySQLInstance
), resulting in the creation of three cluster scoped "primitive" managed resources (an Azure MySQLServer
, MySQLServerVirtualNetworkRule
, and ResourceGroup
).
# A (Crossplane) ResourceDefinition defines a new kind of Crossplane resource. A
# controller watches for this kind and creates CustomResourceDefinitions. This
# type allows us to simplify the CRD authorship process, removing footguns (like
# not enabling the status subresource), and injecting schema to allow Crossplane
# machinery.
apiVersion: crossplane.io/v1alpha1
kind: ResourceDefinition
metadata:
name: mysqlinstances.database.example.org
spec:
# Features of this Crossplane resource. A feature flags this kind of resource
# as having support for a particular feature. This may involve injecting the
# schema required to support that feature into the resource's CRD.
#
# - This resource is a composite; it should have composition machinery.
# - This resource is connectable; it should have connection secret machinery.
# - This resource is an OAM workload; we should publish a WorkloadDefinition.
features:
- composite.crossplane.io
- connectable.crossplane.io
- workload.crossplane.io
# The CRD associated with this ResourceDefinition. The ResourceDefinition
# author may specify an existing CRD. If they do not a CRD will be created and
# set for them.
crdRef:
name: mysqlinstances.database.example.org
# This template is used to create a CRD (if crdRef is unset). Only a single
# version is supported.
crdSpecTemplate:
group: database.example.org
version: v1alpha1
names:
kind: MySQLInstance
listKind: MySQLInstanceList
plural: mysqlinstances
singular: mysqlinstance
scope: Namespaced
validation:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
properties:
engineVersion:
enum:
- "5.6"
- "5.7"
type: string
storageGb:
type: integer
region:
type: string
type: object
type: object
# An example of the resource defined by the above ResourceDefinition. This
# namespaced resource functions like one of today's resource claims.
apiVersion: database.example.org/v1alpha1
kind: MySQLInstance
metadata:
namespace: default
name: sql
spec:
# These fields were specified by the crdTemplateSpec.
engineVersion: "5.7"
storageGB: 10
region: us-west
# This field was injected by the connectable feature. It allows this resource
# to publish its connection details via a secret.
writeConnectionSecretToRef:
name: sql
# This field was injected by the composite feature. It allows compositions to
# be selected or specified like resource classes. This could also be used to
# implement something like the trait concept from OAM. In simple cases where
# there is only one compatible CompositionDefinition this can be omitted.
compositionSelector:
matchLabels:
example: "true"
# This field was injected by the composite feature. It allows a specific
# composition to be requested, and captures which composition was used after
# the fact.
compositionRef:
name: private-mysql-server
# This field was injected by the composite feature. It allows us to create a
# bidirectional relationship between a composite resource and the resources it
# composes. Owner references are insufficient for this purpose because they
# exist only from child to parent, making it very difficult to discover the
# children of a particular resource.
composed:
- ref:
apiVersion: azure.crossplane.io/v1alpha3
kind: ResourceGroup
name: default-sql-34jd2
- ref:
apiVersion: database.azure.crossplane.io/v1beta1
kind: MySQLServer
name: default-sql-3i3d1
- ref:
apiVersion: database.azure.crossplane.io/v1alpha3
kind: MySQLServerVirtualNetworkRule
name: default-sql-2mdus
# A CompositionDefinition defines what happens when an instance of a Crossplane
# resource is created (or updated?). It's a little like a contemporary resource
# class, but its inputs and outputs can be either cluster or namespace scoped
# resources.
apiVersion: crossplane.io/v1alpha1
kind: CompositionDefinition
metadata:
name: private-mysql-server
labels:
example: "true"
spec:
# Reference to a ResourceDefinition. The ResourceDefinition must support the
# composite feature.
definitionRef:
name: mysqlinstances.database.example.org
resources:
- base:
apiVersion: azure.crossplane.io/v1alpha3
kind: ResourceGroup
metadata:
spec:
location: West US
providerRef:
name: example
reclaimPolicy: Delete
patch:
- fromFieldPath: "spec.region"
toFieldPath: "spec.forProvider.location"
transforms:
- type: map
map:
us-west: "West US"
us-east: "East US"
- base:
apiVersion: database.azure.crossplane.io/v1beta1
kind: MySQLServer
spec:
forProvider:
administratorLogin: myadmin
resourceGroupNameSelector:
matchComposite: true
location: West US
sslEnforcement: Disabled
version: "5.6"
sku:
tier: Basic
capacity: 1
family: Gen5
storageProfile:
storageMB: 20480
writeConnectionSecretToRef:
namespace: crossplane-system
providerRef:
name: example
reclaimPolicy: Delete
patch:
- fromFieldPath: "metadata.uid"
toFieldPath: "spec.writeConnectionSecretToRef.name"
- fromFieldPath: "spec.engineVersion"
toFieldPath: "spec.forProvider.version"
- fromFieldPath: "spec.storageGB"
toFieldPath: "spec.forProvider.storageMB"
transforms:
- type: math
math:
multiply: 1024
- fromFieldPath: "spec.region"
toFieldPath: "spec.forProvider.location"
transforms:
- type: map
map:
us-west: "West US"
us-east: "East US"
# Specifies how to map connection details from this primitive resource to
# its composite resource. Only supported if corresponding ResourceDefinition
# supports the connectable feature.
connectionDetails:
- name: username
fromConnectionSecretKey: username
- name: password
fromConnectionSecretKey: password
- name: endpoint
fromConnectionSecretKey: endpoint
- base:
apiVersion: database.azure.crossplane.io/v1alpha3
kind: MySQLServerVirtualNetworkRule
spec:
name: my-cool-vnet-rule
serverNameSelector:
matchComposite: true
resourceGroupNameSelector:
matchComposite: true
properties:
virtualNetworkSubnetIdRef:
name: sample-subnet
reclaimPolicy: Delete
providerRef:
name: azure-provider
Now, an example of the contemporary "static provisioning" pattern. In this example an infrastructure operator defines and creates a new, cluster scoped, SecureDatabase
resource. When a SecureDatabase
is created we create three cluster scoped "primitive" managed resources (an Azure MySQLServer
, MySQLServerVirtualNetworkRule
, and ResourceGroup
). An application operator may then create a MySQLInstance
, explicitly specifying that they'd like to "compose" the aforementioned SecureDatabase
.
# A cluster scoped "SecureDatabase" composite managed resource.
apiVersion: crossplane.io/v1alpha1
kind: ResourceDefinition
metadata:
name: securedatabases.azure.example.org
spec:
features:
- composite.crossplane.io
- connectable.crossplane.io
crdRef:
name: securedatabases.azure.example.org
crdSpecTemplate:
group: azure.example.org
version: v1alpha1
names:
kind: SecureDatabase
listKind: SecureDatabaseList
plural: securedatabases
singular: securedatabase
scope: Cluster
validation:
openAPIV3Schema:
properties:
administratorLogin:
type: string
location:
type: string
sslEnforcement:
type: string
version:
type: string
sku:
properties:
tier:
type: string
capacity:
type: integer
family:
type: string
type: object
storageProfile:
properties:
storageMB:
type: integer
type: object
virtualNetworkSubnetIdRef:
properties:
name:
type: string
type: object
providerRef:
properties:
name:
type: string
type: object
type: object
# The namespaced MySQLInstance's schema is unchanged in the static provisioning
# scenario.
apiVersion: database.example.org/v1alpha1
kind: MySQLInstance
metadata:
namespace: default
name: sql
spec:
engineVersion: "5.7"
storageGB: 10
region: us-west
writeConnectionSecretToRef:
name: sql
# Instead of referencing a composition this MySQLInstance explicitly specifies
# that it composes an existing managed resource (which is itself a composite).
# This is allowed as long as the specified SecureDatabase has no compositeRef.
composed:
- ref:
apiVersion: azure.example.org/v1alpha1
kind: SecureDatabase
name: my-secure-database
# A cluster scoped composite resource.
apiVersion: azure.example.org/v1alpha1
kind: SecureDatabase
metadata:
name: sql
spec:
# The below spec fields are copied verbatim from the Azure MySQLServer and
# MySQLServerVirtualNetworkRule resources.
administratorLogin: myadmin
location: West US
sslEnforcement: Disabled
version: "5.6"
sku:
tier: Basic
capacity: 1
family: Gen5
storageProfile:
storageMB: 20480
virtualNetworkSubnetIdRef:
name: sample-subnet
providerRef:
name: example
# Schema for the remaining spec and status fields are all injected by
# ResourceDefinition features.
writeConnectionSecretToRef:
namespace: crossplane-system
name: sql
compositionSelector:
matchLabels:
example: "true"
compositionRef:
name: secure-mysql-server
# This cluster scoped resource composes other managed resources and is in turn
# composed by a MySQLInstance.
compositeRef:
apiVersion: database.example.org/v1alpha1
kind: MySQLInstance
namespace: default
name: sql
composed:
- ref:
apiVersion: azure.crossplane.io/v1alpha3
kind: ResourceGroup
name: default-sql-34jd2
- ref:
apiVersion: database.azure.crossplane.io/v1beta1
kind: MySQLServer
name: default-sql-3i3d1
- ref:
apiVersion: database.azure.crossplane.io/v1alpha3
kind: MySQLServerVirtualNetworkRule
name: default-sql-2mdus
apiVersion: crossplane.io/v1alpha1
kind: CompositionDefinition
metadata:
name: secure-mysql-server
spec:
definitionRef:
name: securedatabases.azure.example.org
resources:
- base:
apiVersion: azure.crossplane.io/v1alpha3
kind: ResourceGroup
metadata:
spec:
location: West US
providerRef:
name: example
reclaimPolicy: Delete
patch:
- fromFieldPath: "spec.providerRef.name"
toFieldPath: "spec.providerRef.name"
- fromFieldPath: "spec.location"
toFieldPath: "spec.forProvider.location"
- base:
apiVersion: database.azure.crossplane.io/v1beta1
kind: MySQLServer
spec:
forProvider:
administratorLogin: myadmin
resourceGroupNameSelector:
matchComposite: true
location: West US
sslEnforcement: Disabled
version: "5.6"
sku:
tier: Basic
capacity: 1
family: Gen5
storageProfile:
storageMB: 20480
writeConnectionSecretToRef:
namespace: crossplane-system
providerRef:
name: example
reclaimPolicy: Delete
patch:
- fromFieldPath: "spec.providerRef.name"
toFieldPath: "spec.providerRef.name"
- fromFieldPath: "metadata.uid"
toFieldPath: "spec.writeConnectionSecretToRef.name"
- fromFieldPath: "spec.version"
toFieldPath: "spec.forProvider.version"
- fromFieldPath: "spec.storageMB"
toFieldPath: "spec.forProvider.storageMB"
- fromFieldPath: "spec.location"
toFieldPath: "spec.forProvider.location"
# Specifies how to map connection details from this primitive resource to
# its composite resource. Only supported if corresponding ResourceDefinition
# supports the connectable feature.
connectionDetails:
- name: username
fromConnectionSecretKey: username
- name: password
fromConnectionSecretKey: password
- name: endpoint
fromConnectionSecretKey: endpoint
- base:
apiVersion: database.azure.crossplane.io/v1alpha3
kind: MySQLServerVirtualNetworkRule
spec:
name: my-cool-vnet-rule
serverNameSelector:
matchComposite: true
resourceGroupNameSelector:
matchComposite: true
properties:
virtualNetworkSubnetIdRef:
name: sample-subnet
reclaimPolicy: Delete
providerRef:
name: azure-provider
patch:
- fromFieldPath: "spec.providerRef.name"
toFieldPath: "spec.providerRef.name"
- fromFieldPath: "spec.virtualNetworkSubnetIdRef.name"
toFieldPath: "spec.properties.virtualNetworkSubnetIdRef.name"
Owner references are insufficient for this purpose because they exist only from child to parent, making it very difficult to discover the children of a particular resource.
A little more color on this. The two main detractors behind owner references are:
- Owner references are unidirectional from child to parent. It's very hard, given a parent resource, to discover its children if you do not know their kind in advance. Tools like https://github.com/ahmetb/kubectl-tree solve this by discovering and listing every single resource in a cluster in order to inspect all children and build a tree of references. This may be acceptable for a tool but does not seem like a scalable behaviour for a controller.
- Owner references leave the decision of whether to garbage collect children up to the entity that deletes the parent. In some cases this entity may be an application operator, and the infrastructure operator may wish to determine what happens to child resources when the parent is deleted.
As I mentioned in the meeting, I am not sure about the value of ResourceDefinition
(compared to writing CRD for claim and using a generic CompositionResource
as managed resource) because its upside about making it easier to write CRDs is not that big; in the end features array can be predicted by controller to some extent(claim vs managed), user still has to write the rest CRD stuff except for subresources
array. From the implementation side, we still work with bare unstructured.Unstructured
though it might make it easier to construct a new controller specific to that generated CRD.
In addition to that, I think one of the problems we'll face is making sure that different instances of CompositionDefinition
resources correctly specify the fields to be taken from the claim/managed resource in fromFieldPath
s and also the connection secret details stay same, i.e. the case where same resource is provisioned but connection secret looks different. In that case, the app author cannot be sure that the secret keys will stay same even though they used that specific resource, SecureDatabase
or MySQLInstance
.
I was thinking maybe we can expand ResourceDefinition
to include those responsibilities so that it acts as source of truth for both app devs and infra ops. For example, we can move all patch
and connectionDetails
blocks to ResourceDefinition
and CompositionDefinition
would only include resource parameters, just like how it is today with resource classes. In YAML language:
# Maybe ClaimDefinition(namespaced claims) and ResourceDefinition(cluster-scoped resource) would be separate so that we don't need features array.
apiVersion: crossplane.io/v1alpha1
kind: ResourceDefinition
metadata:
name: mysqlinstances.database.example.org
spec:
resources:
- identifier: resourcegroup
apiVersion: azure.crossplane.io/v1alpha3
kind: ResourceGroup
patch:
- fromFieldPath: "spec.providerRef.name"
toFieldPath: "spec.providerRef.name"
- fromFieldPath: "spec.location"
toFieldPath: "spec.forProvider.location"
- identifier: database
apiVersion: database.azure.crossplane.io/v1beta1
kind: MySQLServer
patch:
- fromFieldPath: "metadata.uid"
toFieldPath: "spec.writeConnectionSecretToRef.name"
- fromFieldPath: "spec.engineVersion"
toFieldPath: "spec.forProvider.version"
- fromFieldPath: "spec.storageGB"
toFieldPath: "spec.forProvider.storageMB"
transforms:
- type: math
math:
multiply: 1024
- fromFieldPath: "spec.region"
toFieldPath: "spec.forProvider.location"
transforms:
- type: map
map:
us-west: "West US"
us-east: "East US"
connectionDetails:
- name: username
fromConnectionSecretKey: username
- name: password
fromConnectionSecretKey: password
- name: endpoint
fromConnectionSecretKey: endpoint
- identifier: mysqlvnetrule
apiVersion: database.azure.crossplane.io/v1alpha3
kind: MySQLServerVirtualNetworkRule
features:
- composite.crossplane.io
- connectable.crossplane.io
- workload.crossplane.io
crdRef:
name: mysqlinstances.database.example.org
# This template is used to create a CRD (if crdRef is unset). Only a single
# version is supported.
crdSpecTemplate:
group: database.example.org
version: v1alpha1
names:
kind: MySQLInstance
listKind: MySQLInstanceList
plural: mysqlinstances
singular: mysqlinstance
scope: Namespaced
validation:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
properties:
engineVersion:
enum:
- "5.6"
- "5.7"
type: string
storageGb:
type: integer
region:
type: string
type: object
type: object
Then we can actually generate 2 CRDs from this ResourceDefinition
: MySQLInstance
(namespaced claim or cluster-scoped managed resource) and MySQLInstanceClass
(cluster-scoped resource class) which has the identifiers above as field names.
# A MySQLInstanceClass CRD is generated with field names given in `ResourceDefinition`.spec.resources[*].identifier
apiVersion: crossplane.io/v1alpha1
kind: MySQLInstanceClass
metadata:
name: private-mysql-server
labels:
example: "true"
spec:
resouregroup:
apiVersion: azure.crossplane.io/v1alpha3
kind: ResourceGroup
metadata:
spec:
location: West US
providerRef:
name: example
reclaimPolicy: Delete
database:
apiVersion: database.azure.crossplane.io/v1beta1
kind: MySQLServer
spec:
forProvider:
administratorLogin: myadmin
resourceGroupNameSelector:
matchCompositeIdentifier: resourcegroup # direct identifier
location: West US
sslEnforcement: Disabled
version: "5.6"
sku:
tier: Basic
capacity: 1
family: Gen5
storageProfile:
storageMB: 20480
writeConnectionSecretToRef:
namespace: crossplane-system
providerRef:
name: example
reclaimPolicy: Delete
mysqlvnetrule:
apiVersion: database.azure.crossplane.io/v1alpha3
kind: MySQLServerVirtualNetworkRule
spec:
name: my-cool-vnet-rule
serverNameSelector:
matchCompositeIdentifier: database # direct identifier
resourceGroupNameSelector:
matchCompositeIdentifier: resourcegroup # direct identifier
properties:
virtualNetworkSubnetIdRef:
name: sample-subnet
reclaimPolicy: Delete
providerRef:
name: azure-provider
End-user will create the following:
apiVersion: database.example.org/v1alpha1
kind: MySQLInstance
metadata:
namespace: default
name: sql
spec:
# These fields were specified by the crdTemplateSpec.
engineVersion: "5.7"
storageGB: 10
region: us-west
# This field was injected by the connectable feature. It allows this resource
# to publish its connection details via a secret.
writeConnectionSecretToRef:
name: sql
# Class selector like today.
classSelector:
matchLabels:
example: "true"
# Class ref like today.
classRef:
name: private-mysql-server
composed:
- ref:
apiVersion: azure.crossplane.io/v1alpha3
kind: ResourceGroup
name: default-sql-34jd2
- ref:
apiVersion: database.azure.crossplane.io/v1beta1
kind: MySQLServer
name: default-sql-3i3d1
- ref:
apiVersion: database.azure.crossplane.io/v1alpha3
kind: MySQLServerVirtualNetworkRule
name: default-sql-2mdus
In this scenario, when users write a resource class, they can be limited by the validation of the top-level fields under MySQLInstanceClass.spec
, i.e. you can only specify 1 resourcegroup, 1 vnetrule and 1 mysqlserver with the given identifiers. If we'd like to, we can even go ahead and fetch the CRD's validation objects with given apiVersion and kind from the api-server(from actual CRDs: MySQLServer
, ResourceGroup
and MySQLServerVirtualNetworkRule
) and embed them into the MySQLInstanceClass.spec
fields during CRD generation, which could result in fully validated composition class CRD.
The burden of writing bindings, transformations, fields and their types would be on the one who writes that one ResourceDefinition
. Resource class authorship doesn't get harder since it's just multiple known managed resources.
This will solve the problem where different composition classes point to fields that do not exist in the actual resource as well as the case where MySQLInstance
type resulting in completely different things that app dev didn't expect because resource classes in this scenario do not decide which kinds, bindings and everything independent from each other.
@bassam I like this structure with both of the additional comments. A few thoughts / questions:
CompositeResourceDefinition
? I don't see them as being very useful in this new world and they would likely clutter our user story.