130-crossplane-k8s-helm.sh

# Source: https://gist.github.com/f0d35ef2260208b15ddd390007fdd552

######################################################################
# Production-Ready Kubernetes Clusters Using Crossplane Compositions #
# https://youtu.be/uMC2QQfMctg                                       #
######################################################################

# Referenced videos:
# - Crossplane - GitOps-based Infrastructure as Code through Kubernetes API: https://youtu.be/n8KjVmuHm7A
# - How To Shift Left Infrastructure Management Using Crossplane Composites: https://youtu.be/AtbS1u2j7po

#################
# Setup Cluster #
#################

# Watch https://youtu.be/BII6ZY2Rnlc if you are not familiar with GitHub CLI
gh repo fork vfarcic/devops-toolkit-crossplane \
    --clone

cd devops-toolkit-crossplane

# Using Rancher Desktop for the demo, but it can be any other Kubernetes cluster with Ingress

# If not using Rancher Desktop, replace `127.0.0.1` with the base host accessible through NGINX Ingress
export INGRESS_HOST=127.0.0.1

kubectl create namespace crossplane-system

kubectl create namespace a-team

#############
# Setup AWS #
#############

# Replace `[...]` with your access key ID`
export AWS_ACCESS_KEY_ID=[...]

# Replace `[...]` with your secret access key
export AWS_SECRET_ACCESS_KEY=[...]

echo "[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
" >aws-creds.conf

kubectl --namespace crossplane-system \
    create secret generic aws-creds \
    --from-file creds=./aws-creds.conf

####################
# Setup Crossplane #
####################

helm repo add crossplane-stable \
    https://charts.crossplane.io/stable

helm repo update

helm upgrade --install crossplane crossplane-stable/crossplane \
    --namespace crossplane-system --create-namespace --wait

kubectl apply \
    --filename crossplane-config/provider-kubernetes.yaml

kubectl apply \
    --filename crossplane-config/provider-helm.yaml

kubectl apply \
    --filename crossplane-config/config-k8s.yaml

kubectl get pkgrev

# Wait until all packages are healthy

kubectl apply \
    --filename crossplane-config/provider-config-aws-official.yaml

#############################
# Production-Ready Clusters #
#############################

cat examples/k8s/aws-eks-official.yaml

kubectl --namespace a-team apply \
    --filename examples/k8s/aws-eks-official.yaml

kubectl --namespace a-team get clusterclaims

kubectl get managed,releases,objects

cat packages/k8s/definition.yaml

cat packages/k8s/eks-official.yaml

kubectl get managed,releases,objects

kubectl --namespace a-team \
    get secret a-team-eks-cluster \
    --output jsonpath="{.data.kubeconfig}" \
    | base64 -d >kubeconfig.yaml

cat kubeconfig.yaml

export KUBECONFIG=$PWD/kubeconfig.yaml

kubectl get namespaces

cat examples/k8s/aws-eks.yaml

###########
# Destroy #
###########

unset KUBECONFIG

kubectl --namespace a-team delete \
    --filename examples/k8s/aws-eks-official.yaml

kubectl get managed

# Repeat the previous command until all the managed resources
#   are removed (ignore `object` and `release` resources`)

# Destroy or reset the management cluster

# Destroy the GitOps repo

I'm getting an error at line 68 with fresh microk8s:

devops-toolkit-crossplane]$ kubectl apply         --filename crossplane-config/provider-config-aws.yaml 
error: unable to recognize "crossplane-config/provider-config-aws.yaml": no matches for kind "ProviderConfig" in version "aws.crossplane.io/v1beta1"

No commands reported errors until then.

Any help appreciated.

Did you wait a bit after executing kubectl apply --filename crossplane-config/provider-aws.yaml. The crossplane-config/provider-config-aws.yaml depends on crossplane-config/provider-aws.yaml being up-and-running.

If you did wait for a while, please execute kubectl get pkgrev and paste the output.

I did wait, and kubectl get pkgrev finds no resources.

devops-toolkit-crossplane]$ kubectl get pkgrev
No resources found

Also, just to make sure,

devops-toolkit-crossplane]$ kubectl apply         --filename crossplane-config/provider-aws.yaml
provider.pkg.crossplane.io/crossplane-provider-aws unchanged

Thanks for replying. I'm on the crossplane slack if that's a better place to discuss.

Can you ping me on Slack (@vfarcic)? I found Jason but I'm not sure that's you.

Hi Victor!
Amazing effort and videos, really helpful for many of us, thank you!

I want to let you know that current master contains probably an issue or is it intentional? Let me explain...

After kubectl --namespace a-team delete --filename examples/k8s/aws-eks.yaml and after everything is properly removed from Crossplane and Amazon PoV, there is still existing created VPC. Removing it from AWS WebGUI doesn't make sense because k8s and desired state, removing from Crossplane seems tricky - I tried just to kubectl delete vpc.ec2.aws.crossplane.io/a-team-eks but it was re-created again. Ok, so I've tried to remove all 'spare' objects - in my case kubectl delete object.kubernetes.crossplane.io/a-team-eks-schemahero-ns object.kubernetes.crossplane.io/a-team-eks-aws-pc providerconfigusage.kubernetes.crossplane.io/8fc9a828-e5a0-4e0e-8d4a-fd5f48a7aaf1 vpc.ec2.aws.crossplane.io/a-team-eks providerconfigusage.aws.crossplane.io/c536c9e8-42f3-4abf-a143-2e9efe69447a - hm, call hangs but VPC still exists... nice, so I've deleted the team NS kubectl delete ns a-team without an issue however those 5 objects survived... so repeated previous deletion of 5 objects - even the call hangs again and (almost) all objects still exist, the VPC was deleted at least which was my intention (existing VPC is charged money). So now I don't have a-team NS however still have (from kubectl get crossplane):

NAME                                                                                AGE   CONFIG-NAME   RESOURCE-KIND   RESOURCE-NAME
providerconfigusage.kubernetes.crossplane.io/8fc9a828-e5a0-4e0e-8d4a-fd5f48a7aaf1   10h   a-team-eks    Object          a-team-eks-aws-pc
providerconfigusage.kubernetes.crossplane.io/b9094b7f-bde1-4007-bbed-4e6ac8aa7e77   10h   a-team-eks    Object          a-team-eks-schemahero-ns

NAME                                                       SYNCED   READY   AGE
object.kubernetes.crossplane.io/a-team-eks-aws-pc          False    False   14h
object.kubernetes.crossplane.io/a-team-eks-schemahero-ns   False    False   14h

The rest (not pasted above) are 'system' objects seem not related to the created ClusterClaim. I don't see that as a correct final state but can live with it anyway...

Any idea/advise?

Pavel

The problem is likely with the Load Balancer. When you install Ingress inside a cluster, it creates an external LB which, in turn, it tied to a VPC. As a result, the VPC cannot be deleted as long as that LB is there and, since it was not created by Crossplane, it is unaware of it. You'll experience the same behavior with other tools (e.g., Terraform).

For now, the workaround is to delete the LoadBalancer Service. Something like... kubectl --namespace ingress-nginx delete service a-team-eks-ingress-ingress-nginx-controller.

Hi Victor,
thank you for the hint with LBS, seems forking since then.

It seems there is an other issue (or quite recent change) regarding SecurityGroup definition here. In your case when you specify 'All protocols' and use fromPort and toPort, such an object has never be SYNC=true. The work-around for this might be remove fromPort,toPort and use just ipProtocol: "-1" and ipRanges eventualy. The same might be applied for ingress if necessary. By this way you are able to create a new SecurityGroup similarly like the default EKS SecGroup created by Amazon themself.

Not sure if it is intentional or a bug which should be reported - if an issue isn't reported in GitHub yet....

That's not intentional but, rather, an oversight on my part. Would you like to create a PR with the changes?

Sure, happy to contribute, you have helped me much through you excellent videos already! (PR)

Couple of notes:

1. The same SecurityGroup definition you are using in other demos/repositories as far as I can see
2. Don't forget to build configuration and then update reference here (and everywhere where you are referencing the same cfg.:-D)
3. Not sure if the same work-around can be applied to SecurityGroupRule (there you use similar egress too) - e.g. eks-official.yaml. I've never need to use Rule(s) yet, needs to be tested first

Anyway - this is some kind of orphan? I don't see it in a quite recent file eks-official.yaml but can be seen also in gke-official.yaml...

Thanks a ton for the PR.

1. That's what happens when you make a mistake and then replicate that same mistake in other places without reevaluating the code. Unfortunately, that's an explanation, not an excuse (there is none). I don't have any :(
2. Done :)
3. I can fix and test that one the next time I'm working on that Composition.

this is there only because some of the uses of that one required the key in the secret to being different (to be value). I think that was added for SchemaHero and, in the meantime, I figured out how to use it without that key. It can be removed but, to do that, I need to find all the Gists that use it and modify them first. That's the type of issue I'm experiencing all the time. I create a lot of public Gists, demos, slides, books, etc. and I'm doing my best to keep them always in the working state. The downside is that any change that might not be backward compatible requires going back to all those Gists, demos, slides, etc.

Hi
I am getting below error even though my provider is showing in true and healthy state below is my output.I followed the same process to create a cluster.
kubectl get pkgrev
NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE
configurationrevision.pkg.crossplane.io/crossplane-k8s-61efd4c23f27 True 1 xpkg.upbound.io/devops-toolkit/dot-kubernetes:v0.9.23 Active 2 2 10m

NAME HEALTHY REVISION IMAGE STATE DEP-FOUND DEP-INSTALLED AGE
providerrevision.pkg.crossplane.io/crossplane-provider-aws-a2e16ca2fc1a True 1 crossplane/provider-aws:v0.29.0 Active 14m
providerrevision.pkg.crossplane.io/crossplane-provider-helm-59fc94cd939e True 1 xpkg.upbound.io/crossplane-contrib/provider-helm:v0.13.0 Active 11m
providerrevision.pkg.crossplane.io/crossplane-provider-kubernetes-05560a2c7b16 True 1 xpkg.upbound.io/crossplane-contrib/provider-kubernetes:v0.6.0 Active 10m

kubectl get managed,object,release
Warning: Please use v1beta1 version of SNS group.
NAME READY SYNCED ID VPC CIDR AGE
subnet.ec2.aws.crossplane.io/a-team-eks-1a False 10.0.0.0/24 2m40s
subnet.ec2.aws.crossplane.io/a-team-eks-1b False 10.0.1.0/24 2m40s
subnet.ec2.aws.crossplane.io/a-team-eks-1c False 10.0.2.0/24 2m40s

NAME READY SYNCED ID VPC AGE
internetgateway.ec2.aws.crossplane.io/a-team-eks False 2m40s

NAME READY SYNCED ID VPC AGE
routetable.ec2.aws.crossplane.io/a-team-eks False 2m40s

NAME READY SYNCED ID VPC AGE
securitygroup.ec2.aws.crossplane.io/a-team-eks False 2m40s

NAME READY SYNCED ID CIDR IPV6CIDR AGE
vpc.ec2.aws.crossplane.io/a-team-eks False False 10.0.0.0/16 2m40s

NAME READY SYNCED AGE
cluster.eks.aws.crossplane.io/a-team-eks False 2m40s

NAME READY SYNCED CLUSTER AGE
nodegroup.eks.aws.crossplane.io/a-team-eks False 2m40s

NAME CHART VERSION SYNCED READY STATE REVISION DESCRIPTION AGE
release.helm.crossplane.io/a-team-eks-crossplane crossplane 1.9.0 False 2m40s
release.helm.crossplane.io/a-team-eks-ingress ingress-nginx 4.2.0 False 2m40s

NAME READY SYNCED ROLENAME POLICYARN AGE
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-cni False a-team-eks-nodegroup arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy 2m40s
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-controlplane False a-team-eks-controlplane arn:aws:iam::aws:policy/AmazonEKSClusterPolicy 2m40s
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-registry False a-team-eks-nodegroup arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly 2m40s
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-service False a-team-eks-controlplane arn:aws:iam::aws:policy/AmazonEKSServicePolicy 2m40s
rolepolicyattachment.iam.aws.crossplane.io/a-team-eks-worker False a-team-eks-nodegroup arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy 2m40s

NAME READY SYNCED AGE
role.iam.aws.crossplane.io/a-team-eks-controlplane False 2m40s
role.iam.aws.crossplane.io/a-team-eks-nodegroup False 2m40s

NAME KIND PROVIDERCONFIG SYNCED READY AGE
object.kubernetes.crossplane.io/a-team-eks-aws-p Provider a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-aws-pc ProviderConfig a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-config-app Configuration a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-config-monitoring Configuration a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-config-sql Configuration a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-helm-provider Provider a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-k8s-provider-cc ControllerConfig a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-k8s-provider-crb ClusterRoleBinding a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-k8s-provider-sa ServiceAccount a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-kubernetes-provider Provider a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-ns-dev Namespace a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-ns-prod Namespace a-team-eks False 2m40s
object.kubernetes.crossplane.io/a-team-eks-schemahero-cr ClusterRole a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-crb ClusterRoleBinding a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-crd-db CustomResourceDefinition a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-crd-migration CustomResourceDefinition a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-crd-table CustomResourceDefinition a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-ns Namespace a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-secret Secret a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-service Service a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-schemahero-sts StatefulSet a-team-eks False 2m39s
object.kubernetes.crossplane.io/a-team-eks-sql-provider Provider a-team-eks False 2m39s

Given that all the resources (except providers and configs) are False, I'm guessing that there's something wrong with the authentication to AWS. Can you describe one of the resources and post it here? Some of them are probably False because they depend on others that were not created so describing one of those that can be created independently of others. So, let's see what are the events of, let's say, the VPC. Something like kubectl describe vpc.ec2.aws.crossplane.io/a-team-eks should do.

create failed: failed to create the VPC resource: api error UnauthorizedOperation: You are not authorized to perform this operation getting this error even though I have assume the role which has access to create resources

I'm guessing that the creds you used starting from https://gist.github.com/vfarcic/f0d35ef2260208b15ddd390007fdd552#file-130-crossplane-k8s-helm-sh-L35 are invalid or do not have the permissions to create, in this case, the VPC. Can you confirm that the access key ID put into the env. var. AWS_ACCESS_KEY_ID and the secret access key stored in AWS_SECRET_ACCESS_KEY have the permissions and that the commands after that line did create the secret with those keys?

Yes my access_key and secret_key has the required permission to create or delete the resources. I am using cross-account in aws . Do i need to specify the specific role it requires to create resources

I'm not sure what the exact cause of the issue is, but I am fairly certain that those keys alone do not give Crossplane sufficient permissions to create AWS resources. Check the ProviderConfig spec (you can find it in https://marketplace.upbound.io) and see whether something is missing.

Hi @vfarcic
Any idea about how my kube config should look like for my k8s provider? Do you have a sample so that the k8s provider can access remote eks cluster?
crossplane-contrib/provider-kubernetes#121

@ashishvaishno You need to install Kubernetes provider and create a secret with KubeConfig pointing to the cluster you want to operate through Crossplane. With those two up and running, you need to create ProviderConfig that points to that secret. I don't have an example of that since I tend to create those clusters with Crossplane Compositions which, among other things, do all that for me. Here's one example of such a composition:
https://github.com/vfarcic/crossplane-kubernetes/blob/main/package/civo-port.yaml

In that example, I already have Kubernetes provider installed in the control plane cluster. From there on, https://github.com/vfarcic/crossplane-kubernetes/blob/main/package/civo-port.yaml#L80 is the part that configures the provider to use the secret previously generated by that same composition.

Please let me know if you have difficulties replicating that and I'll do my best to create a better example.

@vfarcic. Thanks for your instant reply 😄 In my case both the eks clusters were created without using crossplane. i was looking at your tutorial https://www.youtube.com/watch?v=uMC2QQfMctg&t=349s, where your kube config has a token for the user. As you also mentioned in the video that the token is short lived, how can I create a kubeconfig in such case if the remote cluster is not managed by crossplane?

@ashishvaishno If the cluster is not managed by Crossplane, you'd need to fetch the config without Crossplane (e.g., with aws CLI). From there on, you either give it to others or, if it's for Crossplane usage, create a secret with it. You might also be able to fetch the config with Observe Only Resources (https://github.com/crossplane/crossplane/blob/master/design/design-doc-observe-only-resources.md) but I'm not sure whether that would work (I haven't tried it with Kube config myself).

I'd probably start with OOR first and, if that does not work, fall back to aws.

the aws cli doesnt seems to be present in the kubernetes provider pod 😢

2023-05-31T11:13:06.648Z	DEBUG	provider-kubernetes	Cannot connect to provider	{"controller": "managed/object.kubernetes.crossplane.io", "request": "/xxxxxx-qbx82-wx525", "uid": "a26bad78-97ee-4af4-bc53-7c4cc33b965a", "version": "5789603", "external-name": "xxxxxx-qbx82-wx525", "error": "cannot create new Kubernetes client: cannot create Kubernetes client: Get \"https://xxxxxx.eks.amazonaws.com/api?timeout=32s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins", "errorVerbose": "Get \"https://xxxxxx.eks.amazonaws.com/api?timeout=32s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins\ncannot create Kubernetes client\ngithub.com/crossplane-contrib/provider-kubernetes/internal/clients.NewKubeClient\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/clients/client.go:38\ngithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object.(*connector).Connect\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object/object.go:174\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594\ncannot create new Kubernetes client\ngithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object.(*connector).Connect\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object/object.go:176\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594"}

let me try the observe-only-resource 😄

@vfarcic I installed the latest version of crossplane, but it doesnt have the alpha feature flag --enable-alpha-management-policies 😕

k exec -it -n crossplane-system crossplane-5697555f88-qtl9h -c crossplane -- crossplane --version
v1.12.1
k exec -it -n crossplane-system crossplane-5697555f88-qtl9h -c crossplane -- crossplane core start --help
Usage: crossplane core start

Start Crossplane controllers.

Flags:
  -h, --help                               Show context-sensitive help.
  -d, --debug                              Print verbose logging statements.
  -v, --version                            Print version and quit.

      --profile=host:port                  Serve runtime profiling data via HTTP at /debug/pprof.
  -n, --namespace="crossplane-system"      Namespace used to unpack and run packages ($POD_NAMESPACE).
      --service-account="crossplane"       Name of the Crossplane Service Account ($POD_SERVICE_ACCOUNT).
  -c, --cache-dir="/cache"                 Directory used for caching package images ($CACHE_DIR).
  -l, --leader-election                    Use leader election for the controller manager ($LEADER_ELECTION).
  -r, --registry="index.docker.io"         Default registry used to fetch packages when not specified in tag ($REGISTRY).
      --ca-bundle-path=STRING              Additional CA bundle to use when fetching packages from registry ($CA_BUNDLE_PATH).
      --webhook-tls-secret-name=STRING     The name of the TLS Secret that will be used by the webhook servers of core Crossplane and providers ($WEBHOOK_TLS_SECRET_NAME).
      --webhook-tls-cert-dir=STRING        The directory of TLS certificate that will be used by the webhook server of core Crossplane. There should be tls.crt and tls.key files ($WEBHOOK_TLS_CERT_DIR).
      --user-agent="crossplane/v1.12.1"    The User-Agent header that will be set on all package requests ($USER_AGENT).
  -s, --sync-interval=1h                   How often all resources will be double-checked for drift from the desired state.
      --poll-interval=1m                   How often individual resources will be checked for drift from the desired state.
      --max-reconcile-rate=10              The global maximum rate per second at which resources may checked for drift from the desired state.
      --esstls-secret-name=STRING          The name of the TLS Secret that will be used by Crossplane and providers as clients of External Secret Store plugins ($ESS_TLS_SECRET_NAME).
      --esstls-certs-dir=STRING            The path of the folder which will store TLS certificates to be used by Crossplane and providers for communicating with External Secret Store plugins ($ESS_TLS_CERTS_DIR).

Alpha Features:
  --enable-environment-configs                      Enable support for EnvironmentConfigs.
  --enable-external-secret-stores                   Enable support for External Secret Stores.
  --enable-composition-functions                    Enable support for Composition Functions.
  --enable-composition-webhook-schema-validation    Enable support for Composition validation using schemas.

k describe po -n crossplane-system crossplane-5697555f88-qtl9h | grep -i image
    Image:         crossplane/crossplane:v1.12.1

@ashishvaishno I probably misunderstood your scenario so let me say what I understood and you correct me.

You have Kubernetes clusters created somehow (NOT with Crossplane) and now you'd like to use Crossplane to manage resources in those clusters. To do that, you need to configure the Crossplane Kubernetes provider with authentication to all those clusters and for that, you need Kube config stored as a secret inside the cluster where Crossplane is running (if those were created with Crossplane you would already have those secrets). So, you need to fetch Kube configs whichever way you would normally fetch them and create (yourself) Kubernetes secrets with those configs. From there on, Crossplane will know how to interact with those clusters and do whatever you'd want it to do.

Now, if I'm right in those assumptions (and I probably am not), all you have to do is 1) fetch Kube configs from your laptop or any other machine that has aws CLI, 2) create secrets with those configs, and 3) create Kubernetes provider configs that point to those secrets. Now, bear in mind that you should not store manifests of those secrets in Git. That would be unsafe. Either you create secrets and delete the manifests from your laptop (or wherever you created them from) or you use something like External Secrets Operator (ESO) or Sealed Secrets.

@vfarcic Below is the rough interpretation of my architecture

• I have provisioned a new eks cluster(with terraform) and installed crossplane on it. I have also installed the k8s provider there.
• I already had an old eks cluster(also provisioned with terraform) with different application works loads (cross-account-eks-cluster)
• For my POC, I would like to use crossplane to provision new resources on the cross-account-eks-cluster
• My config looks like this

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xxxxx (ca of cross-account-eks-cluster)
    server: xxxxxx (url from cross-account-eks-cluster)
  name: cross-account-eks-cluster
contexts:
- context:
    cluster: cross-account-eks-cluster
    user: cross-account-eks-cluster
  name: cross-account-eks-cluster
current-context: cross-account-eks-cluster
kind: Config
preferences: {}
users:
- name: cross-account-eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
        - --region
        - xxxxx
        - eks
        - get-token
        - --cluster-name
        - cross-account-eks-cluster

• I pass this as a base 64 encoded secret to my kubernetes ProviderConfig, (I am not going to store it in git 😉)

---
apiVersion: v1
kind: Secret
metadata:
  name: cluster-config
  namespace: crossplane-system
data:
  kubeconfig-dev: xxxxxxx
---
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
  name: kubernetes-provider-config-dev
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: cluster-config
      key: kubeconfig-dev

• Now while trying to create a namespace on the cross-account-eks-cluster I get the below error from the k8s-provider-pod as aws-cli is not installed in it 😢

2023-05-31T11:13:06.648Z	DEBUG	provider-kubernetes	Cannot connect to provider	{"controller": "managed/object.kubernetes.crossplane.io", "request": "/xxxxxx-qbx82-wx525", "uid": "a26bad78-97ee-4af4-bc53-7c4cc33b965a", "version": "5789603", "external-name": "xxxxxx-qbx82-wx525", "error": "cannot create new Kubernetes client: cannot create Kubernetes client: Get \"https://xxxxxx.eks.amazonaws.com/api?timeout=32s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins", "errorVerbose": "Get \"https://xxxxxx.eks.amazonaws.com/api?timeout=32s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins\ncannot create Kubernetes client\ngithub.com/crossplane-contrib/provider-kubernetes/internal/clients.NewKubeClient\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/clients/client.go:38\ngithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object.(*connector).Connect\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object/object.go:174\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594\ncannot create new Kubernetes client\ngithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object.(*connector).Connect\n\tgithub.com/crossplane-contrib/provider-kubernetes/internal/controller/object/object.go:176\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:213\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/reconciler/managed/reconciler.go:761\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/[email protected]/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nruntime.goexit\n\truntime/asm_amd64.s:1594"}

• I also tried the observe only feature, but I dont find that feature flag while installing crossplane https://gist.github.com/vfarcic/f0d35ef2260208b15ddd390007fdd552?permalink_comment_id=4592230#gistcomment-4592230

@ashishvaishno You cannot use Kube config that requires aws to authenticate. Unfortunately, I don't remember how I created it in the past but I do remember that there is such an option.

@vfarcic I also tried to create a sa and a kubernetes.io/service-account-token, but this also didnt work 😢
I cannot create client certificate as well, cos EKS currently does not support Kubelet client auth
Not sure what else to try 😞

I'm not sure how to do it. I remember that, in the past, I was generating EKS Kube config that does not require aws CLI but I don't remember anymore how I did it and since I moved to Crossplane, I've been using it to generate other clusters so I haven't faced that issue in a long while.

Unfortunately, I don't have time to try it out myself today and then I'll be traveling for 4-5 weeks and probably won't be in front of a computer (except for work). If Google-ing does not work, try creating a small EKS cluster with Crossplane. That will result in a Secret with Kube config. You might be able to reconstruct it from there.

I followed the instructions and clusterclaims is: Synced: true but Ready: false
seeing this:
Warning SelectComposition 47s (x10 over 4m48s) defined/compositeresourcedefinition.apiextensions.crossplane.io cannot select Composition: no compatible Compositions found

Nothing is being deployed, as i would expect, like vpc, subnets, etc and then cluster

@jecausey I made quite a few changes (improvements) in that repo since I created this Gist and one of those broke the demo. I just updated the Gist. It should work now.

@vfarcic Thank you!!! Totally working now!