Imagine if you could create, for people in your company, a platform that would provide them with the same experience they have when working with AWS, Google Cloud, Azure, or any other public Cloud provider. Imagine if there would be a service for everything they do.
Do you need a database that works exactly as we expect it to work in this company with all the security, backup, compliance, and other policies we have?
Well...
There is a service for that.
Do you need to run a backend application in Kubernetes? There is a service for that as well.
Do you need a Kubernetes cluster itself? There is yet another service for that.
All you have to do is define a simple manifest that contains only the things you care about and abstracts all the unavoidable complexity. From there on, all you have to do is submit that manifest, the desired state, to the control plane API and observe the actual state. Even better, you can forget about the API and just push it to Git.
Wouldn't that be awesome?
Wouldn't it be great if we could replicate the experience of using a public Cloud provider but made specifically for our needs? Wouldn't it be great if there were a clearly defined API and a clear separation between the tasks end-users need to perform and the tasks that are the responsibility of the platform itself?
TODO: Lower-third: Crossplane, https://crossplane.io, crossplane.png
If that sounds like something you might need, then Crossplane is just the project that will get you there. It enables us with capabilities of creating control planes based on the same principles public cloud providers have. It democratizes technology that was previously reserved mostly for big Cloud providers like AWS, Azure, and Google Cloud. It enables us to create internal developer platforms.
Let me give you a sneak peek into some of the capabilities we'll build throughout this tutorial.
But, before I do that, I need to give you an early warning so that you can decide whether this tutorial is for you or not. I will not use a typical teaching approach where there is a long introduction, followed by an even longer theory, followed by an explanation of each of the components, followed by a bunch of diagrams, and so on and so forth. This is a tutorial that reflects the way I learn. I learn by doing. I learn by having my hands on the keyboard at all times. I learn by making mistakes and fixing them. I learn by experimenting. That's the approach I'll take with this tutorial. I'll explain theory and concepts, and I'll show diagrams, but only through hands-on exercises. I'll explain what we did rather than do what I explained.
Does that make sense?
If it does, you're in the right place. Otherwise... Go away, while you still can.
With that warning out of the way, let's take a quick glimpse at some of the things we'll build throughout this tutorial. That should give you a good idea of what you can expect and let you decide whether Crossplane is something you want to learn and adopt.
I lied when I said that we'll take "a quick glimpse at some of the things we'll build throughout this tutorial". We will, but not right away. First, we need to create a control plane cluster with Crossplane, Compositions, Argo CD, and a few other tools that will be required to show you what we're building.
Now, to make things easy, instead of providing step-by-step setup instructions for this and all other chapters, I created scripts that will do that for us. You can run them and follow the instructions they present through questions, or you can choose to inspect the scripts if you prefer setting up everything manually.
However, we have a bit of a problem. To run those setup scripts as well as the instructions that follow in the hands-on parts of the tutorial, we'll need tools. We'll need quite a few CLIs like, for example, kubectl, crossplane, gum, gh, hyperscaler-specific CLIs, and so on and so forth.
One option would be for me to give you the instructions on how to install all the CLIs we'll need. That, however, might result in you spending considerable time reading those instructions and installing those CLIs. We'll do something else. We'll run everything in Nix. It allows us to create ad-hoc Shell environments with all the tools we might need. It's awesome and it will help us streamline this tutorial and avoid complications that might arise from using different operating systems.
TODO: Lower-third: Nix, https://nix.dev/install-nix, nix.png
TODO: Thumbnail: 0ulldVwZiKA
FIXME: Install Nix by following the instructions at https://nix.dev/install-nix
If you are not familiar with Nix, you might want to check out this video, or, simply install it.
TODO: Lower-third: Docker, https://docs.docker.com/engine/install, docker.png
FIXME: Install Docker by following the instructions at https://docs.docker.com/engine/install
Apart from Nix, we'll need to install one more thing. I don't think we should run Docker in Nix, so we'll need it on the host machine. You probably already have it. If you don't, please install it by following the install instructions.
TODO: Lower-third: GitHub CLI, https://github.com/cli/cli?tab=readme-ov-file#installation, github.png
FIXME: Install GitHub CLI by following the instructions at https://github.com/cli/cli?tab=readme-ov-file#installation
Finally, we'll need gh (GitHub CLI) to fork the repository with examples we'll use throughout this tutorial, including the shell.nix file that will bring in all the tools we'll need. Please install it if you do not have it already.
You can find additional information about GitHub CLI in this video.
Finally, each chapter has an associated Gist that contains all the commands we'll execute. You can use it to copy and paste commands instead of pausing videos in an attempt to read what's on the screen and type the same in your terminal. The Gist is in the description of the video.
That's it. Now we're ready to set up everything required for me to show you a glimpse of the future.
First, we'll fork the repo,...
TODO: Overlay: screen-01
gh repo fork vfarcic/crossplane-tutorial --clone --remote...then enter the crossplane-tutorial directory, ...
cd crossplane-tutorial...and select the fork as the default repository.
gh repo set-defaultNext, we are going to start a Nix Shell with all the tools we'll need in this chapter.
nix-shell --run $SHELLExecuting nix-shell with all the required packages might take a while since there are quite a few CLIs we'll need install. It takes longer the first time. The packages it's downloading are cached so subsequent executions will be almost instant. Be patient. It will be worth it. Once it's done, we'll get a Shell with everything we'll need.
As an alternative, you can skip running nix-shell but, in that case, you'll need to ensure that all the requirements are fulfilled. You'll see what those are in a moment when we execute the setup script.
TODO: Fast-forward to the end of the nix-shell execution.
Next, we'll make the setup script executable...
chmod +x setup/00-intro.sh...and execute it.
./setup/00-intro.shConfirm that you are ready to start unless you prefer to inspect the script and set up everything manually.
Next, double-check that you meet all the requirements. If you are running the script from inside a Nix Shell, as I recommended, you do meet all the requirements, except for Docker. Nix took care of that. So, if you do meet all the requirements, just say... Yes!
I'll fast-forward parts that are too boring to watch and only comment on key parts of what the script does.
The script starts by creating a kind cluster with NGINX Ingress. Once it's up and running, it installs Crossplane and a few Configurations. Don't worry if you don't know what they are. We'll dive into configurations and everything else related to Crossplane and the ecosystem later. Right now, we're just fast-forwarding through all that so that you can see some of the features we'll be learning throughout this tutorial.
Now, those configurations will, ultimately, create hundreds of custom resource definitions. That takes time so expect to wait for a while until all the providers are up and running.
Once all the providers are ready, we're presented with a choice of a hyperscaler.
I'll choose AWS for this chapter, and you can choose any of the "big three". To keep things interesting, and for me not to be labeled as biased, I'll pick a different one in each chapter so the outputs you'll see in the video might differ from those you'll see on your terminal. The logic for all the providers is the same so that should not pose a problem, as long as you're aware that what you see in this video might not be exactly the same as what you'll see in your terminal, unless you happen to pick the same hyperscaler which, as I mentioned, is AWS in this chapter.
I assumed that the downside of having a discrepancy between what you see in this video and what you'll experience by, potentially, choosing a different hyperscaler, is justified by you being able to pick a hyperscaler that best suits your needs.
Once the hyperscaler is selected, the script sets up everything required for Crossplane to work with it.
Finally, the script installed and configured Argo CD.
Throughout the execution of the script, a few environment variables were placed into the .env file. We'll need those, so let's source it.
source .envThat's it. Now we're ready to take a look at some of the features we might expect to learn through this tutorial.
Here's what the future looks like.
cat examples/$HYPERSCALER-intro.yamlThe output is as follows.
---
apiVersion: devopstoolkitseries.com/v1alpha1
kind: ClusterClaim
metadata:
name: cluster-01
spec:
id: cluster-01
compositionSelector:
matchLabels:
provider: aws
cluster: eks
parameters:
nodeSize: small
minNodeCount: 3
---
apiVersion: v1
kind: Secret
metadata:
name: silly-demo-db-password
data:
password: cG9zdGdyZXM=
---
apiVersion: devopstoolkitseries.com/v1alpha1
kind: SQLClaim
metadata:
name: silly-demo-db
spec:
id: silly-demo-db
compositionSelector:
matchLabels:
provider: aws
db: postgresql
parameters:
version: "13"
size: small
---
apiVersion: devopstoolkitseries.com/v1alpha1
kind: AppClaim
metadata:
name: silly-demo
spec:
id: silly-demo
compositionSelector:
matchLabels:
type: backend-db
location: remote
parameters:
namespace: production
image: c8n.io/vfarcic/silly-demo:1.4.52
port: 8080
host: silly-demo.acme.com
dbSecret:
name: silly-demo-db
namespace: a-team
kubernetesProviderConfigName: cluster-01
There are around fifty lines of YAML. That might look intimidating but, once you see what that YAML creates, you'll realize that it is infinitely simpler than if we tried to accomplish the same result any other way.
It starts with a ClusterClaim that will create a Kubernetes cluster in your favorite hyperscaler. Which hyperscaler it will be depends on the matchLabels. In my case, it will be an AWS EKS cluster. Then there are a few parameters that specify that the nodes of the cluster should be small and that the minimum number of nodes (minNodeCount) should be 3. I don't have to worry about exact node sizes. Crossplane will translate small to whatever that means in AWS.
Please note that if you're using Google Cloud, minNodeCount is set to 1 and not 3 because it will be a regional cluster running in three zones, and GKE will create 1 in each, so there will be 3 in total.
To make things more interesting, ClusterClaim will not only create a managed Kubernetes cluster, but also make it production-ready by setting up Cilium, installing Crossplane, creating a few Namespaces, and quite a few other things.
Further on we have a Secret that contains the encoded password for the database. We will improve on that later when we integrate Crossplane with solutions for managing secrets.
The third definition is SQLClaim which will create a managed PostgreSQL database in the hyperscaler of choice. But that's not all. Not only that a database server will be created, but a new user and a database will be created in that server as well.
Finally, there is AppClaim which will run a backend application in the yet-to-be-created cluster defined through ClusterClaim. That backend app will be automatically connected to the database defined through the SQLClaim.
In other words, those fifty or so lines of YAML will create a production-ready managed Kubernetes cluster, a managed PostgreSQL database, and a backend application. Moreover, those three will be interconnected. The application will be running in that cluster and will be connected to the database.
There are probably a few other things that will happen and we'll comment on them later.
Now, as you'll see soon, creating all that often requires experience in AWS, Google, Azure, Kubernetes, databases, and quite a few other technologies. But, as a user of Crossplane Compositions, we don't need to worry about any of that. Right now, we are consumers of a service created by someone else.
Now, you might expect me to execute a command that will make all that happen, but that's not what we'll do. Instead, we'll treat it as the desired state and push it to Git where it belongs.
So, we'll copy the manifest to the a-team directory in the repo which happens to be monitored by Argo CD,...
cp examples/$HYPERSCALER-intro.yaml a-team/intro.yaml...and add, commit, and push changes to the Git repo.
git add .
git commit -m "Intro"
git pushThat's it. That's the only action I'll do in this chapter. That's all it takes.
FIXME: Open http://argocd.127.0.0.1.nip.io in a browser.
Now, please open http://argocd.127.0.0.1.nip.io in a browser. Use admin as the username and admin123 as the password, and sign in.
There is only one application managed by Argo CD. That's the one that monitors the a-team directory. To see the "real" action, I'll enter into the application and... Lo and behold! Those fifty or so lines of YAML expanded into quite a few resources.
We can also observe what's going on by interacting directly with the control plane cluster through kubectl by retrieving clusterclaims,sqlclaims,appclaims.
FIXME: Create a YT short about crossplane trace.
kubectl --namespace a-team get clusterclaims,sqlclaims,appclaimsThe output is as follows (truncated for brevity).
NAME CLUSTERNAME CONTROLPLANE NODEPOOL SYNCED READY ...
.../cluster-01 cluster-01 ReconcileError ReconcileError True False ...
NAME SYNCED READY CONNECTION-SECRET AGE
.../silly-demo-db True False 7s
NAME HOST SYNCED READY CONNECTION-SECRET AGE
.../silly-demo silly-demo.acme.com True False 7s
We can see that the clusterclaim, sqlclaim, and appclaim, the resources we defined in YAML, are there. None of them are READY. It'll take a bit of time until the Kubernetes cluster, the database, the application, and everything in between are created and fully operational.
Now, those three resources "expanded" into quite a few lower-level resources which we can observe by retrieving all managed resources.
kubectl get managedThe output is as follows (truncated for brevity).
NAME READY SYNCED EXTERNAL-NAME AGE
internetgateway.../cluster-01 False True 18s
internetgateway.../silly-demo-db False 18s
NAME READY SYNCED EXTERNAL-NAME AGE
mainroutetableassociation.../cluster-01 False 18s
mainroutetableassociation.../silly-demo-db False 18s
NAME READY SYNCED EXTERNAL-NAME AGE
route.../cluster-01 False 18s
route.../silly-demo-db False 18s
NAME READY SYNCED EXTERNAL-NAME AGE
routetableassociation.../cluster-01-1a False 18s
routetableassociation.../cluster-01-1b False 18s
routetableassociation.../cluster-01-1c False 18s
routetableassociation.../silly-demo-db-1a False 18s
routetableassociation.../silly-demo-db-1b False 18s
routetableassociation.../silly-demo-db-1c False 18s
NAME READY SYNCED EXTERNAL-NAME AGE
routetable.../cluster-01 False True 18s
routetable.../silly-demo-db False 18s
...
That output shows AWS resources (and a few others). If you chose Google Cloud or Azure, your output will differ.
In my case, I got AWS internet gateways, route table associations, main route table associations, routes, route table associations, route tables, security group rules, security groups, subnets, VPCs, an addon, a cluster auth, a cluster, a node group, role policy attachments, roles, an RDS instance, and a subnet group. Those are the AWS resources that Crossplane is managing. Further on, there are some Helm releases and Kubernetes objects that will be created in the control plane and the new cluster. Finally, there is a database that will be created in the managed database server.
If you're using Google Cloud or Azure, the number of managed resources will be smaller. Nevertheless, no matter what you chose, it's clear that those fifty or so lines of YAML are very simple to define and understand when compared to everything they produce.
An easier way to explore resources managed by Crossplane is through the crossplane CLI. Among other features, it contains the trace command which is, at the time of this recording, a beta release. By the time you watch this, it might become GA so you might need to remove beta from the command that follows.
crossplane beta trace clusterclaim cluster-01 --namespace a-teamThe output is as follows (truncated for brevity).
NAME SYNCED READY STATUS
ClusterClaim/cluster-01 (a-team) True False Waiting: ...
└─ CompositeCluster/cluster-01-lkm96 True False Creating...
├─ InternetGateway/cluster-01 True True Available
├─ MainRouteTableAssociation/cluster-01 True True Available
├─ RouteTableAssociation/cluster-01-1a True False Creating
├─ RouteTableAssociation/cluster-01-1b False - ReconcileError...
├─ RouteTableAssociation/cluster-01-1c True False Creating
├─ RouteTable/cluster-01 True True Available
├─ Route/cluster-01 False - ReconcileError...
...
That output shows a tree-like structure of all the resources managed by the ClusterClaim resource (one of the resources we defined in YAML).
We can see that ClusterClaim claim created the CompositeCluster composite resource which, in turn, created a bunch of managed resources. We'll go through claims, Composite Resources, and Managed Resources later. What matters, for now, is that we can see what are all the resources that are managed by the ClusterClaim we defined earlier.
Another important note is that some of those resources are synced while others are not. Some are ready, and others are not. Unlike some other tools, resources managed by Crossplane are eventually consistent. Some cannot be synchronized because they miss some information from other resources and some are not ready because their preconditions were not met or they are in the process of being created.
As you'll see later, everything will be eventually consistent.
We can observe a similar output if we trace the SQLClaim, the second Crossplane resource we defined.
crossplane beta trace sqlclaim silly-demo-db --namespace a-teamThe output is as follows (truncated for brevity).
NAME SYNCED READY STATUS
SQLClaim/silly-demo-db (a-team) True False Waiting:...
└─ SQL/silly-demo-db-lp9xm True False Creating...
├─ VPC/silly-demo-db True True Available
├─ Subnet/silly-demo-db-a True False Creating
├─ Subnet/silly-demo-db-b True True Available
├─ Subnet/silly-demo-db-c True False Creating
├─ SubnetGroup/silly-demo-db False - ReconcileError...
├─ InternetGateway/silly-demo-db True True Available
├─ RouteTable/silly-demo-db True True Available
├─ Route/silly-demo-db False - ReconcileError...
├─ MainRouteTableAssociation/silly-demo-db False - ReconcileError...
├─ RouteTableAssociation/silly-demo-db-1a False - ReconcileError...
├─ RouteTableAssociation/silly-demo-db-1b False - ReconcileError...
├─ RouteTableAssociation/silly-demo-db-1c False - ReconcileError...
├─ SecurityGroup/silly-demo-db True True Available
├─ SecurityGroupRule/silly-demo-db False - ReconcileError...
├─ Instance/silly-demo-db False - ReconcileError...
├─ ProviderConfig/silly-demo-db - -
├─ Database/silly-demo-db False - ReconcileError...
└─ ProviderConfig/silly-demo-db-sql - -
The database server and everything else required to run the database are also not yet ready. It will be ready soon.
Finally, we can trace the application itself defined as the AppClaim.
crossplane beta trace appclaim silly-demo --namespace a-teamThe output is as follows (truncated for brevity).
NAME SYNCED READY STATUS
AppClaim/silly-demo (a-team) True False Waiting...
└─ App/silly-demo-9pfkj True False Creating...
├─ Object/silly-demo-deployment False - ReconcileError...
├─ Object/silly-demo-service False - ReconcileError...
├─ Object/silly-demo-ingress False - ReconcileError...
└─ Object/silly-demo-secret False - ReconcileError...
It's "normal" that the application is not yet running. It should run on the new cluster and until the cluster is created, there is no place for it to run. As with the other two, all I can say is that it will become eventually consistent.
Here's what you should do next.
Take a short break. Get a coffee.
Once you're back, we can trace the SQLClaim again.
crossplane beta trace sqlclaim silly-demo-db --namespace a-teamThe output is as follows.
NAME SYNCED READY STATUS
SQLClaim/silly-demo-db (a-team) True True Available
└─ SQL/silly-demo-db-lp9xm True True Available
├─ VPC/silly-demo-db True True Available
├─ Subnet/silly-demo-db-a True True Available
├─ Subnet/silly-demo-db-b True True Available
├─ Subnet/silly-demo-db-c True True Available
├─ SubnetGroup/silly-demo-db True True Available
├─ InternetGateway/silly-demo-db True True Available
├─ RouteTable/silly-demo-db True True Available
├─ Route/silly-demo-db True True Available
├─ MainRouteTableAssociation/silly-demo-db True True Available
├─ RouteTableAssociation/silly-demo-db-1a True True Available
├─ RouteTableAssociation/silly-demo-db-1b True True Available
├─ RouteTableAssociation/silly-demo-db-1c True True Available
├─ SecurityGroup/silly-demo-db True True Available
├─ SecurityGroupRule/silly-demo-db True True Available
├─ Instance/silly-demo-db True True Available
├─ ProviderConfig/silly-demo-db - -
├─ Database/silly-demo-db True True Available
├─ ProviderConfig/silly-demo-db-sql - -
└─ Object/silly-demo-db True True Available
This time, all the resources required to run PostgreSQL are Available. The managed database is up and running.
Let's see what's going on with the cluster.
crossplane beta trace clusterclaim cluster-01 --namespace a-teamThe output is as follows (truncated for brevity).
NAME SYNCED READY STATUS
ClusterClaim/cluster-01 (a-team) True True Available
└─ CompositeCluster/cluster-01-lkm96 True True Available
├─ InternetGateway/cluster-01 True True Available
├─ MainRouteTableAssociation/cluster-01 True True Available
├─ RouteTableAssociation/cluster-01-1a True True Available
├─ RouteTableAssociation/cluster-01-1b True True Available
├─ RouteTableAssociation/cluster-01-1c True True Available
├─ RouteTable/cluster-01 True True Available
├─ Route/cluster-01 True True Available
...
All the resources required to run a cluster are up and running as well. That probably means that Crossplane could apply resources related to the backend application. It should run inside that cluster and it should be connected to the database. Since both of those are fully operational now, the application should be running as well.
Let's check it out.
crossplane beta trace appclaim silly-demo --namespace a-teamThe output is as follows.
NAME SYNCED READY STATUS
AppClaim/silly-demo (a-team) True True Available
└─ App/silly-demo-9pfkj True True Available
├─ Object/silly-demo-deployment True True Available
├─ Object/silly-demo-service True True Available
├─ Object/silly-demo-ingress True True Available
└─ Object/silly-demo-secret True True Available
If you chose Azure, there is a bug in the CompositeResourceDefinition that results in the
silly-demo-secretresource not being created. Please ignore it until I fix the issue. It should not affect the rest of the tutorial.
Everything is up and running. Those three resources we defined at the beginning expanded into dozens of other resources. Some of them are resources in a hyperscaler while others are resources in the new Kubernetes cluster, with a few others sprinkled for good taste.
We are almost done. The last thing we'll do is feed my paranoia. I need to confirm that everything is indeed working as expected. We'll do that by entering into the newly created cluster and taking a peek to see whether the backend application is indeed running. To do that, we'll export the KUBECONFIG variable...
export KUBECONFIG=$PWD/kubeconfig.yaml...and retrieve the Kube config from the hyperscaler of choice.
# Execute only if using Google Cloud
gcloud container clusters get-credentials cluster-01 \
--region us-east1 --project $PROJECT_ID
# Execute only if using AWS
aws eks update-kubeconfig --region us-east-1 \
--name cluster-01 --kubeconfig $KUBECONFIG
# Execute only if using Azure
az aks get-credentials --resource-group cluster01 \
--name cluster-01 --file $KUBECONFIGThis was one of the cases when instructions differed depending on the hyperscaler you chose. You'll find the instructions for your choice in the Gist.
Now that the cluster config is set, we can check whether it indeed has three nodes I specified.
kubectl get nodesThe output is as follows.
NAME STATUS ROLES AGE VERSION
ip-10-0-0-241.ec2.internal Ready <none> 30s v1.27.7-eks-e71965b
ip-10-0-1-82.ec2.internal Ready <none> 26s v1.27.7-eks-e71965b
ip-10-0-2-216.ec2.internal Ready <none> 32s v1.27.7-eks-e71965b
The nodes are there.
How about the application? It should be running inside the production Namespace.
kubectl --namespace production get all,ingressesThe output is as follows.
NAME READY STATUS RESTARTS AGE
pod/silly-demo-998c464fc-8j9jj 1/1 Running 0 17m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/silly-demo ClusterIP 172.20.219.47 <none> 8080/TCP 17m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/silly-demo 1/1 1 1 17m
NAME DESIRED CURRENT READY AGE
replicaset.apps/silly-demo-998c464fc 1 1 1 17m
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/silly-demo <none> silly-demo.acme.com 80 17m
The application is indeed up and running and it is connected to the PostgreSQL database.
As I mentioned earlier, this was only a glimpse of some of the features Crossplane offers. There's much more to it and I did not want to prolong this section more than necessary.
The goal was not to teach you how to use Crossplane and all its nitty-gritty details. That's coming next. So far, the objective was for you to see some of the things it can do so that you can decide whether that's something you might be interested in. I hope that the answer is a resounding YES.
Crossplane is unique. It's special. It changes the way we manage resources of any kind. It enables us to create control planes. It enables us to do things that were, in the past, reserved for only a handful of companies.
So... What do you say? Shall we continue?
Each section starts with instructions on how to create everything needed for that section and ends with instructions on how to destroy everything we created. That way you can explore each section independently from others. You can take a break without having to run and pay for, all the resources we created. I wanted to make it easy, but also cheap. No one should spend more time and more money than necessary.
In that spirit, here are the instructions on how to destroy everything we created.
unset KUBECONFIG
chmod +x destroy/00-intro.sh
./destroy/00-intro.sh
exit