Online OpenShift Assisted Installer w/Calico as Default CNI

README.md

OpenShift Assisted Installer w/Calico as Default CNI

Red Hat quietly released a new method for installing bare metal OpenShift clusters via a tool called Assisted-Installer (or Assisted-Service). What makes this service really unique is that it reduces the infrastructure requirements for deploying bare metal clusters (i.e. PXE/UEFI, web servers, etc). Reduction of these legacy bare metal requirements opens up some interesting opportunities for carrier-based deployments such as RAN, MEC, uCPE, Content Delivery, and many other Edge or FE types of solutions.

Of course, there's a straightforward user-interface which can be leveraged, but I think what is most interesting about the Assisted-Service centers around customizations through it's REST API. This is in fact what I'm going to cover in this blog.

So how does this all work for bare metal without UEFI, DHCP and other traditional bootstrapping methods, you might ask? The Assisted-Installer is used to produce a LiveISO which includes a call-back URL for the REST API of the Assisted-Service. Once the bare metal server is online (either through DHCP or a Static IP address), it will make a secure API call to the Assisted-Service API. Once the deployment has been configured for the endpoint, the administrator can begin the installation process from the Assisted-Installer.

All that's really required to start your evaluation is to have a Red Hat account. So let's begin the demonstration with a customized deployment of Calico CNI. We'll go even further and enable Calico's eBPF dataplane, just for fun.

Required Links

Assisted-Service Red Hat Cloud
Red Hat Token

Part I - Create a Cluster in Assisted-Service

In order to use the Assisted-Service API via cloud.redhat.com, you will need to create a bearer token. Red Hat provides a user-level "OpenShift Cluster Manager API Token" which can be used to provide a bearer token.

1. Once you have created an OpenShift Cluster Manager API Token, use the "Copy to clipboard" function and provide it as the variable OFFLINE_ACCESS_TOKEN.

OFFLINE_ACCESS_TOKEN="<PASTE_TOKEN_HERE>"

export TOKEN=$(curl \
--silent \
--data-urlencode "grant_type=refresh_token" \
--data-urlencode "client_id=cloud-services" \
--data-urlencode "refresh_token=${OFFLINE_ACCESS_TOKEN}" \
https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token | \
jq -r .access_token)

IMPORTANT: This token will expire often (5 minutes). So if you recieve a 400 - Token is expired, reissue the command above.

VERIFY: You can check to see if the TOKEN variable is set via the following conditional:

if [ -z ${TOKEN+x} ]; 
  then echo "Token is undefined. Please check formatting before continuing\!"; else echo "Token is ready\!"; 
fi

2. Review the following contents, and modify them as needed. These variables will be used throughout the rest of the demonstration and have been test to work:

ASSISTED_SERVICE_API="api.openshift.com"
CLUSTER_VERSION="4.8"
CLUSTER_IMAGE="quay.io/openshift-release-dev/ocp-release:4.8.0-fc.3-x86_64"
CLUSTER_NAME="calico-poc"
CLUSTER_DOMAIN="jinkit.com"
CLUSTER_NET_TYPE="Calico"
CLUSTER_CIDR_NET="10.128.0.0/14"
CLUSTER_CIDR_SVC="172.30.0.0/16"
CLUSTER_HOST_NET="192.168.3.0/24"
CLUSTER_HOST_PFX="23"
CLUSTER_WORKER_HT="Enabled"
CLUSTER_WORKER_COUNT="0"
CLUSTER_MASTER_HT="Enabled"
CLUSTER_MASTER_COUNT="0"
CLUSTER_SSHKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDE1F7Fz3MGgOzst9h/2+5/pbeqCfFFhLfaS0Iu4Bhsr7RenaTdzVpbT+9WpSrrjdxDK9P3KProPwY2njgItOEgfJO6MnRLE9dQDzOUIQ8caIH7olzxy60dblonP5A82EuVUnZ0IGmAWSzUWsKef793tWjlRxl27eS1Bn8zbiI+m91Q8ypkLYSB9MMxQehupfzNzJpjVfA5dncZ2S7C8TFIPFtwBe9ITEb+w2phWvAE0SRjU3rLXwCOWHT+7NRwkFfhK/moalPGDIyMjATPOJrtKKQtzSdyHeh9WyKOjJu8tXiM/4jFpOYmg/aMJeGrO/9fdxPe+zPismC/FaLuv0OACgJ5b13tIfwD02OfB2J4+qXtTz2geJVirxzkoo/6cKtblcN/JjrYjwhfXR/dTehY59srgmQ5V1hzbUx1e4lMs+yZ78Xrf2QO+7BikKJsy4CDHqvRdcLlpRq1pe3R9oODRdoFZhkKWywFCpi52ioR4CVbc/tCewzMzNSKZ/3P0OItBi5IA5ex23dEVO/Mz1uyPrjgVx/U2N8J6yo9OOzX/Gftv/e3RKwGIUPpqZpzIUH/NOdeTtpoSIaL5t8Ki8d3eZuiLZJY5gan7tKUWDAL0JvJK+EEzs1YziBh91Dx1Yit0YeD+ztq/jOl0S8d0G3Q9BhwklILT6PuBI2nAEOS0Q=='

VERIFY: Verify that you can correctly talk with the API with the following curl command (install jq before running this command):

curl -s -X GET "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters" \
  -H "accept: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  | jq -r

If you are able to verify/return a response via the command above, you can continue to the next steps and deploy a cluster with modifications!

3. Next, download your pull_secret from the following URL: https://cloud.redhat.com/openshift/install/pull-secret

HINT: All of these instructions are intended to be run from the current directory. Considering this, make sure that the pull-secret.txt and installation are where you want them to be before continuing.

4. Next, create a variable with the raw contents of your pull-secret.txt file. This is important, because escape characters should be included as part of this output.

PULL_SECRET=$(cat pull-secret.txt | jq -R .)

5. Now create an Assisted-Service deployment .json file:

cat << EOF > ./deployment.json
{
  "kind": "Cluster",
  "name": "$CLUSTER_NAME",
  "openshift_version": "$CLUSTER_VERSION",
  "ocp_release_image": "$CLUSTER_IMAGE",
  "base_dns_domain": "$CLUSTER_DOMAIN",
  "hyperthreading": "all",
  "cluster_network_cidr": "$CLUSTER_CIDR_NET",
  "cluster_network_host_prefix": $CLUSTER_HOST_PFX,
  "service_network_cidr": "$CLUSTER_CIDR_SVC",
  "user_managed_networking": true,
  "vip_dhcp_allocation": false,
  "host_networks": "$CLUSTER_HOST_NET",
  "hosts": [],
  "ssh_public_key": "$CLUSTER_SSHKEY",
  "pull_secret": $PULL_SECRET
}
EOF

HINT: If you recieve an error that you cannot overwrite the file, then make sure to run setopt clobber in your shell.

HINT: There's a new option in OpenShift v4.8.x that allows you to create a single node OpenStack cluster (referred to as SNO). If this is what you want, then you will need to add the following lines to the ./deployment.json file:

  "high_availability_mode": "None",

As an example, a SNO deployment will look like this:

cat << EOF > ./deployment.json
{
  "kind": "Cluster",
  "name": "$CLUSTER_NAME",
  "openshift_version": "$CLUSTER_VERSION",
  "ocp_release_image": "$CLUSTER_IMAGE",
  "base_dns_domain": "$CLUSTER_DOMAIN",
  "hyperthreading": "all",
  "cluster_network_cidr": "$CLUSTER_CIDR_NET",
  "cluster_network_host_prefix": $CLUSTER_HOST_PFX,
  "service_network_cidr": "$CLUSTER_CIDR_SVC",
  "user_managed_networking": true,
  "vip_dhcp_allocation": false,
  "high_availability_mode": "None",
  "host_networks": "$CLUSTER_HOST_NET",
  "hosts": [],
  "ssh_public_key": "$CLUSTER_SSHKEY",
  "pull_secret": $PULL_SECRET
}
EOF

6. Create the cluster via the Assisted-Service API:

curl -s -X POST "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters" \
  -d @./deployment.json \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  | jq '.id'

7. IMPORTANT: that this will generate a CLUSTER_ID, which will need to be exported for future use. Export this variable from the output of the previous command in Step 5:

curl -s -X POST "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters" \
  -d @./deployment.json \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  | jq '.id'
"0da7cf59-a9fd-4310-a7bc-97fd95442ca1"

CLUSTER_ID="0da7cf59-a9fd-4310-a7bc-97fd95442ca1"

8. Now update the cluster install-config via the Assisted-Service API:

curl \
  --header "Content-Type: application/json" \
  --request PATCH \
  --data '"{\"networking\":{\"networkType\":\"Calico\"}}"' \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/install-config"

VERIFY: You can review your changes by issuing the following curl request:

curl -s -X GET \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/install-config" \
  | jq -r

Part II - Generate and Download the Assisted-Service ISO

1. Create another json file, this time called iso-params.json which will be used to generate the deployment ISO:

cat << EOF > ./iso-params.json
{
  "ssh_public_key": "$CLUSTER_SSHKEY",
  "pull_secret": $PULL_SECRET
}
EOF

2. Now use the following command to POST a request for Assisted-Service to build the deployment ISO:

curl -s -X POST "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/image" \
  -d @iso-params.json \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  | jq '.'

3. Use curl to download the ISO just generated. This ISO will be used to build the OpenShift cluster, as with any other Assisted-Service deployment:

curl \
  -H "Authorization: Bearer $TOKEN" \
  -L "http://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/image" \
  -o ai-liveiso-$CLUSTER_ID.iso

4. Lastly, boot the bare metal instance from this ISO just downloaded.

Part III - Prepare and Deploy the Calico Manifests

1. Create a manifests folder, and download the manifests for Calico. Be sure to check Calico's documentation for any potential changes:

mkdir manifests
curl https://docs.projectcalico.org/manifests/ocp/crds/01-crd-installation.yaml -o manifests/01-crd-installation.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/01-crd-imageset.yaml -o manifests/01-crd-imageset.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/01-crd-tigerastatus.yaml -o manifests/01-crd-tigerastatus.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_bgpconfigurations.yaml -o manifests/crd.projectcalico.org_bgpconfigurations.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_bgppeers.yaml -o manifests/crd.projectcalico.org_bgppeers.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_blockaffinities.yaml -o manifests/crd.projectcalico.org_blockaffinities.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_clusterinformations.yaml -o manifests/crd.projectcalico.org_clusterinformations.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_felixconfigurations.yaml -o manifests/crd.projectcalico.org_felixconfigurations.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_globalnetworkpolicies.yaml -o manifests/crd.projectcalico.org_globalnetworkpolicies.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_globalnetworksets.yaml -o manifests/crd.projectcalico.org_globalnetworksets.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_hostendpoints.yaml -o manifests/crd.projectcalico.org_hostendpoints.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_ipamblocks.yaml -o manifests/crd.projectcalico.org_ipamblocks.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_ipamconfigs.yaml -o manifests/crd.projectcalico.org_ipamconfigs.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_ipamhandles.yaml -o manifests/crd.projectcalico.org_ipamhandles.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_ippools.yaml -o manifests/crd.projectcalico.org_ippools.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_kubecontrollersconfigurations.yaml -o manifests/crd.projectcalico.org_kubecontrollersconfigurations.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_networkpolicies.yaml -o manifests/crd.projectcalico.org_networkpolicies.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_networksets.yaml -o manifests/crd.projectcalico.org_networksets.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/00-namespace-tigera-operator.yaml -o manifests/00-namespace-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-rolebinding-tigera-operator.yaml -o manifests/02-rolebinding-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-role-tigera-operator.yaml -o manifests/02-role-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-serviceaccount-tigera-operator.yaml -o manifests/02-serviceaccount-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-configmap-calico-resources.yaml -o manifests/02-configmap-calico-resources.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-tigera-operator.yaml -o manifests/02-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/01-cr-installation.yaml -o manifests/01-cr-installation.yaml

2. Next, we need to base64 encode each of the Calico manifests and upload each of them to the Assisted-Installer in order to be included as part of the deployment manifests directory. Before starting, verify if any manifests have been previously uploaded for this cluster. If either nothing or brackets are returned, continue (this is what we want):

curl -s -X GET \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/manifests"

3. Set variables for the location of the Calico directory. This will be used in the next command, where we'll base64 encode each of the yaml docs:

MANIFESTS=(manifests/*.yaml)

4. Run the following BASH/ZSH loop to POST each base64 encoded manifest to the Assisted-Service API automatically:

total=${#MANIFESTS[@]}
i=0
for file in "${MANIFESTS[@]}"; do
    i=$(( i + 1 ))
    eval "CALICO_B64_MANIFEST=$(cat $file | base64 -w 0)";
    eval "BASEFILE=$(basename $file)";
    printf "Processing file: $file \n"
    printf "Basename of file: $BASEFILE \n"
    curl  \
    --header "Content-Type: application/json"  \
    --request POST  \
    -H "Authorization: Bearer $TOKEN" \
    --data "{\"file_name\":\"$BASEFILE\", \"folder\":\"manifests\", \"content\":\"$CALICO_B64_MANIFEST\"}"  \
    "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/manifests"
done
printf "Total Manifests: $total \n"

VERIFY: You can review your changes by issuing the following curl request:

curl -s -X GET \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/manifests"

8. IMPORTANT: At this point, you MUST boot each of your servers via the LiveISO downloaded from "Part II, Step 3" and verify that each have correctly checked into the Assisted-Installer UI.
   DO NOT change any settings, but you can safely click "Next" until you see the list of staged hosts.

9. Now it is time to update the cluster with the host-level subnet:

curl \
  --header "Content-Type: application/json" \
  --request PATCH \
  --data '"{\"host_networks\":\"'$CLUSTER_HOST_NET'\"}"' \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID"

NOTE: If you recieve an error, be sure that you have hosts staged.

10. (a) Finally, it is time to start the installation! There are two ways you can do this; the first is via an API call (of course)

curl -s -X POST \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/actions/install"

10. (b) The other method is to confirming the cluster deployment in the WebIO by clicking Next a couple of times, and finally Finish.

That should do it! You've deployed a custom OpenShift cluster using the Assisted-Installer.