Skip to content

Instantly share code, notes, and snippets.

@maelvls

maelvls/README.md

Last active April 4, 2022 12:39
Show Gist options
  • Save maelvls/97e95e110d561ec3b7194daa260d95d2 to your computer and use it in GitHub Desktop.
Save maelvls/97e95e110d561ec3b7194daa260d95d2 to your computer and use it in GitHub Desktop.
Download ZIP
Deploy a dev version of cert-manager to your GKE or OpenShift cluster
Raw
README.md

Deploy a dev version of cert-manager to your GKE or OpenShift cluster

If you would like to use the new Make flow (we dropped Bazel!), here is a tutorial on how test cert-manager on a non-kind cluster such as GKE or OpenShift. In this tutorial, I assume that you have cloned cert-manager/cert-manager and that you have a shell session open in that folder. I also assume that your GKE or OpenShift cluster has access to the Docker Hub registry (i.e., you cluster has access to the internet).

You will need docker, helm, and crane installed. If you are on macOS or on Linux, you can install helm and crane using Homebrew:

brew install helm crane

First, build the images:

make -j8 bin/containers/cert-manager-{controller,acmesolver,cainjector,webhook,ctl}-linux-amd64.tar

Now, go to https://hub.docker.com and create a registry named cm (for "cert-manager"). We will be pushing the 5 images to that registry under different tags.

🌟 For information, the image names look like this:

maelvls/cm:controller-v1.8.0-alpha.0-17-g719a48996b1b1c
maelvls/cm:acmesolver-v1.8.0-alpha.0-17-g719a48996b1b1c
maelvls/cm:cainjector-v1.8.0-alpha.0-17-g719a48996b1b1c
maelvls/cm:webhook-v1.8.0-alpha.0-17-g719a48996b1b1c
maelvls/cm:ctl-v1.8.0-alpha.0-17-g719a48996b1b1c

Let us set a variable with the repository name in Docker Hub:

REPO=maelvls/cm

Log into Docker Hub:

docker login docker.io -u maelvls

Push the images (the REPO env var that we have set will be used here):

for i in bin/containers/cert-manager-{controller,acmesolver,cainjector,webhook,ctl}-linux-amd64.tar; do \
crane push $i $REPO:$(tar xfO $i manifest.json | jq ".[0].RepoTags[0]" -r | perl -ne '/cert-manager-(.*)-amd64:(.*)/ && print "$1-$2"'); \
done

Finally, let us build the Helm chart, and then install cert-manager with Helm (REPO is also required here):

# First, build the Helm chart tarball:
make bin/cert-manager.tgz

# Then, use that Helm chart:
helm upgrade --install cert-manager bin/cert-manager.tgz --create-namespace -n cert-manager --wait \
  --set image.repository=$REPO \
  --set cainjector.image.repository=$REPO \
  --set webhook.image.repository=$REPO \
  --set startupapicheck.image.repository=$REPO \
  --set image.tag="$(tar xfO bin/containers/cert-manager-controller-linux-amd64.tar manifest.json | jq '.[0].RepoTags[0]' -r | perl -ne '/cert-manager-(.*)-amd64:(.*)/ && print "$1-$2"')" \
  --set cainjector.image.tag="$(tar xfO bin/containers/cert-manager-cainjector-linux-amd64.tar manifest.json | jq '.[0].RepoTags[0]' -r | perl -ne '/cert-manager-(.*)-amd64:(.*)/ && print "$1-$2"')" \
  --set webhook.image.tag="$(tar xfO bin/containers/cert-manager-webhook-linux-amd64.tar manifest.json | jq '.[0].RepoTags[0]' -r | perl -ne '/cert-manager-(.*)-amd64:(.*)/ && print "$1-$2"')" \
  --set startupapicheck.image.tag="$(tar xfO bin/containers/cert-manager-ctl-linux-amd64.tar manifest.json | jq '.[0].RepoTags[0]' -r | perl -ne '/cert-manager-(.*)-amd64:(.*)/ && print "$1-$2"')" \
  --set installCRDs=true \
  --set featureGates="AdditionalCertificateOutputFormats=true\,ExperimentalCertificateSigningRequestControllers=true\,ServerSideApply=true" \
  --set "webhook.extraArgs={--feature-gates=AdditionalCertificateOutputFormats=true}" \
  --set "cainjector.extraArgs={--feature-gates=}" \
  --set "extraArgs={--acme-http01-solver-image=$REPO:$(tar xfO bin/containers/cert-manager-acmesolver-linux-amd64.tar manifest.json | jq '.[0].RepoTags[0]' -r | perl -ne '/cert-manager-(.*)-amd64:(.*)/ && print "$1-$2"')}" >/dev/null

At this point, cert-manager should be ready to be used.

Run a specific end-to-end test

Imagine that you are trying to implement a new feature in cert-manager. Since the end-to-end suite takes a long time to run (approximately 30 minutes), you would like to run a specific end-to-end test. In this section, you will learn:

  1. To figure out which end-to-end test might be of interest to you,
  2. To run this single test.

Understanding how make works

You might be wondering: where can I find where the go build command is being run, and where is docker build? Let us take the example of building the controller image tarball that was shown above:

make bin/containers/cert-manager-acmesolver-linux-amd64.tar

To see the list of intermediate "targets" (i.e., the intermediate steps) that make is calling to get to that image tarball, you can run:

make clean
make bin/containers/cert-manager-controller-linux-amd64.tar --dry-run --debug \
  | perl -pe 's/^( *(?:Must remake target|Successfully remade target|File .* does not exist))/# \1/g'

The output shows the steps (the character # was added to every "debug" line for readability purposes):

#  File 'bin/containers/cert-manager-controller-linux-amd64.tar' does not exist.
#    File 'bin/scratch/build-context/cert-manager-controller-linux-amd64/controller' does not exist.
#      File 'bin/server/controller-linux-amd64' does not exist.
#        File 'bin/server' does not exist.
#       Must remake target 'bin/server'.
mkdir -p bin/server
#       Successfully remade target file 'bin/server'.
#     Must remake target 'bin/server/controller-linux-amd64'.
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 GOMAXPROCS= \
  go build -o bin/server/controller-linux-amd64 -ldflags '-w -s' -trimpath cmd/controller/main.go
#     Successfully remade target file 'bin/server/controller-linux-amd64'.
#      File 'bin/scratch/build-context/cert-manager-controller-linux-amd64' does not exist.
#     Must remake target 'bin/scratch/build-context/cert-manager-controller-linux-amd64'.
mkdir -p bin/scratch/build-context/cert-manager-controller-linux-amd64
#     Successfully remade target file 'bin/scratch/build-context/cert-manager-controller-linux-amd64'.
#   Must remake target 'bin/scratch/build-context/cert-manager-controller-linux-amd64/controller'.
ln -f bin/server/controller-linux-amd64 bin/scratch/build-context/cert-manager-controller-linux-amd64/controller
#   Successfully remade target file 'bin/scratch/build-context/cert-manager-controller-linux-amd64/controller'.
#    File 'bin/scratch/build-context/cert-manager-controller-linux-amd64/cert-manager.license' does not exist.
#      File 'bin/scratch/cert-manager.license' does not exist.
#        File 'bin/scratch/license.yaml' does not exist.
#          File 'bin/scratch' does not exist.
#         Must remake target 'bin/scratch'.
mkdir -p bin/scratch
#         Successfully remade target file 'bin/scratch'.
#       Must remake target 'bin/scratch/license.yaml'.
sed -e "s/YEAR/2021/g" < hack/boilerplate/boilerplate.sh.txt > bin/scratch/license.yaml
#       Successfully remade target file 'bin/scratch/license.yaml'.
#        File 'bin/scratch/license-footnote.yaml' does not exist.
#       Must remake target 'bin/scratch/license-footnote.yaml'.
echo -e "# To view licenses for cert-manager dependencies, see the LICENSES file in the\n# cert-manager repo: https://github.com/cert-manager/cert-manager/blob/491223c93cb863d49efe54c09daedcaea13aaa22/LICENSES" > bin/scratch/license-footnote.yaml
#       Successfully remade target file 'bin/scratch/license-footnote.yaml'.
#     Must remake target 'bin/scratch/cert-manager.license'.
cat bin/scratch/license.yaml bin/scratch/license-footnote.yaml > bin/scratch/cert-manager.license
#     Successfully remade target file 'bin/scratch/cert-manager.license'.
#   Must remake target 'bin/scratch/build-context/cert-manager-controller-linux-amd64/cert-manager.license'.
ln -f bin/scratch/cert-manager.license bin/scratch/build-context/cert-manager-controller-linux-amd64/cert-manager.license
#   Successfully remade target file 'bin/scratch/build-context/cert-manager-controller-linux-amd64/cert-manager.license'.
#    File 'bin/scratch/build-context/cert-manager-controller-linux-amd64/cert-manager.licenses_notice' does not exist.
#      File 'bin/scratch/cert-manager.licenses_notice' does not exist.
#     Must remake target 'bin/scratch/cert-manager.licenses_notice'.
cp bin/scratch/license-footnote.yaml bin/scratch/cert-manager.licenses_notice
#     Successfully remade target file 'bin/scratch/cert-manager.licenses_notice'.
#   Must remake target 'bin/scratch/build-context/cert-manager-controller-linux-amd64/cert-manager.licenses_notice'.
ln -f bin/scratch/cert-manager.licenses_notice bin/scratch/build-context/cert-manager-controller-linux-amd64/cert-manager.licenses_notice
#   Successfully remade target file 'bin/scratch/build-context/cert-manager-controller-linux-amd64/cert-manager.licenses_notice'.
#    File 'bin/release-version' does not exist.
#      File 'FORCE' does not exist.
#     Must remake target 'FORCE'.
#     Successfully remade target file 'FORCE'.
#   Must remake target 'bin/release-version'.
test "v1.7.0-beta.0-258-g491223c93cb863" == "" || echo v1.7.0-beta.0-258-g491223c93cb863 > bin/release-version
#   Successfully remade target file 'bin/release-version'.
#    File 'bin/containers' does not exist.
#   Must remake target 'bin/containers'.
mkdir -p bin/containers
#   Successfully remade target file 'bin/containers'.
# Must remake target 'bin/containers/cert-manager-controller-linux-amd64.tar'.
docker build --quiet \
	-f hack/containers/Containerfile.controller \
	--build-arg BASE_IMAGE=gcr.io/distroless/static@sha256:49f33fac9328ac595cb74bd02e6a186414191c969de0d8be34e6307c185acb8e \
	-t cert-manager-controller-amd64:v1.7.0-beta.0-258-g491223c93cb863 \
	bin/scratch/build-context/cert-manager-controller-linux-amd64/ >/dev/null
docker save cert-manager-controller-amd64:v1.7.0-beta.0-258-g491223c93cb863 -o bin/containers/cert-manager-controller-linux-amd64.tar >/dev/null
# Successfully remade target file 'bin/containers/cert-manager-controller-linux-amd64.tar'.

The following corresponds to the controller binary:

#     Must remake target 'bin/server/controller-linux-amd64'.
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 GOMAXPROCS= \
  go build -o bin/server/controller-linux-amd64 -ldflags '-w -s' -trimpath cmd/controller/main.go

Now, where is the target bin/server/controller-linux-amd64 located in the Makefile? It is possible to search for it. This target is defined in the file make/server.mk:

$ grep bin/server/controller-linux-amd64 -R Makefile **/*.mk --files-with-matches
make/server.mk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment