complete setup of a local k3d cluster on macos with flux enabled on local git server

k3d_local_git.sh

#!/usr/bin/env bash

set -m  # enable job control

## change your LAN ip address in the variable "PUBLICIP" before run
## run with "-d" command line switch to debug each step, script will wait for keypress to go on
## integrated mods by KingdonB in his fork: https://gist.github.com/kingdonb/dec74f3b74ffbb83b54d53d5c033e508
## added proper coredns patching via "coredns-custom" configmap
## added automatic /etc/hosts file modification if needed, sudo password will be asked in case
## added (commented out) lines to add Flux extra controllers to the setup
## deployed podinfo as in Flux Get Started guide
## added kustomize patch to change color in podinfo deployment
## issue: https://github.com/fluxcd/flux/issues/3594

CONFIG_HOME=~/testgit/
CONFIG_REPO=testrepo
CLONE_PATH=${CONFIG_HOME}/${CONFIG_REPO}
BARE_REPO_PATH=${CONFIG_HOME}/gitsrv/${CONFIG_REPO}.git
CLUSTER_NAME=testclu
PUBLICIP=10.17.12.102
PUBLICDNS=e4t.example.com
GITSRV_IMG=jkarlos/git-server-docker

echo "creating folder structure and ssh keys"
mkdir -p ${CONFIG_HOME}/{${CONFIG_REPO},sshkeys,gitsrv}
ssh-keygen -b 521 -o -t ecdsa -N "" -f ${CONFIG_HOME}/sshkeys/identity

echo "creating a test repo for the gitsrv"
cd ${CLONE_PATH}
git init --shared=true
echo Nothing > README.md
git add .
git commit -m "1st commit"
git clone --bare ${CLONE_PATH} ${BARE_REPO_PATH}

[ "$1" == "-d" ] && read -p "Press [Enter] key to continue and create k3d cluster..."

echo "creating a k3d local test cluster"
k3d cluster create ${CLUSTER_NAME}

echo "Waiting for CoreDNS deploying"
ATTEMPTS=0
ROLLOUT_STATUS_CMD="kubectl rollout status deployment/coredns -n kube-system"
until $ROLLOUT_STATUS_CMD || [ $ATTEMPTS -eq 120 ]; do
  $ROLLOUT_STATUS_CMD
  ATTEMPTS=$((ATTEMPTS + 1))
  sleep 1
done

echo "CoreDNS deployed, waiting for its ConfigMap"
while true ; do
    result=$(kubectl -n kube-system get configmap coredns 2>&1)
    if [[ $result == *NotFound* ]] || [[ $result == *refused* ]]; then
        sleep 1
    else
        break
    fi
done

[ "$1" == "-d" ] && read -p "Press [Enter] key to continue and patch coredns..."

echo "checking /etc/hosts line presence, or adding it if missing"
if grep -q "$PUBLICDNS" /etc/hosts ; then
    echo "/etc/hosts file already modded"
else
    echo "127.0.0.1 $PUBLICDNS" | sudo tee -a /etc/hosts > /dev/null
    echo "added missing '127.0.0.1 $PUBLICDNS' line to your /etc/hosts file"
fi

echo "CoreDNS ConfigMap ready, patching it"
    read -rd '' coredns_custom << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  example.server: |
    example.com {
      hosts {
          ${PUBLICIP} ${PUBLICDNS}
          fallthrough
      }
      whoami
    }
EOF

echo "$coredns_custom" > ${CONFIG_HOME}/coredns-custom.yaml
kubectl apply -f ${CONFIG_HOME}/coredns-custom.yaml
kubectl get pods -n kube-system|grep coredns|cut -d\  -f1|xargs kubectl delete pod -n kube-system
# to test new dns record working, use: kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

[ "$1" == "-d" ] && read -p "Press [Enter] key to continue and create local git server..."

echo "creating the local gitserver pointing to the folders created above"
docker run -d -p 2222:22 \
-v ${CONFIG_HOME}/sshkeys:/git-server/keys \
-v ${CONFIG_HOME}/gitsrv:/git-server/repos \
--name gitsrv $GITSRV_IMG

echo "wait for port 2222 to be opened"
while ! nc -z localhost 2222; do
  sleep 0.1
done

echo "adding ssh key and testing ssh"
ssh-add -D ${CONFIG_HOME}/sshkeys/identity
ssh-add ${CONFIG_HOME}/sshkeys/identity
sleep 1
ssh git@${PUBLICDNS} -p 2222

[ "$1" == "-d" ] && read -p "Press [Enter] key to continue and bootstrap Flux on local git server and cluster..."

echo "bootstrapping flux on test cluster, using SAME private key already used for the git server"
flux bootstrap git --branch=develop --path=. \
--url=ssh://git@${PUBLICDNS}:2222/git-server/repos/${CONFIG_REPO}.git \
--private-key-file=../sshkeys/identity

[ "$1" == "-d" ] && read -p "Press [Enter] key to continue cloning the git repo and adding coredns ..."

echo "cloning created git repo and adding coredns patch"
cd ${CONFIG_HOME}
rm -rf ${CLONE_PATH}
git clone ssh://git@${PUBLICDNS}:2222/git-server/repos/${CONFIG_REPO}.git -b develop
cd ${CLONE_PATH}
mv ${CONFIG_HOME}/coredns-custom.yaml ${CLONE_PATH}
git add -A && git commit -m "Add coredns patch"
git push

[ "$1" == "-d" ] && read -p "Press [Enter] key to continue and add podinfo git repo..."

echo "Add podinfo repository to Flux"
flux create source git podinfo \
  --url=https://github.com/stefanprodan/podinfo \
  --branch=master \
  --interval=30s \
  --export > podinfo-source.yaml
git add -A && git commit -m "Add podinfo GitRepository"
git push

[ "$1" == "-d" ] && read -p "Press [Enter] key to continue and add podinfo application..."

echo "Deploy podinfo application"
flux create kustomization podinfo \
  --target-namespace=default \
  --source=podinfo \
  --path="./kustomize" \
  --prune=true \
  --wait=true \
  --interval=5m \
  --export > podinfo-kustomization.yaml
git add -A && git commit -m "Add podinfo Kustomization"
git push

echo "Waiting for PodInfo deploying"
ATTEMPTS=0
ROLLOUT_STATUS_CMD="kubectl rollout status deployment/podinfo -n default"
FLUX_RECONCILE_CMD="flux reconcile kustomization flux-system --with-source"

$FLUX_RECONCILE_CMD &
until $ROLLOUT_STATUS_CMD || [ $ATTEMPTS -eq 120 ]; do
  $ROLLOUT_STATUS_CMD
  ATTEMPTS=$((ATTEMPTS + 1))
  sleep 1
done

echo "opening http://localhost:9898 in your browser (only on Mac - do it manually on other systems)"
kubectl port-forward deployment/podinfo 9898:9898 &
PID=$!
open http://localhost:9898

[ "$1" == "-d" ] && read -p "Press [Enter] key to continue and change podinfo background via kustomize patch..."

# kill -9 $PID
echo "adding kustomize patch to alter podinfo background"
cd ${CLONE_PATH}
IFS= read -rd '' podinfopatch << EOF
  patches:
    - patch: |-
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: podinfo
        spec:
          template:
            spec:
              containers:
              - name: podinfod
                env:
                - name: PODINFO_UI_COLOR
                  value: '#ffff00'
        target:
          name: podinfo
          kind: Deployment
EOF
echo -e "$podinfopatch" >> podinfo-kustomization.yaml
git add -A
git commit -m "changed color in podinfo deployment via kustomize patch"
git push
$FLUX_RECONCILE_CMD # flux will wait for the new deploy to become ready, but we cannot create a new
$ROLLOUT_STATUS_CMD # port-forward on the same port before it finished terminating! wait for that,
fg %2               # then also wait for port-forward to exit, to be sure it closes its port here.

#we can now also get away with opening the browser just once!
# echo "open http://localhost:9898 in your browser"
exec kubectl port-forward deployment/podinfo 9898:9898 &
PID=$!
# open http://localhost:9898

echo; echo
# [ "$1" == "-d" ] &&
read -p "Press [Enter] key to continue and close portforward, ending script (or ^C to leave it running)"
kill -9 $PID

echo "to remove everything, run the following lines"
echo "docker stop gitsrv"
echo "docker rm gitsrv"
echo "k3d cluster delete testclu"
echo "rm -rf ~/testgit"

# to add flux extra controllers, uncomment and run next lines:
# echo "adding image-reflector-controller,image-automation-controller Flux extra controllers"
# cd ${CLONE_PATH}/flux-system
# flux install --components-extra=image-reflector-controller,image-automation-controller --export > gotk-components.yaml
# git add .
# git commit -m "adding Flux image-reflector-controller and image-automation-controller"
# git push
# $FLUX_RECONCILE_CMD &

Hi @kingdonb , sorry if i write here but bot closed comments on that issue, i've no other ways to contact you right now... i've found a way to cleanly patch coredns, by creating a "coredns-custom" cm and applying it to the cluster (and restarting coredns), i created one as follows, applied, i can dig example.org but not the nodehosts record i added... to test this, i used the dnsutils pod mentioned in previous link. Thanks in advance

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        hosts /etc/coredns/NodeHosts {
          192.168.1.13 e4t.example.com
          ttl 60
          reload 15s
          fallthrough
        }
    }
  example.server: |
    example.org {
      log
      whoami
    }

@kingdonb hi, I succeeded in patching coredns "the right way", look at my updated gist please :)
I obviously integrated all your precious mods there (credit given in top lines), and added /etc/hosts modification too
I added the configmap to the test repo, just for the sake of it :D

@fragolinux You're awesome, and you're welcome to contact me, that issue is just not the appropriate forum for this conversation, as it's not a place for support of Flux v2 at-large, but as you must have seen I'm interested in this conversation and I'm happy to continue it :)

It was me who closed the issue and locked it though, not a bot. (It looked like a good time to close it, not trying to shut down though!)

Do you have any interest in giving a demo of this? We do Flux demos in many different venues all the time, and I think there's an informal benchmark or metric for "how many seconds of boilerplate and such must we endure as part of the talk before the real demo can begin" that we like to minimize, in order to help progress things and keep the demo pace fast enough that people don't lose interest...

Well this script certainly is making a few leaps forward in terms of speed to "Day 1 Operations"

I'm almost thinking, we should have something like a more formal community of practice for Flux Demos. Most people don't need much polish, like you said there were places in this script where you made shortcuts and I shared criticism to try to make it better embody the "spirit of the principles of GitOps" – and now we're doing all that! This is awesome, I want it to happen every month 😆

Are you familiar with the Flux Bug Scrub weekly event? We're talking about doing some informal talks there soon, maybe you could join us and present the script! (I could present it, but I don't want to take credit!)

Edit: https://fluxcd.io/#calendar <- the Flux Bug Scrub event info is linked from here, with time in UTC

I'd be proud to share that, I just don't know if I'm able to explain each detail in a proper way, I don't speak english so often and my accent could be a "bit"thick :)
please contact me at fragolino AT gmail.com, maybe it's better to continue privately? Thanks a lot, your appreciation means A LOT for me :)

Awesome. I sent you an e-mail :)

@fragolinux I adopted your changes as I checked out what you've done since we talked, this is good!

I see there is some demo-magic with -d and it makes the port-forward stuff work and the action to progress in a way where you can follow what's going on. Awesome.

I thought about adding ingress and had some idea with "nohup" to let the port-forward survive the script, but now that I see it's killed on purpose and why, I understand the reasoning and would leave it this way, except it doesn't quite seem to work. (I guess the -d is for debug!)

That's something I would consider a great "exercise for the reader" and so I might still leave it this way. But in the next iteration of this script, I'd either add ingresses to resolve this issue fully, or use the actual "demo-magic" to make the action even easier to follow.

The problem with port-forward is of course that it forwards your port directly to a pod, cannot forward to a virtual IP, and so even if you forward with a service name, you'll get a direct connection to a pod which is broken when the pod dies. It's possible to pick a pod, but it's generally not possible to seamlessly transition a port-forward unless you have a good way of stopping it from triggering until the old one is killed. (Maybe there is a good easy answer for that, let me take a quick look actually while it's on my mind...)

https://github.com/paxtonhare/demo-magic

Here is the demo magic I'm talking about, you can see it in action in this webinar we put on a while back:
https://community.cncf.io/events/details/cncf-cloud-native-scale-presents-the-power-of-three-flux-flagger-linkerd-1/

It's a slightly more elaborate demo-ish polished version of exactly what you've done here for debugging. Also goes great with bat which is better cat I guess, or a colorized pager.

I made some enhancements

You can also choose to end the script with ^C, which leaves everything in place including the port-forward (thanks to exec, this works I think) or end it with Enter key, which cleans up port-forward and echos the instructions what to do next in order to tear this all down.

12ac58c (my gist is updated) has this change now. (A gist does not make it easy to link to a specific revision, except in raw format!)

TIL what nohup and exec are actually for and how I've probably been using them wrong, or at least confusing them both for about 20 years.

Still not sure I am using these right, but now the hand-off between port forwards is smooth in the least, it works as I intended, and you can really call this "one key Flux" (so long as it's understood that the one key 'y' comes after you download and run the script.)

I can think of two more enhancements at least, and one beyond that I would probably do for myself but I would not include in the script as long as it's used as a "first exposure to Flux and Kubernetes" vector as you explained your intention for it. But I may have other plans!

1. Display the cleanup instructions no matter what, by setting a trap in the script (even if exited with ^C you should get some instructions about what to do next.)
2. Subshell that monitors for rollout progress, join a previous port-forward, and start a new one after it's finished. (As many times as needed while the script still runs! And why not after that...)
3. (Add an ingress and dispose of the port-forwarding stuff -- I suggest you don't do this!)

Like I mentioned before, port-forward (a simple mechanism) is easier to understand than Ingress (another API to learn, and it won't necessarily be consistently available once you veer off k3d) so I would prefer (1&2) as it comes logically before (3), for the specific purpose of this script as you built it and intended it to be used in your company.

Except for me, (3) is more in line with everything I already do, so adding it may provide further opportunities down the line (like Flagger) and every cluster I personally use has at least one Ingress class already planned or in the cards for me. So I will probably do that next.

Thanks again for contributing this idea!

@kingdonb in revision 9, why saving the pid at 169 now, if kill is commented out at 174?

Hi Kingdon! Well, as you imagine, I added the "-d" cli option just for the demo you want me to do...

I thought it's not that good to have a script running full throttle which in 2 or 3 minutes will do everything, without the time to explain what's going on during the live :)

I prepared a setup with screen splitted in 4, with 2 k9s to show what's going on in the cluster in various areas, as script goes on, and on the left a panel with the script running and an other, just as you say with BAT (are you in my mind? :D ), showing the code to explain it :)

Ingress, I thought about that but I thought it was beyond the scope of this script, which is just to show how you can have a fully working "gitops" setup locally, with a cluster and a git repo working together... port-forward is enough in this setup, as you just need to see the demo service is up and running...

I planned to show in the live how the resources where patched AS IT HAPPENS, how to modify "inline" the podinfo background, how flux will revert back this as soon as it will reconcile the podinfo kustomization, how to patch the latter via git and how this desired state is then applied in the right way to the deployment...

The kill part was indeed needed to avoid port-forward conflicts as they're lost on pod recreation, your way to deal with this is cleaner, thanks!

I was even thinking to switch from docker desktop to rancher desktop, which does not need port-forwarding as you can access services directly, as you showed in one of your recent videos... but I thought this was enough for this presentation... :)

Right now I'm integrating all of this in the local setup for our developers, simulating what we already do on cloud services, daily :)

I was aiming to make it work with one key, the -d mode will be nice for the demo of this script 👍

...but when I show this off later myself, at least the first time I'm going to be running a stop-watch, and the key feature will be "it takes care of all the stuff we've already demonstrated in prior videos, in less than 200 seconds without any of my attention required except for this single keystroke about halfway through"

This gives roughly just enough time to speak quickly about "what that is we did many times before," or even enough to split between that and a summary of what we'll do next.

As for saving the PID, I commented out just enough to communicate what changes I would make, I wasn't aiming for clean or polished – just reliably functional. There is no reason to save that PID now. You can cherry-pick and amend to your heart's content. I also noticed the trailing slash/duplicated slash issue myself, and had that fix saved in a Git stash.

One other minor improvement, it took me a while to figure out that fg %1 was wrong and I should fg %2 – I also learned then about fg %+ and fg %- one of which is probably correct and wouldn't require anymore counting up the number of times we suspend and toss a port-forward process into the background. With a minor enhancement like that, it would be easy to:

(2) Subshell that monitors for rollout progress, join a previous port-forward, and start a new one after it's finished. (As many times as needed while the script still runs!)

@fragolinux It's been a while since I was this excited about a BASH script, thanks again for sharing this 😃