idp-at-dev.md

Intro

FIXME: The goals: FIXME: Do whatever needs to do done to develop something and test it under similar conditions as if it's running in production. FIXME: Everything (or almost everything) is in the repo.

FIXME: Example: Backend up that depends on a front-end and a database

FIXME: Welcome to DevOps Toolkit, the channel where we...

FIXME: Shock

FIXME: Establish expectations

FIXME: What's the ending?

Setup

FIXME: Create a branch

git clone https://github.com/vfarcic/idp-full-demo

cd idp-full-demo

devbox shell

Watch How to Create Custom CLIs for Internal Developer Platforms with Nushell if you are not familiar with Nushell. Alternatively, you can inspect the platform script and transform the instructions in it to Bash or ZShell if you prefer not to use that Nushell script.

FIXME: Test Google FIXME: Test AWS FIXME: Test Azure FIXME: Test UpCloud FIXME: Remove kind

platform setup dev

source .env

FIXME: Move to platform

yq --inplace \
    ".spec.rules[0].host = \"silly-demo-frontend.$INGRESS_HOST\"" \
    frontend/ingress.yaml

yq --inplace \
    ".spec.template.spec.containers[0].env[0].value = \"http://staging.silly-demo.$INGRESS_HOST\"" \
    frontend/deployment.yaml

yq --inplace \
    ".spec.parameters.host = \"staging.silly-demo.$INGRESS_HOST\"" \
    crossplane/app-staging-db.yaml

exit

Development Tools

devbox shell

source .env

okteto version

which okteto

cat devbox.json

cat platform

platform --help

Staging Environment

NOTE: Other apps that should be accessed from the one we're developing need only a config with the Service name and Namespace. NOTE: Apps that should be accessing the one we're developing are a challenge that can be solved with a service mesh (Telepresence and mirrord assume local development).

cat crossplane/$PROVIDER-sql.yaml

# FIXME: If Google
echo "{\"password\": \"IWillNeverTell\" }" \
    | gcloud secrets --project $PROJECT_ID \
    create db-root-password --data-file=-

kubectl --namespace staging apply \
    --filename crossplane/$PROVIDER-sql.yaml

crossplane beta trace sqlclaim silly-demo-db --namespace staging

Wait until all the resources are Available.

kubectl --namespace staging get secrets

kubectl --namespace staging get externalsecrets,pushsecrets

Wait until all the STATUS of the pushsecret is Synced.

cat crossplane/app-staging-db.yaml

kubectl --namespace staging apply \
    --filename crossplane/app-staging-db.yaml

crossplane beta trace appclaim silly-demo --namespace staging

FIXME: Switch to AppClaim

cat frontend/*.yaml

kubectl --namespace staging apply --filename frontend

kubectl --namespace staging get all,ingresses

kubectl --namespace staging wait deployment silly-demo-frontend

kubectl --namespace staging wait pods --for=condition=Ready \
    --selector app.kubernetes.io/name=silly-demo-frontend

echo "http://silly-demo-frontend.$INGRESS_HOST"

Open the URL from the output of the previous command in a browser.

# cat crossplane/app-dev-db.yaml

# platform apply dev vfarcic --db silly-demo-db

Execute the command that follows in the second (new) terminal session.

# curl "http://vfarcic.silly-demo.$INGRESS_HOST"

# curl -XPOST \
#   "http://vfarcic.silly-demo.$INGRESS_HOST/video?id=z7Nfl-u-hLI&title=Neon"

# curl -XPOST \
#   "http://vfarcic.silly-demo.$INGRESS_HOST/video?id=WAm3ypS0_wg&title=IDP%20History"

# curl "http://vfarcic.silly-demo.$INGRESS_HOST/videos" | jq .

# kubectl --namespace vfarcic get externalsecrets,secrets

Press ctrl+c (in the first terminal) to stop the watcher.

NOTE: That could be disruptive. How about ephemeral databases?

Development Environments (FIXME: Remove?)

NOTE: We need production-like, not a stripped down version of the app. Local Kubernetes cluster is too complicated to setup. Going remote is too expensive. What's left?

curl "http://staging.silly-demo.$INGRESS_HOST"

curl -XPOST \
    "http://staging.silly-demo.$INGRESS_HOST/video?id=z7Nfl-u-hLI&title=Neon"

curl "http://staging.silly-demo.$INGRESS_HOST/videos" | jq .

cat .mirrord/mirrord.json

NOTE: Even though it states that the default location for the config file is .mirrord/mirrord.json it needs to be specified explicitly.

mirrord exec --config-file .mirrord/mirrord.json air go run .

Open a new terminal session in the same directory

Execute the commands that follow in the second (new) terminal session.

devbox shell

source .env

curl "localhost:8080"

FIXME: Fix stealing

curl "http://staging.silly-demo.$INGRESS_HOST"

curl -XPOST \
    "localhost:8080/video?id=WAm3ypS0_wg&title=IDP%20History"

curl "localhost:8080/videos" | jq .

curl "http://staging.silly-demo.$INGRESS_HOST/videos" | jq .

Development Environments (FIXME: Remove?)

NOTE: We'll use Crossplane but the same logic would apply for any other tech (e.g., Helm)

NOTE: Assumption is that a repo with all the files already exists. That repo could have been created from a template.

NOTE: Not going through permissions and multi-tenancy. Check Capsule...

FIXME: Add a DB

FIXME: Add a frontend

cat crossplane/app-dev.yaml

NOTE: If we'd use the commercial version of Okteto we might not need to do what we're about to do.

FIXME: Remove parameters.namespace from AppClaim.

yq --inplace \
    ".spec.id = \"egulliksen-silly-demo\"" \
    crossplane/app-dev.yaml

yq --inplace \
    ".spec.parameters.namespace = \"egulliksen\"" \
    crossplane/app-dev.yaml

yq --inplace \
    ".spec.parameters.host = \"egulliksen.silly-demo.$INGRESS_HOST\"" \
    crossplane/app-dev.yaml

yq --inplace \
    ".spec.parameters.db.secret = \"egulliksen-silly-demo-db\"" \
    crossplane/app-dev.yaml

kubectl --namespace egulliksen get all,ingresses

curl "http://egulliksen.silly-demo.$INGRESS_HOST"

NOTE: How do we update it every time we make a change to the code? Build a new image, push it to a registry, update manifests and apply them? That takes too long. We should have near instant updates.

cat okteto.yaml

NOTE: We could have put deployment into okteto.yaml as well, but that's only available in the commercial version (which is worth considering).

okteto context

okteto up --namespace egulliksen

Wait until it is up-and-running.

Open a new terminal session in the same directory

Execute the commands that follow in the second (new) terminal session.

devbox shell

source .env

curl "http://egulliksen.silly-demo.$INGRESS_HOST"

Edit the message This is a silly demo in root.go.

Execute the command that follows in the second terminal session.

curl "http://egulliksen.silly-demo.$INGRESS_HOST"

From here on, execute the commands in the first terminal session unless specified otherwise.

Press ctrl+c to stop the watcher.

kubectl --namespace egulliksen get all,ingresses

okteto down --namespace egulliksen

kubectl --namespace egulliksen delete \
    --filename crossplane/app-dev.yaml

platform apply dev --help

platform apply dev vfarcic

Undo the previously made changes to root.go.

Execute the command that follows in the second (new) terminal session.

curl "http://vfarcic.silly-demo.$INGRESS_HOST"

curl "http://vfarcic.silly-demo.$INGRESS_HOST/videos"

Press ctrl+c to stop the watcher.

platform uninstall dev vfarcic

cat platform

NOTE: There is Okteto VS Code extension.

NOTE: How do we connect it to other apps in the system (e.g., DB, frontend, etc.)

Frontend (mirrord, service mesh, ?)

curl "http://vfarcic.silly-demo.$INGRESS_HOST"

Ephemeral Databases (Neon?)

FIXME:

Testing

go test -v -tags unit $PWD/...

air go test -v -tags unit $PWD/...

Press ctrl+c to stop watching for changes.

URL="http://egulliksen.silly-demo.$INGRESS_HOST" \
    go test -v -cover -tags integration $PWD/...

FIXME: Do it in a loop with air

Catalog (Backstage)

FIXME: Not very useful during. Developers are in VS Code or Jetbrains something something and not in Backstage while developing. Backstage comes in handy later...

IDE (VS Code)

FIXME:

FIXME: Pros and Cons

TODO: Header: Cons; Items: FIXME:

TODO: Header: Pros; Items: FIXME:

Destroy

kubectl --namespace staging delete \
    --filename crossplane/$PROVIDER-sql.yaml

mut command = { kubectl get managed --output name }
if ($name | is-not-empty) {
    $command = {
        (
            kubectl get managed --output name
                --selector $"crossplane.io/claim-name=($name)"
        )
    }
}

mut resources = (do $command)
mut counter = ($resources | wc -l | into int)

while $counter > 0 {
    print $"($resources)\nWaiting for remaining (ansi green_bold)($counter)(ansi reset) managed resources to be (ansi green_bold)removed(ansi reset)...\n"
    sleep 10sec
    $resources = (do $command)
    $counter = ($resources | wc -l | into int)
}

platform destroy kubernetes $PROVIDER

exit