Skip to content

Instantly share code, notes, and snippets.

@joeljacobson

joeljacobson/SelfServiceDemo.md

Last active February 10, 2025 16:28
Show Gist options
  • Save joeljacobson/051570977be90872db3005108614ac52 to your computer and use it in GitHub Desktop.
Save joeljacobson/051570977be90872db3005108614ac52 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
SelfServiceDemo.md

Self Service Demo

This demo illustrates how a central platform team can provide govenance and control within an organisation enabling application teams to deploy, manage and share Kafka resources within their environments, empowering them to act autonomously.

Accompanying deck - https://docs.google.com/presentation/d/1qi4ufN0T3anZO8MS-BAG5qwWSmmdMperguAsSypFU78/edit#slide=id.g3275e09e81c_0_113

For this demo we will have 3 terminal windows open. The first one is for the central platform team to configure predefined rules for the application teams. The second terminal represents an application team called Orders, and the last represents a Business Intelligence application team.

First, let's start Conduktor Scale, add a sink connector, and add the Self Service demo.
./start.sh
./add_connect.sh
./add_self_serve.sh

Now we can see a some resources have been created. We have:

  • Application Instances - We have created one for the JDBC sink connector, the Business Intelligence and the Order applicaton teams.
  • Applications - We now have the deployed applications themselves, again, one for the Sink connector, the Business Intelligence dev app and the Orders dev app.
  • Groups - We have defined the Data Stewards/Project Owners for the Business Intelligence and Orders application instances.
  • Topic Policies - These are the policies set by the platform team that the application teams must adhere to.

Now let's authenticate as each team on the subsequent terminal windows so that we can use the conduktor-cli, in the first one we will authenticate as the Order app team source setup_order_cli.sh and in the other terminal window we will authenticate as the Business Intelligence app team source setup_bi_cli.sh

Log into the UI as Admin on http://localhost:8080/ and navigate to the Application Catalog. Let's take a look at the Orders App and navigate to the Instances tag. Here we can see the applications that have been deployed e.g order-app-dev and all owned resource types must be prefixed with order-. What's also useful is here we can see the policies that the application team must adhere to e.g default-dev-topic-policy feel free to click them and talk through the yaml definitions.

Let's now log in as the user Alice who belongs in the Order app team and navigate to the topics tab. We will now create some topics using the conduktor-cli. In the Order app terminal run conduktor apply -f resources/self-serve/order-app-resources/topics.yaml. We see that 3 topics have been created. You may take this oppotunity to talk through the topics.yaml, this will give the indexer enough time for them to show in the topic catalog.

If we go back to the UI we will see the 3 topics created under the topics tab. By now we should see the indexed topics in the topic catalog. Notice how we only see 2 of the 3 topics we created. The reason for this is that if you inspected the topic.yaml you would see that the first topic created is the order-requests topic has been marked as catalogVisibility: PRIVATE. Each application team has the flexibilty to hide topics if they choose. A topic might contain data that needs to be cleansed before it is useful.

  1. Let's produce some messages in the order-validated topic. Under the produce tab click generate once and under the flow section, insert 10 records, generate random value, then produce the messages.
  2. In the topics tab, let's try to create a new topic as Alice, name the topic Alice and try to create. This will fail as we observed earlier that all topics in the Order app team must be prefixed with order-. Let's fix it by naming it order-alice.
  3. Now we have a different error, we did not add a data-classifiction label. Let's fix that. Add a data-classification label with one of the following values: C0,C1,C2.
  4. Now we are able to create the topic.

Now we will demonstrate data sharing and how another teams, in this case the BI team, can subscribe to topics owned by the order app team

Let's log out of the UI and log in as the user Bob. Let's go to the Topic Catalog, here we can see the public topics that are owned by the order app team. As part of the BI team we want to run some analytics on the order-validated topic and we would like to request read access to this topic.

  1. Click on the order-validated topic and click the subscribe button.
  2. Select bi-app under Your Application and copy the yaml
  3. In the BI terminal, let's create this PR in the Order app directory for them to approve and apply vi resources/self-serve/order-app-resources/order-validated-bi-app-dev.yaml
  4. Let's say Bob is impatient and tried to apply this himself, from the BI app terminal, try to apply this change conduktor apply -f resources/self-serve/order-app-resources/order-validated-bi-app-dev.yaml You will notice that Bob is missing the permissions to do so.
  5. From the Order app terminal let's apply the file conduktor apply -f resources/self-serve/order-app-resources/order-validated-bi-app-dev.yaml. Now we have successfully applied this change
  6. Navigate to the Topics tab and observe the order-validated topic.

If you remember, we only requested READ access to that topic. Let's try to prodiuce a message and we will observe that Bob has insufficient permissions.

We can revoke access to the topic with conduktor delete -f resources/self-serve/order-app-resources/order-validated-bi-app-dev.yaml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment