kafka-gitops-demo.md

Kafka GitOps Demo

A demo with custom ACLs. Ensure you have kafka-gitops installed on your machine.

Setup

First, clone the kafka-gitops repository and start up the docker compose found here. This gives us a principal test to work with and set ACLs on.

Second, set environment variables so kafka-gitops can connect to the cluster:

export KAFKA_BOOTSTRAP_SERVERS=localhost:9092 \
  && export KAFKA_SASL_JAAS_USERNAME=test \
  && export KAFKA_SASL_JAAS_PASSWORD=test-secret \
  && export KAFKA_SASL_MECHANISM=PLAIN \
  && export KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT

Example

Create a example.yaml file that looks like this:

topics:
  test:
    partitions: 6
    replication: 1
    configs:
      cleanup.policy: compact

services:
  my-service:
    type: application
    principal: User:test

customServiceAcls:
  my-service:
    read-topic-test:
      name: test
      type: TOPIC
      pattern: LITERAL
      host: "*"
      principal: User:test
      operation: READ
      permission: ALLOW
    allow-all-groups:
      name: "*"
      type: GROUP
      pattern: LITERAL
      host: "*"
      principal: User:test
      operation: READ
      permission: ALLOW

Plan

Now, we can generate a plan against the cluster:

kafka-gitops -f example.yaml plan -o plan.json

This will output:

Generating execution plan...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

The following actions will be performed:

Topics: 1 to create, 0 to update, 0 to delete.

+ [TOPIC] test
	+ cleanup.policy: compact


ACLs: 2 to create, 0 to update, 0 to delete.

+ [ACL] my-service-0
	 + resource_name: test
	 + resource_type: TOPIC
	 + resource_pattern: LITERAL
	 + resource_principal: User:test
	 + host: *
	 + operation: READ
	 + permission: ALLOW


+ [ACL] my-service-1
	 + resource_name: *
	 + resource_type: GROUP
	 + resource_pattern: LITERAL
	 + resource_principal: User:test
	 + host: *
	 + operation: READ
	 + permission: ALLOW


Plan: 3 to create, 0 to update, 0 to delete.

We can see, as we defined in our file, that we will create the new topic as well as the two ACLs.

Apply

Run the apply:

kafka-gitops -f example.yaml apply -p plan.json

This will output something like this:

Executing apply...

Applying: [CREATE]

+ [TOPIC] test
	+ cleanup.policy: compact


Successfully applied.

Applying: [CREATE]

+ [ACL] my-service-0
	 + resource_name: test
	 + resource_type: TOPIC
	 + resource_pattern: LITERAL
	 + resource_principal: User:test
	 + host: *
	 + operation: READ
	 + permission: ALLOW


Successfully applied.

Applying: [CREATE]

+ [ACL] my-service-1
	 + resource_name: *
	 + resource_type: GROUP
	 + resource_pattern: LITERAL
	 + resource_principal: User:test
	 + host: *
	 + operation: READ
	 + permission: ALLOW


Successfully applied.

[SUCCESS] Apply complete! Resources: 3 created, 0 updated, 0 deleted.

Verify

Run the plan command again, and we should see that there are no needed changes.

kafka-gitops -f example.yaml plan

This will output:

Generating execution plan...

[SUCCESS] There are no necessary changes; the actual state matches the desired state.

Deletion

Awesome, we've now created things via kafka-gitops. It will also delete topics/ACLs that are not defined within the state file.

Create a properties file so we are able to be authenticated to create topics:

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="test" password="test-secret";
sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT

For example, manually add a topic like this:

kafka-topics --command-config admin.properties --bootstrap-server localhost:9092 \
  --create --topic delete-me --partitions 6 --replication-factor 1

Additionally, manually add an ACL like this:

kafka-acls --command-config admin.properties --bootstrap-server localhost:9092  --add \
  --allow-principal User:test --operation WRITE --topic delete-me

Plan

Now, re-run your plan against the cluster without updating your state file.

kafka-gitops -f example.yaml plan -o plan.json

Output will look like:

Generating execution plan...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - delete

The following actions will be performed:

Topics: 0 to create, 0 to update, 1 to delete.

- [TOPIC] delete-me


ACLs: 0 to create, 0 to update, 1 to delete.

- [ACL] Unnamed ACL
	 - resource_name: test
	 - resource_type: TOPIC
	 - resource_pattern: LITERAL
	 - resource_principal: User:test
	 - host: *
	 - operation: WRITE
	 - permission: ALLOW


Plan: 0 to create, 0 to update, 2 to delete.

Conclusion

As you can see, the new plan lets us know the manually created topic and ACL will be removed when applied. You can then run the apply command from earlier to execute the plan.

I hope that helps!

Hi devshawn,
Thank you for helping with your answers.
I have two questions after studying kafka-gitops in depth.

1. Can we assign multiple ACL's at once for a given topic or group?
   Example: I have a topic 'gitops-test-topic' and if I would like to add 4 users to read and write from that topic.
   Is it possible to add all the 4 principal at once in services or customServiceAcls like how we add ACL's using command line?

services:
  test-service:
    type: application
    principal: User:testservice1 User:testservice2 User:testservice3 User:testservice4
    produces:
      - gitops-test-topic
    consumes:
      - gitops-test-topic

or Do I need to define ACL's 4 times in services section?

And

2. As documented here it is really a helpful option to specify the topics not to delete when we execute the state file.
   When I add a topic to blacklist, I can see that topic is not deleted but the ACL's assigned to that topics are listed in deleted section when I execute the plan.

Is there a option not to delete the ACL's when topic is listed in blacklist? (I can see workaround for this is to define the ACL's again in the state file).

Please correct me if my understanding is wrong.
Thanks in advance!! :)

Hi devshawn,

We have existing kafka cluster that we run on production and would like to automate kafka topic/acl using kafka-gitops. We have topics that are auto-generated based on auto-scale and would like to know if its possible to blacklist based on regular expression or some pattern. Is this currently doable?

Thanks in advance!