kafka-gitops-demo.md

Kafka GitOps Demo

A demo with custom ACLs. Ensure you have kafka-gitops installed on your machine.

Setup

First, clone the kafka-gitops repository and start up the docker compose found here. This gives us a principal test to work with and set ACLs on.

Second, set environment variables so kafka-gitops can connect to the cluster:

export KAFKA_BOOTSTRAP_SERVERS=localhost:9092 \
  && export KAFKA_SASL_JAAS_USERNAME=test \
  && export KAFKA_SASL_JAAS_PASSWORD=test-secret \
  && export KAFKA_SASL_MECHANISM=PLAIN \
  && export KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT

Example

Create a example.yaml file that looks like this:

topics:
  test:
    partitions: 6
    replication: 1
    configs:
      cleanup.policy: compact

services:
  my-service:
    type: application
    principal: User:test

customServiceAcls:
  my-service:
    read-topic-test:
      name: test
      type: TOPIC
      pattern: LITERAL
      host: "*"
      principal: User:test
      operation: READ
      permission: ALLOW
    allow-all-groups:
      name: "*"
      type: GROUP
      pattern: LITERAL
      host: "*"
      principal: User:test
      operation: READ
      permission: ALLOW

Plan

Now, we can generate a plan against the cluster:

kafka-gitops -f example.yaml plan -o plan.json

This will output:

Generating execution plan...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

The following actions will be performed:

Topics: 1 to create, 0 to update, 0 to delete.

+ [TOPIC] test
	+ cleanup.policy: compact


ACLs: 2 to create, 0 to update, 0 to delete.

+ [ACL] my-service-0
	 + resource_name: test
	 + resource_type: TOPIC
	 + resource_pattern: LITERAL
	 + resource_principal: User:test
	 + host: *
	 + operation: READ
	 + permission: ALLOW


+ [ACL] my-service-1
	 + resource_name: *
	 + resource_type: GROUP
	 + resource_pattern: LITERAL
	 + resource_principal: User:test
	 + host: *
	 + operation: READ
	 + permission: ALLOW


Plan: 3 to create, 0 to update, 0 to delete.

We can see, as we defined in our file, that we will create the new topic as well as the two ACLs.

Apply

Run the apply:

kafka-gitops -f example.yaml apply -p plan.json

This will output something like this:

Executing apply...

Applying: [CREATE]

+ [TOPIC] test
	+ cleanup.policy: compact


Successfully applied.

Applying: [CREATE]

+ [ACL] my-service-0
	 + resource_name: test
	 + resource_type: TOPIC
	 + resource_pattern: LITERAL
	 + resource_principal: User:test
	 + host: *
	 + operation: READ
	 + permission: ALLOW


Successfully applied.

Applying: [CREATE]

+ [ACL] my-service-1
	 + resource_name: *
	 + resource_type: GROUP
	 + resource_pattern: LITERAL
	 + resource_principal: User:test
	 + host: *
	 + operation: READ
	 + permission: ALLOW


Successfully applied.

[SUCCESS] Apply complete! Resources: 3 created, 0 updated, 0 deleted.

Verify

Run the plan command again, and we should see that there are no needed changes.

kafka-gitops -f example.yaml plan

This will output:

Generating execution plan...

[SUCCESS] There are no necessary changes; the actual state matches the desired state.

Deletion

Awesome, we've now created things via kafka-gitops. It will also delete topics/ACLs that are not defined within the state file.

Create a properties file so we are able to be authenticated to create topics:

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="test" password="test-secret";
sasl.mechanism=PLAIN
security.protocol=SASL_PLAINTEXT

For example, manually add a topic like this:

kafka-topics --command-config admin.properties --bootstrap-server localhost:9092 \
  --create --topic delete-me --partitions 6 --replication-factor 1

Additionally, manually add an ACL like this:

kafka-acls --command-config admin.properties --bootstrap-server localhost:9092  --add \
  --allow-principal User:test --operation WRITE --topic delete-me

Plan

Now, re-run your plan against the cluster without updating your state file.

kafka-gitops -f example.yaml plan -o plan.json

Output will look like:

Generating execution plan...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - delete

The following actions will be performed:

Topics: 0 to create, 0 to update, 1 to delete.

- [TOPIC] delete-me


ACLs: 0 to create, 0 to update, 1 to delete.

- [ACL] Unnamed ACL
	 - resource_name: test
	 - resource_type: TOPIC
	 - resource_pattern: LITERAL
	 - resource_principal: User:test
	 - host: *
	 - operation: WRITE
	 - permission: ALLOW


Plan: 0 to create, 0 to update, 2 to delete.

Conclusion

As you can see, the new plan lets us know the manually created topic and ACL will be removed when applied. You can then run the apply command from earlier to execute the plan.

I hope that helps!

Hi devshawn,

We have existing kafka cluster that we run on production and would like to automate kafka topic/acl using kafka-gitops. We have topics that are auto-generated based on auto-scale and would like to know if its possible to blacklist based on regular expression or some pattern. Is this currently doable?

Thanks in advance!