rgw-kafka-security-gsoc-26.md

Kafka Security

Background

Bucket notification integration with Kafka is a very useful feature in the RGW. However, some security features needed for such integrations are missing. so, in this project we will try to make bucket notifications over kafka more secure. The following features are missing:

• GSSAPI
• OAUTHBEARER
• mTLS
• passing in CA without a file (useful for rook integration)

The main challenge in the above would be in automating the tests, so they could easily run locally, As a stretch goal, we should make sure these tests can also run in teuthology.

Evaluation Stage

Step 1 - Build Ceph and Run Bucket Notifications Tests

First would be to have a Linux based development environment, as a minimum you would need a 4 CPU machine, with 8G RAM and 50GB disk. Unless you already have a Linux distro you like, I would recommend choosing from:

• Fedora (42/43) - my favorite!
• Ubuntu (24.04 LTS)
• WSL (Windows Subsystem for Linux), though it would probably take much longer...
• RHEL9/Centos9
• Other Linux distros - try at your own risk :-)

Once you have that up and running, you should clone the Ceph repo from github (https://github.com/ceph/ceph). If you don_t know what github and git are, this is the right time to close these gaps :-) And yes, you should have a github account, so you can later share your work on the project.

Install any missing system dependencies use:

./install-deps.sh

Note that the first build may take a long time, so the following cmake parameter could be used to minimize the build time. With a fresh ceph clone use the following:

./do_cmake.sh -DBOOST_J=$(nproc) -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DWITH_MGR_DASHBOARD_FRONTEND=OFF \
  -DWITH_DPDK=OFF -DWITH_SPDK=OFF -DWITH_SEASTAR=OFF -DWITH_CEPHFS=OFF -DWITH_RBD=OFF -DWITH_KRBD=OFF -DWITH_CCACHE=OFF -Gninja

Then invoke the build process (using ninja) from within the build directory (created by do_cmake.sh). Assuming the build was completed successfully, you can run the unit tests (see: https://github.com/ceph/ceph#running-unit-tests).

Now you are ready to run the ceph processes, as explained here: https://github.com/ceph/ceph#running-a-test-cluster You probably would also like to check the developer guide (https://docs.ceph.com/docs/master/dev/developer_guide/) and learn more on how to build Ceph and run it locally (https://docs.ceph.com/docs/master/dev/quick_guide/).

Ceph's bucket notification documentation:

• https://docs.ceph.com/en/latest/radosgw/notifications/
• notification as part of the bucket operations API: https://docs.ceph.com/en/latest/radosgw/s3/bucketops/#create-notification
• S3 compatibility: https://docs.ceph.com/en/latest/radosgw/s3-notification-compatibility/

Run bucket notification tests for persistent notifications using an HTTP endpoint:

• start the vtsart cluster:

$ MON=1 OSD=1 MDS=0 MGR=0 RGW=1 ../src/vstart.sh -n -d

• on a separate terminal start an HTTP endpoint:

$ wget https://gist.githubusercontent.com/mdonkers/63e115cc0c79b4f6b8b3a6b797e485c7/raw/a6a1d090ac8549dac8f2bd607bd64925de997d40/server.py
$ python server.py 10900

• install the awc cli tool
• configure the tool according to the access and secret keys showing in the output of the vstart.sh command
• set the region to default
• create a persistent topic pointing to the above HTTP endpoint:

$ aws --endpoint-url http://localhost:8000 sns create-topic --name=fishtopic \
  --attributes='{"push-endpoint": "http://localhost:10900", "persistent": "true"}'

• create a bucket:

$ aws --endpoint-url http://localhost:8000 s3 mb s3://fish

• create a notification on that bucket, pointing to the above topic:

$ aws --endpoint-url http://localhost:8000 s3api put-bucket-notification-configuration  --bucket fish \
  --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::fishtopic", "Events": []}]}'

Leaving the event list empty is equivalent to setting it to ["s3:ObjectCreated:*", "s3:ObjectRemoved:*"]

• create a file, and upload it:

$ head -c 512 </dev/urandom > myfile
$ aws --endpoint-url http://localhost:8000 s3 cp myfile s3://fish

• on the HTTP terminal, see the JSON output of the notifications

Step 2 - Run Kafka Tests

In step 1, we tested bucket notifications against a local HTTP server. But the real goal is to test it against a kafka broker. You also need to install Kafka which can be downloaded from: https://kafka.apache.org/downloads

Then edit the Kafka server properties file (/path/to/kafka/config/server.properties) to have the following line:

listeners=PLAINTEXT://localhost:9092

After following the above steps, start the Zookeeper and Kafka services (in separate terminals): For starting Zookeeper service run:

bin/zookeeper-server-start.sh config/zookeeper.properties

and then start the Kafka service:

bin/kafka-server-start.sh config/server.properties

After running vstart.sh, Zookeeper, and Kafka services you're ready to create a topic pointing at the kafka broker:

$ aws --endpoint-url http://localhost:8000 sns create-topic --name=kafkatopic \
  --attributes='{"push-endpoint": "kafka://localhost", "persistent": "true"}'

Unlike in the http case, where the server was also the consumer, in case of kafka the broker just routes the messages. So, in a separate terminal, you would need to run a kafka consumer:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafkatopic

Then, associate the kafka topic to the bucket:

$ aws --endpoint-url http://localhost:8000 s3api put-bucket-notification-configuration  --bucket fish \
  --notification-configuration='{"TopicConfigurations": [{"Id": "notif2", "TopicArn": "arn:aws:sns:default::kafkatopic", "Events": []}]}'

And upload objects to the bucket. The notifications should appear in the kafka consumer terminal

Step 3 - Kafka over mTLS

You will first need to build a custom version of ceph based on this PR: ceph/ceph#61572 Then setup a new topic that uses ssl with CA location and CERT location:

$ aws --endpoint-url http://localhost:8000 sns create-topic --name=mtlstopic \
  --attributes='{"push-endpoint": "kafka://localhost", "persistent": "true", "use-ssl": "true", "ca-location": "<path>", "cert-location":, "<path>"}'

Generate the certificates (you can look at: https://github.com/ceph/ceph/tree/main/src/test/rgw/bucket_notification#kafka-security-tests) Then configure the kafka broker for mTLS, restart the zookeeper and broker, and repeat the test from step 2, only this time with "mtlstopic".

note that the kafka consumer does not need to connect over SSL

Hi @yuvalif, I have sent you a mail with the gist of my work documented and associated screenshots. Please review the same, thanks!