yuvalif

Warm and Fuzzy

Background

The RGW's frontend is an S3 REST API server, and in this project we would like to use a REST API fuzzer to test the RGW for security issues (and other bugs). Would recommend exploring the Restler tool. Very good intro in this video. Feed it with the AWS S3 OpenAPI spec, and see what happens when we let it connect to the RGW.

Project

Initial (evaluation) Phase

• run Ceph with a radosgw. you can use cephadm to install and run ceph in containers or build it from source and run it a vstart cluster

yuvalif

The More the Merrier

Background

Persistent bucket notifications are a very useful and powerful feature. To learn more about it, you can look at this tech talk and usecase example.

Persistent notifications are usually better that synchronous notification, due to several reasons:

• the queue they are using is, in fact, a RADOS object. This gives the queue the reliability level of RADOS
• they do not add the delay of sending the notification to the broker to the client request round trip time
• they allow for temporary disconnects with the broker or broker restarts without affecting the service
• they have a time and attempts retry mechanism

yuvalif

• follow the instructions from here: https://github.com/yuvalif/cephalocon-24-demo/blob/main/README.md
• however, the cluster-test.yaml file needs 2 modifications
  • don't enable monitoring
  • andable tracing

################################################################################################################
# Define the settings for the rook-ceph cluster with common settings for a small test cluster.
# All nodes with available raw devices will be used for the Ceph cluster. One node is sufficient
# in this example.

yuvalif

Test

this test assumes ceph cluster with RGW is deployed via vstart

• create the "log' bucket:

aws --endpoint-url http://localhost:8000 s3 mb s3://all-logs

Standard Mode

• create a bucket for standard logging:

yuvalif

Phase 0

• draft PR
• initial PR
• initial test PR

Phase 1

• add "flush" REST API call to fix the issue of lazy-commit. use POST /<bucket name>/?logging as the command
• add admin command to get bucket logging info: radosgw-admin bucket logging get
• handle copy correctly:
• in "Journal" mode, we should just see the "PUT" of the new object (existing behavior)

yuvalif

install:

sudo dnf -y install epel-release
curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash
sudo dnf -y install erlang         
curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
sudo dnf -y install rabbitmq-server  

add conf to allow guest when server is not on localhost:

yuvalif

existing functionality

• frontend requests tracing on the RGW
• OSD traces
• jaeger orchestration via cephadm
• multipart upload tracing when the process is done across multiple RGWs
• end2end (RGW<->OSD) tracing of PUT object operations
• conditional tracing on the RGW using Lua scripting

work in progress

• ceph/ceph#59365

yuvalif

start a single zone cluster with a realm:

OSD=1 MON=1 MDS=0 MGR=0 ../src/test/rgw/test-rgw-multisite.sh 1

create a persistent topic with a bucket and notifications:

aws --region=zg1 --endpoint-url http://localhost:8001 sns create-topic --name=fishtopic --attributes='{"push-endpoint": "kafka://localhost", "persistent": "true"}'
aws --region=zg1 --endpoint-url http://localhost:8001 s3 mb s3://fish
aws --region=zg1 --endpoint-url http://localhost:8001 s3api put-bucket-notification-configuration --bucket fish --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::fishtopic", "Events": []}]}'

yuvalif

based on the data from here:

sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/j/jericho-html-3.3-30.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-util-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/x/xmlgraphics-commons-2.9-3.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-css-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/updates/testing/40/Everything/x86_64/Packages/d/ditaa-0.10-24.fc40.noarch.rpm

yuvalif

pacific

get tag range:

git tag -l "v16*"

list of related commits (needs some cleanup):

git log  --oneline v16.0.0...v16.2.15 --grep="amqp|kafka|pubsub|notif" --no-merges --extended-regexp

quincy