Yuval Lifshitz yuvalif

RGW tcmalloc Profiling

Background

All daemons in ceph are using tcmalloc as the memory allocator to achieve better performance. In a recent PR the ability to get information on how tcmalloc performs in the RGW was added. In this project, we should use the profiling information from RGW runs to tune the tcmalloc parameters so that would be more suitable for the memory use of the RGW.

Evaluation Stage

Step 1 - Build Ceph and Run Basic Tests

First would be to have a Linux based development environment, as a minimum you would need a 4 CPU machine, with 8G RAM and 50GB disk. Unless you already have a Linux distro you like, I would recommend choosing from:

Kafka Security

Background

Bucket notification integration with Kafka is a very useful feature in the RGW. However, some security features needed for such integrations are missing. so, in this project we will try to make bucket notifications over kafka more secure. The following features are missing:

GSSAPI
OAUTHBEARER
mTLS
passing in CA without a file (useful for rook integration)

The main challenge in the above would be in automating the tests, so they could easily run locally,

radosgw-admin UX and documentation improvements

Background

Currently, documenting radosgw-admin commands is a manual and error-prone process. After implementing a new command, the "usage" string should be updated accordingly in the code, where there could be a mismatch between the actually command and its arguments and what is documented in the usage. After that the man page needs to be updated manually, as well as the admin guide. Any references to this command in other places in our documentation also need to be manually updated.

We would like to solve this program with a more programmatic approach:

Declare command & argument semantics explicitly in code using a cli/args framework that supports auto-generation of context-aware "usage" docs
Investigate how this can then be used to auto-generate the man page, admin guides and any other related documentation (maybe using some python code)
See if we can easily reference these command descriptions in other places in our documentation

Setup

start k8s with kinikube with an extra disk:

minikube start --extra-disks=1 --driver=kvm2

install rook:

kubectl create -f https://raw.githubusercontent.com/rook/rook/refs/heads/master/deploy/examples/crds.yaml
kubectl create -f https://raw.githubusercontent.com/rook/rook/refs/heads/master/deploy/examples/common.yaml
kubectl create -f https://raw.githubusercontent.com/rook/rook/refs/heads/master/deploy/examples/operator.yaml

# code
  - you can and SHOULD use AI when writing code
  - please avoid some of the things that AI likes to do in code:
    - write long, unnecessary comments on self explanatory code
    - use non-ascii characters in comments
    - generate redundant (even though it may be correct) code
    - write repetitive code that could be easily refactored
    - reimplement functionality that can be taken from a library
  - in short, PRs that are AI generated without human guidance tend to be unnecessarily long

Setup

start a vstart cluster with RGW

Alternative 1

Add object locking to all bucket creations via a lua.

upload the following script in prerequest context:

-- enablog object lock on bucket creation

copied from: https://claude.ai/share/e4bed98a-9049-44b3-9aee-173bba941120

When a Kafka producer sets partitions explicitly, there are several important trade-offs to consider:

Pros of Explicit Partition Assignment

Guaranteed Message Ordering: Messages sent to the same partition are guaranteed to maintain their order. This is crucial for use cases where sequence matters, like financial transactions or event sourcing.
Predictable Data Locality: You can ensure related messages always go to the same partition, which helps with consumer processing efficiency and stateful operations.
Load Distribution Control: You have fine-grained control over how messages are distributed across partitions, allowing you to optimize for your specific access patterns.
Deterministic Behavior: Your application's behavior becomes more predictable since you know exactly where each message will land.

machine

 lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda              8:0    0 893.8G  0 disk
└─sda1           8:1    0 893.7G  0 part /
nvme1n1        259:0    0   1.5T  0 disk
nvme0n1        259:1    0   1.5T  0 disk
├─vg_nvme-lv_1 253:0    0  89.4G  0 lvm
├─vg_nvme-lv_2 253:1    0  89.4G  0 lvm

	# channel for rgw send notifications
	apiVersion: messaging.knative.dev/v1
	kind: InMemoryChannel
	metadata:
	name: text-channel
	---
	# subscription for the python-ceph-vectordb app which listens notifications from the channel
	apiVersion: messaging.knative.dev/v1
	kind: Subscription
	metadata: