Yuval Lifshitz yuvalif

existing functionality

frontend requests tracing on the RGW
OSD traces
jaeger orchestration via cephadm
multipart upload tracing when the process is done across multiple RGWs
end2end (RGW<->OSD) tracing of PUT object operations
conditional tracing on the RGW using Lua scripting

work in progress

ceph/ceph#59365

start a single zone cluster with a realm:

OSD=1 MON=1 MDS=0 MGR=0 ../src/test/rgw/test-rgw-multisite.sh 1

create a persistent topic with a bucket and notifications:

aws --region=zg1 --endpoint-url http://localhost:8001 sns create-topic --name=fishtopic --attributes='{"push-endpoint": "kafka://localhost", "persistent": "true"}'
aws --region=zg1 --endpoint-url http://localhost:8001 s3 mb s3://fish
aws --region=zg1 --endpoint-url http://localhost:8001 s3api put-bucket-notification-configuration --bucket fish --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::fishtopic", "Events": []}]}'

based on the data from here:

sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/j/jericho-html-3.3-30.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-util-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/x/xmlgraphics-commons-2.9-3.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-css-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/releases/40/Everything/x86_64/os/Packages/b/batik-1.14-13.fc40.noarch.rpm
sudo dnf install http://rpmfind.net/linux/fedora/linux/updates/testing/40/Everything/x86_64/Packages/d/ditaa-0.10-24.fc40.noarch.rpm

pacific

get tag range:

git tag -l "v16*"

list of related commits (needs some cleanup):

git log  --oneline v16.0.0...v16.2.15 --grep="amqp|kafka|pubsub|notif" --no-merges --extended-regexp

quincy

backtrace:

#0  0x0000561fd85bf9a4 in rgw::notify::Manager::tokens_waiter::token::~token (this=0x7f625e4e8bf8, __in_chrg=<optimized out>) at /home/yuvalif/ceph3/src/rgw/driver/rados/rgw_notify.cc:199
#1  0x0000561fd85c9d75 in rgw::notify::Manager::process_queue(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::any_io_
executor> >)::{lambda(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::any_io_executor> >)#7}::operator()(spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::any_io_executor> >)
 const (__closure=__closure@entry=0x561fdf2abc18, yield=...) at /home/yuvalif/ceph3/src/rgw/driver/rados/rgw_notify.cc:457                                                                                                                    
#2  0x0000561fd85ca050 in spawn::detail::spawn_helper<boost::asio::executor_binder<void (*)(), boost::as

run vstart cluster
create the parts:

$ head -c 12M </dev/urandom > myfile1
$ head -c 12M </dev/urandom > myfile2

create the bucket:

$ aws s3api create-bucket --bucket aws-bkt5 --endpoint-url http://localhost:8000

generic setup

git config --global user.name "My Name"
git config --global user.email "my@email"
git config --global credential.helper cache
git config --global credential.helper "cache --timeout=86400"
git config --global diff.tool vimdiff
git config --global difftool.prompt false
git config --global merge.tool vimdiff
git config --global mergetool.prompt false

Tidy Up Song

Background

Static analysis is a powerful tool when searching for code issues. It does, however, have its costs. Careful tuning is required so that the developers are not overwhelmed with "false positive" detections, while on the other hand - critical errors are not missed. It also requires simple integration into the development flow, so that new issues are continuously detected and addressed.

In this project we use clang-tidy for automated static analysis

	# bucket notification parser according to the "pacific" formatting:
	# https://github.com/ceph/ceph/blob/pacific/src/rgw/rgw_pubsub.h
	# Note: JSON formatting does not match the one define at:
	# https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html
	# but includes all data except: metadata, tags and opaque data

	import struct
	import sys

	def prase_string(buff, offset):

	from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
	import threading
	from multiprocessing import Process
	import random
	import json

	class HTTPPostHandler(BaseHTTPRequestHandler):
	"""HTTP POST hanler class storing the received events in its http server"""
	def do_POST(self):
	"""implementation of POST handler"""