multisite-reshard-test.md

setup

start:

MON=1 OSD=1 MDS=0 MGR=0 ../src/test/rgw/test-rgw-multisite.sh 2 --rgw_max_objs_per_shard=50 \
    --rgw_reshard_thread_interval=60 --rgw_user_quota_bucket_sync_interval=90 \
    --rgw_data_notify_interval_msec=0 --rgw_sync_log_trim_interval=0

• rgw_max_objs_per_shard: make sure reshard is happening often
• rgw_reshard_thread_interval: make sure reshard happens more than once within the time of the test (200s)
• rgw_data_notify_interval_msec: disable out-of-band sync triggers (so they won't hide any issues)
• rgw_sync_log_trim_interval: disable log trimming (for debugging and to make sure that the problems are not caused by trimming)

verify the number was set, and that dynamic reshard is enabled:

bin/ceph -c run/c2/ceph.conf daemon run/c2/out/radosgw.8002.asok config get rgw_max_objs_per_shard
bin/ceph -c run/c2/ceph.conf daemon run/c2/out/radosgw.8002.asok config get rgw_dynamic_resharding
bin/ceph -c run/c2/ceph.conf daemon run/c2/out/radosgw.8002.asok config get rgw_reshard_thread_interval
bin/ceph -c run/c2/ceph.conf daemon run/c2/out/radosgw.8002.asok config get rgw_user_quota_bucket_sync_interval
bin/ceph -c run/c2/ceph.conf daemon run/c2/out/radosgw.8002.asok config get rgw_data_notify_interval_msec
bin/ceph -c run/c2/ceph.conf daemon run/c2/out/radosgw.8002.asok config get rgw_sync_log_trim_interval

test

• install hsbench
• create the bucket:

hsbench -a 1234567890 -s pencil -u http://localhost:8001 -bp my-bucket -b 1 -r zg1 -m i

• run 2 clients simultanously, each pointing to a different zone. each client run with 4 threads and for 200 seconds.

hsbench -a 1234567890 -s pencil -u http://localhost:8001 -bp my-bucket -t 4 -b 1 -r zg1 -m p -z 4K -d 200 -op obj1

hsbench -a 1234567890 -s pencil -u http://localhost:8002 -bp my-bucket -t 4 -b 1 -r zg1 -m p -z 4K -d 200 -op obj2

debug

some useful commands:

bin/radosgw-admin -c run/c1/ceph.conf bucket sync status --bucket my-bucket000000000000
bin/radosgw-admin -c run/c2/ceph.conf bucket sync status --bucket my-bucket000000000000

bin/radosgw-admin -c run/c1/ceph.conf bucket layout --bucket my-bucket000000000000
bin/radosgw-admin -c run/c2/ceph.conf bucket layout --bucket my-bucket000000000000

AWS_ACCESS_KEY_ID=1234567890 AWS_SECRET_ACCESS_KEY=pencil aws --endpoint-url http://localhost:8001 s3 ls s3://my-bucket000000000000 > rgw1
AWS_ACCESS_KEY_ID=1234567890 AWS_SECRET_ACCESS_KEY=pencil aws --endpoint-url http://localhost:8002 s3 ls s3://my-bucket000000000000 > rgw2
diff rgw1 rgw2