test-sync-notifications-part2.md

this is a followup on: https://gist.github.com/yuvalif/d50bcc3d27121732ffbbafe7b7146112 for an RGW of version "squid" or above, where bucket notifications and topics are synced between zones

• deploy a zonegroup with 2 zones:

MON=1 OSD=1 MDS=0 MGR=0 ../src/test/rgw/test-rgw-multisite.sh 2

• export credentials:

export AWS_ACCESS_KEY_ID=1234567890
export AWS_SECRET_ACCESS_KEY=pencil

• download a python server:

wget https://gist.githubusercontent.com/mdonkers/63e115cc0c79b4f6b8b3a6b797e485c7/raw/a6a1d090ac8549dac8f2bd607bd64925de997d40/server.py

• run an HTTP receiver "per zone". in one terminal run:

python server.py 10900

• and in another terminal run:

python server.py 11900

• on zone 1 create a topic for the zone 1 HTTP receiver:

AWS_DEFAULT_REGION=zg1 aws --endpoint-url http://localhost:8101 sns create-topic --name=fishtopic1 --attributes='{"push-endpoint": "http://localhost:10900"}'

• on zone 2 create a topic for the zone 2 HTTP receiver:

AWS_DEFAULT_REGION=zg2 aws --endpoint-url http://localhost:8201 sns create-topic --name=fishtopic2 --attributes='{"push-endpoint": "http://localhost:11900"}'

it does not really matter on which zone the topic is created, since the topic information is synced however, the above is what we expect from the application that sets the topics

• create a bucket:

aws --endpoint-url http://localhost:8101 s3 mb s3://mybucket

• create a sync notification for both topics on the above bucket:

aws --endpoint-url http://localhost:8101 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic1", "Events": ["s3:ObjectSynced:*"]}]}'
aws --endpoint-url http://localhost:8201 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic2", "Events": ["s3:ObjectSynced:*"]}]}'

• upload a file to zone 1:

head -c 1M < /dev/urandom > myfile-zg1-1
aws --endpoint-url http://localhost:8101 s3 cp myfile-zg1-1 s3://mybucket

• once the object is synced, the RGW in zone 2 will send updates to both HTTP receivers

use the "x-amz-id-2" field in the notification to verify the originating RGW

• upload a file to zone 2:

head -c 1M < /dev/urandom > myfile-zg1-2
aws --endpoint-url http://localhost:8201 s3 cp myfile-zg1-2 s3://mybucket

• same behavior is observed
• set negative zone filter on the notification. each zone exclude itself:

aws --endpoint-url http://localhost:8101 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic1", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-1", "Type": "OUT"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

aws --endpoint-url http://localhost:8201 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic2", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-2", "Type": "OUT"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

• upload files to zone 1 and zone 2, and verify that:
  • when uploading a file to zone 1, only the zone 2 HTTP receiver get a notification from zone 2 RGW
  • when uploading a file to zone 2, only the zone 1 HTTP receiver get a notification from zone 1 RGW

Hi @yuvalif, the last two aws command to add the zone based notification should be as follows:

aws --endpoint-url http://localhost:8101 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic1", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-1", "Type": "OUT"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

aws --endpoint-url http://localhost:8201 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic2", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-2", "Type": "OUT"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

Hi @yuvalif, the last two aws command to add the zone based notification should be as follows:

aws --endpoint-url http://localhost:8101 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic1", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-1", "Type": "OUT"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

aws --endpoint-url http://localhost:8201 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic2", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-2", "Type": "OUT"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

thanks, i'll fix above

@yuvalif, after modifying the notifications to honor the zone filter as you have mentioned, shouldn't the behavior be like the following ?

you upload a file 1 to zone 1 via rgw 1 (on port 8101) => object gets synced on zone group 2 => rgw 2 registers this sync event => and sends a notification only to topic 1

in a similar way, you upload a file to zone 2 via rgw 2 (on port 8201) => object gets sync on zone group 1 => rgw 1 registers this sync event => and sends a notification to topic 2

Hi @yuvalif, to address the issue brought up here: https://tracker.ceph.com/attachments/7198.

We need to define the notifications as follows:

aws --endpoint-url http://localhost:8101 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic1", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-2", "Type": "OUT"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

aws --endpoint-url http://localhost:8201 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic2", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-1", "Type": "OUT"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

Please notice the change in the zone name for each notification.

From what I can understand the expectation is as follows:

We need to ensure that sync notifications don't go to the zone where the object was created -> this can be achieved with the definition I have given.

The exisiting definition would behave in the way I mentioned in the previous comment.

{"FilterRules": [{"Name": "zg1-2", "Type": "OUT"}]}}
the problem is that the application defining topic1 in ints zone does not know the other zones, and cannot filter them out (e.g. if we have 10 zones).
so, seems like we just need to set our own zone with an "IN" filter?

looking again in the original discussions:
https://tracker.ceph.com/issues/68788
https://tracker.ceph.com/issues/68789

• the first idea was that the application will send the zone as an object attribute, and we would use the negative filter on the attribute (attribute filtering was already supported)
• this was an issue, because the application does not really know what the zone is
• so, we decidced to allow zone filtering (which an admin sets) without the need for application level conf. in this case, as you mentioend above we only need positive filtering

Yes @yuvalif, setting the notification to only honor object sync on respective zones with an IN makes more logical sense.

We only expect topics to receive events registered on that particular zone.

Yes @yuvalif, setting the notification to only honor object sync on respective zones with an IN makes more logical sense.

We only expect topics to receive events registered on that particular zone.

sounds good. please test. if it is working I'll update accordingly

setting the filters to IN is working as expected, @yuvalif. the notifications to update on the gist are as follows.

aws --endpoint-url http://localhost:8101 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic1", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-1", "Type": "IN"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'

aws --endpoint-url http://localhost:8201 s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:zg1::fishtopic2", "Filter": {"Zones": {"FilterRules": [{"Name": "zg1-2", "Type": "IN"}]}}, "Events": ["s3:ObjectSynced:*"]}]}'