Cinder RBD Replication Guide

rbd-replication.md

Replication Notes

Assumption: You have two clusters, access to both, and a pool that exists in both clustsers and you wish to replicate some or all images in that pool to the other cluster.

Mirroring in both directions is required for Cinder to properly implement failover and failback.

Make sure you have the rbd-mirror package installed.

Run the Mirror Daemon

I assume you already have the MONs and OSDs deployed, running, and in a healthy state. The mirror daemon doesn't technically have to run on the same node as the MON or OSD, but I'm developing with single-node clusters so this is my setup.

On the primary cluster

$ rbd-mirror --cluster primary --setuser ceph --setgroup ceph

On the secondary cluster

$ rbd-mirror --cluster=primary --setuser ceph --setgroup ceph

Enable Mirroring (per-image on a specific pool)

This will enable image-mode mirroring on a pool

$ rbd --cluster primary   mirror pool enable volumes image
$ rbd --cluster secondary mirror pool enable volumes image

$ rbd --cluster primary mirror pool info volumes
Mode: image
Peers: none

$ rbd --cluster secondary mirror pool info volumes
Mode: image
Peers: none

Configure Mirror Peers

$ rbd --cluster primary mirror pool peer add volumes client.admin@secondary
506b17d9-5157-4c6a-a63e-94617fb34ea6

$ rbd --cluster secondary mirror pool peer add volumes client.admin@primary
b2853704-2536-4cb3-860d-51622dcaf353

Verify Mirror Configuration

$ rbd --cluster primary mirror pool info volumes
Mode: image
Peers:
  UUID                                 NAME      CLIENT
  506b17d9-5157-4c6a-a63e-94617fb34ea6 secondary client.admin

$ rbd --cluster secondary mirror pool info volumes
Mode: image
Peers:
  UUID                                 NAME    CLIENT
  b2853704-2536-4cb3-860d-51622dcaf353 primary client.admin

From here, the cinder driver will take care of the rest. It will enable per-image replication for volumes created normally, from an existing clone, and from retyping.

Manual Image Mirroring

$ rbd --cluster primary create volumes/test --size 1M
$ rbd info volumes/test
rbd image 'a':
        size 1024 kB in 1 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.3ba2d74b0dc51
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        flags:

$ rbd --cluster primary feature enable volumes/test journaling
$ rbd --cluster primary mirror image enable volumes/test

$ rbd --cluster primary ls -l volumes
NAME  SIZE PARENT FMT PROT LOCK 
a    1024k          2

$ rbd --cluster secondary ls -l volumes
NAME  SIZE PARENT FMT PROT LOCK
a    1024k          2      excl

$ rbd info volumes/test
rbd image 'test':
        size 1024 kB in 1 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.3ba2d74b0dc51
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
        flags: 
        journal: 3ba2d74b0dc51
        mirroring state: enabled
        mirroring global id: aba6f0d2-b044-4ed1-9dcf-d5c9918386a3
        mirroring primary: true

Manual Failover

$ rbd --cluster secondary mirror image promote --force volumes/test

Manual Failback

$ <resync required>

Cinder Configuration

The Ceph backend definition in cinder.conf must be updated to define the secondary cluster:

replication_device = backend_id:secondary,
                     conf:/etc/ceph/secondary.conf
                     user:cinder,
                     pool:volumes

Cinder Type Configuration

The Cinder RBD driver must be told to enable replication for a particular volume. This is done with volume types.

$ cinder type-create REPL
$ cinder type-key    REPL set volume_backend_name=ceph
$ cinder type-key    REPL set replication_enabled='<is> True'

Replicated Volume Creation

$ cinder create --volume-type REPL --name fingers-crossed 1

Cinder Failover

$ cinder failover-host client@ceph

The article is useful. So I want to know how to recovery when disaster on primary site.

Hello.
I set up two clusters ceph (version 12.2.9 luminous). The first cluster has the name of the "primary", the second "secondary". Between the two clusters has configured two-way replication.
Images are created and replicated successfully. I can not configure the Сinder. Help me please.

My settings:

file /etc/cinder/cinder.conf
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
replication_device = backend_id:secondary,conf:/etc/ceph/secondary.conf,user:cinder,pool:volumes
rbd_max_clone_depth = 5
rbd_flatten_volume_from_snapshot = False
rbd_user = cinder
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/primary.conf
volume_backend_name = ceph
rbd_cluster_name = primary
glance_api_version = 2
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
rbd_user = cinder
rbd_secret_uuid = 5e54739d-37cf-49e8-93f5-5489f51d8ef0

$ ls /etc/ceph
primary.client.cinder.keyring primary.client.glance.keyring primary.conf secondary.client.cinder.keyring secondary.client.glance.keyring secondary.conf

$ cinder extra-specs-list
+--------------------------------------+------------+---------------------------------------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+------------+---------------------------------------------------------------------+
| 1ed647e6-6b57-47bd-bc1b-49a7625e0bcd | replicated | {'replication_enabled': ' True', 'volume_backend_name': 'ceph'} |
| 1fef1cf4-3cbc-4f8a-9f06-ef07444c2570 | type2 | {} |
| 709a93bc-b8fb-4b45-8af4-d4422d10fc73 | type1 | {} |
+--------------------------------------+------------+---------------------------------------------------------------------+
$ cinder list
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+
| aa788944-6764-436d-b680-82db5e594d59 | error | fingers-crossed | 1 | replicated | false | |
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+

error in log file /var/log/cinder/scheduler.log
NoValidBackend: No valid backend was found

@BRO36, can you please share the guide for setup two way replication, I'm not able to figure out the how to use custom cluster name in ceph. Thanks in Advance.

@thisismsreddy
What exactly do you fail? Show me what you do I may be able to find a mistake.

@BRO36, it's turned out I am looking at this completely wrong. All recent versions of ceph remove the support for setting up the custom cluster name. :)

Hello.
I set up two clusters ceph (version 12.2.9 luminous). The first cluster has the name of the "primary", the second "secondary". Between the two clusters has configured two-way replication.
Images are created and replicated successfully. I can not configure the Сinder. Help me please.

My settings:

file /etc/cinder/cinder.conf
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
replication_device = backend_id:secondary,conf:/etc/ceph/secondary.conf,user:cinder,pool:volumes
rbd_max_clone_depth = 5
rbd_flatten_volume_from_snapshot = False
rbd_user = cinder
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/primary.conf
volume_backend_name = ceph
rbd_cluster_name = primary
glance_api_version = 2
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
rbd_user = cinder
rbd_secret_uuid = 5e54739d-37cf-49e8-93f5-5489f51d8ef0

$ ls /etc/ceph
primary.client.cinder.keyring primary.client.glance.keyring primary.conf secondary.client.cinder.keyring secondary.client.glance.keyring secondary.conf

$ cinder extra-specs-list
+--------------------------------------+------------+---------------------------------------------------------------------+
| ID | Name | extra_specs |
+--------------------------------------+------------+---------------------------------------------------------------------+
| 1ed647e6-6b57-47bd-bc1b-49a7625e0bcd | replicated | {'replication_enabled': ' True', 'volume_backend_name': 'ceph'} |
| 1fef1cf4-3cbc-4f8a-9f06-ef07444c2570 | type2 | {} |
| 709a93bc-b8fb-4b45-8af4-d4422d10fc73 | type1 | {} |
+--------------------------------------+------------+---------------------------------------------------------------------+
$ cinder list
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+
| aa788944-6764-436d-b680-82db5e594d59 | error | fingers-crossed | 1 | replicated | false | |
+--------------------------------------+--------+-----------------+------+-------------+----------+-------------+

error in log file /var/log/cinder/scheduler.log
NoValidBackend: No valid backend was found

The error may be causing since cinder scheduler is not able to find a suitable backend with the required additional spec (replication_enabled=true)

I think while adding the extra specs to the replicated volume type you give the wrong value.
It should be replication_enabled=' True'
not replication_enabled=' True'

You can enable debug in cinder and check for the scheduler logs while creating a replicated volume.

@arun-vinod
Which mode did you use for pool? Image or pool ? https://docs.ceph.com/docs/master/rbd/rbd-mirroring/#enable-mirroring. I had a problem because i used "pool" mode.