Ceph File System snapshot mirroring
You can replicate a Ceph File System (CephFS) to a remote Ceph File System on another IBM Storage Ceph cluster. A Ceph File System supports asynchronous replication of snapshots directories.
To use CephFS mirrors, both the source and the target storage clusters must be running IBM Storage Ceph 7.0 or later.
The Ceph File System (CephFS) supports asynchronous replication of snapshots to a remote CephFS on another IBM Storage Ceph cluster. Snapshot synchronization copies snapshot data to a remote Ceph File System, and creates a new snapshot on the remote target with the same name. You can configure specific directories for snapshot synchronization.
cephfs-mirror). This snapshot data is synchronized by doing a bulk copy to the
remote CephFS. The chosen order of synchronizing snapshot pairs is based on the creation using the
snap-id.The CephFS snapshot mirroring includes features, for example snapshot incarnation or high
availability. These can be managed through Ceph Manager mirroring module, which is
the recommended control interface.
Ceph Manager Module and interfaces
The Ceph Manager mirroring module is disabled by default. It provides interfaces
for managing mirroring of directory snapshots. Ceph Manager interfaces are mostly wrappers around
monitor commands for managing CephFS mirroring. They are the recommended control interface.
The Ceph Manager mirroring module is implemented as a Ceph Manager plugin. It is
responsible for assigning directories to the cephfs-mirror daemons for
synchronization.
mirroring module also provides a family of commands to control
mirroring of directory snapshots. The mirroring module does not manage the
cephfs-mirror daemons. The stopping, starting, restarting, and enabling of the
cephfs-mirror daemons is controlled by systemctl, but managed by
cephadm.fs snapshot mirror
prefix as compared to the monitor commands with the fs mirror prefix. Assure that
you are using the module command prefix to control the mirroring of directory snapshots.Snapshot incarnation
A snapshot might be deleted and recreated with the same name and different content. The user could synchronize an "old" snapshot earlier and recreate the snapshot when the mirroring was disabled. Using snapshot names to infer the point-of-continuation would result in the “new” snapshot, an incarnation, never getting picked up for synchronization.
Snapshots on the secondary file system store the snap-id of the snapshot it was
synchronized from. This metadata is stored in the SnapInfo structure on the Ceph
Metadata Server.
High availability
You can deploy multiple cephfs-mirror daemons on two or more nodes to achieve
concurrency in synchronization of directory snapshots. When cephfs-mirror daemons
are deployed or terminated, the Ceph Manager mirroring module discovers the
modified set of cephfs-mirror daemons and rebalances the directory assignment
amongst the new set thus providing high availability.
cephfs-mirror daemons share the synchronization load using a simple M/N policy,
where M is the number of directories and N is the number of cephfs-mirror
daemons.
Re-addition of Ceph File System mirror peers
When re-adding or reassigning a peer to a CephFS in another cluster, ensure that all mirror daemons have stopped synchronization to the peer. You can verify this with the fs mirror status command. The Peer UUID should not show up in the command output.
Purge synchronized directories from the peer before re-adding it to another CephFS, especially those directories which might exist in the new primary file system. This is not required if you are re-adding a peer to the same primary file system it was earlier synchronized from.
For more information about the fs mirror status command, see Viewing the mirror status for a Ceph File System.