MongoDB: Delayed Secondary Member of a Replica Set and How It Can Be Useful

MongoDB: Delayed Secondary Member of a Replica Set and How It Can Be Useful

In this blog post, we will discuss delayed members of a replica set and how it can be useful. A delayed member of the replica set is similar to like other members of the replica set but its data set mirrors the set delayed state of the data. Let’s say if a delayed node is delayed by one hour and the current time is 11:00, then its data set reflects recent operation till 10:00.

A delayed member copies and applies the operation from the primary’s oplog with a delay. The amount of delay for the delayed member must be greater than or equal to the expected maintenance window duration. The time of delay should be less than the oplog window. A delayed secondary can not acknowledge a write before the configured slaveDelay or secondaryDelaySecs.

Setting the delayed member’s vote to 0 and hidden can improve the cluster performance. Consider if you have primary-secondary-delayed and secondary architecture, and delayed nodes are non-voting with a delay of time 15 minutes. When the write concern is “majority” and a non-delayed member becomes unavailable, then at that time application has to wait for acknowledgment of writes for about 15 minutes. In this case, the commit point will start to lag. The storage engine will start keeping all the changes after the commit point on the disk to retain a durable history. It will increase the I/O writes over time, which as a result impacts the writes and cache pressure.

Requirement for a node to be a delayed secondary member:

• Priority zero: A delayed secondary node must have 0 priority. Its priority should be set to 0 to stop this node from being a primary node. Since it’ll be a delayed member and reflect the delayed state of the data set, it should be prevented to become a primary node.
• Hidden: A delayed node must be hidden. As this node will reflect the delayed state of the data set, the application should always be stopped to see and query the delayed node.
• Oplog: Oplog window’s capacity must be more than the specified amount of delay. 

In sharding, delayed secondary members have limited benefits if the balancer is running. Since delayed members clone the chunk migration with a delay, the state of the delayed member will not be useful for recovering to a previous state of the sharded cluster if migration occurred during the delay window.

One can configure a secondary node as a delayed member secondary by adjusting the members[n].priority value to 0, members[n].hidden value to true, and its members[n].slaveDelay value to the number of seconds to delay. For example, configure a node as delayed as below:

Note: Reconfiguring a replica set could lead to an election, so it is recommended to do the above in the scheduled maintenance window.

How can delayed secondary members of a replica set be useful?

Since delayed secondary members are delayed by some amount of time, it can be useful if one needs it to roll back to an earlier time or it can act as a hot backup to shelter against mistakes like accidentally dropping the entire database/collection. Or, consider, if an application upgrade was done but was unsuccessful or had operator errors. In that case, delayed secondary members can be useful to recover from.

Conclusion

A delayed secondary can be useful in case of accidentally dropping the entire database/collection and that drop has not been applied yet i.e till the specified delay time. You also need to make sure that you have enough oplog windows to allow it to be in sync with the primary. However, for a comprehensive recovery solution, we would recommend having a backup solution. You can check the Percona backup solution, to learn more please visit Percona Backup for MongoDB.

We also encourage you to try our products for MongoDB like Percona Server for MongoDB or Percona Operator for MongoDB.