Failover Scenarios
Last updated
Last updated
This section describes the exact behavior of Memphis in different failure scenarios based on three parameters:
Ability to read
Ability to write
Potential data loss
This section also aims to educate and explain Memphis's cluster mode's internals so its users can make design decisions that suit their workloads perfectly.
The test environment is based on a station with three replicas (mirrors) over three memphis brokers in cluster mode.
(*) The small circle within the brokers emphasizes if they are a "Leader" or "Follower."
Single broker failure, leader or follower. Produce / Consume is working usually. No data loss.
Two out of three replicas/brokers are down. Produce/Consume will be stopped until at least one replica/follower is available. No data loss.
The entire cluster is down.
Produce/Consume will be stopped until at least one leader and replica/follower are available. No data loss.
The kubernetes worker that holds one of the "followers" is down. The cluster will wait for the broker to be "Ready." Produce / Consume is working usually. No data loss.
The kubernetes worker that holds the "leader" is down. The "leader" role has been taken over by broker 2. Producers / Consumers will might require a reconnect. No data loss.
The kubernetes workers that hold the "followers"/"Leader and a follower" are down. No quorum for stream. No cluster leader. The station is temporarily unavailable. Produce/Consume will be stopped until at least one replica/follower is available. No data loss.
The Kubernetes cluster is down; therefore, the Memphis cluster is down.
Produce/Consume will be stopped until at least one leader and replica/follower are available. No data loss.
One of the K8S worker's PVs owned by one of the followers is down.
The affected broker and PV require a manual restart.
The leader's volume is down. Data may not be available during the new leader election process. Producing messages is suspended until a new leader is elected. The affected broker and PV require a manual restart.
Both followers' volumes are down. No quorum for stream. No cluster leader. The station is temporarily unavailable. Produce/Consume will be stopped until at least one replica/follower is available. No data loss.
Both leader and follower volumes are down. No quorum for stream. No cluster leader. The station is temporarily unavailable. Produce/Consume will be stopped until at least one replica/follower is available. No data loss.
All volumes are down; therefore, the Memphis cluster is down.
Produce/Consume will be stopped until at least one leader and replica/follower are available. No data loss.
