LogoLogo
CloudDiscordGitHub
  • 👉Getting Started
    • Introduction
    • Quick start
    • Learn by example
    • Case studies
    • How to contribute?
  • ⭐Memphis Broker
    • Architecture
    • Key concepts
      • Message broker
      • Station
      • Producer API
      • Consumer API
      • Consumer Group
      • Storage and Redundancy
      • Security/Authentication
      • Scaling
      • Ordering
      • Dead-letter Station (DLS)
      • Delayed messages
      • Data exchange
      • Idempotency (Duplicate processing)
      • Failover Scenarios
      • Troubleshooting process
      • Connectors
    • Best practices
      • Producer optimization
      • Compression
    • Memphis configuration
    • Comparisons
      • NATS Jetstream vs Memphis
      • RabbitMQ vs Memphis
      • AWS SQS vs Memphis
      • Apache Kafka vs Memphis
      • Apache Pulsar vs Memphis
      • ZeroMQ vs Memphis
      • Apache NiFi vs Memphis
    • Privacy Policy
  • ⭐Memphis Schemaverse
    • Overview
    • Getting started
      • Management
      • Produce/Consume
        • Protobuf
        • JSON Schema
        • GraphQL
        • Avro
    • Comparison
    • KB
  • 📦Open-Source Installation
    • Kubernetes
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
      • Terraform
        • Deploy on AWS
        • Deploy on GCP
        • Deploy on DigitalOcean
      • Guides
        • Deploy/Upgrade Memphis utilizing predefined secrets
        • Monitoring/Alerts Recommendations
        • Production Best Practices
        • NGINX Ingress Controller and Cloud-Agnostic Memphis Deployments
        • Migrate Memphis storage between storageClass's
        • Expanding Memphis Disk Storage
        • Scale-out Memphis cluster
        • TLS - Deploy Memphis with TLS Connection to Metadata Frontend
        • TLS - Memphis TLS websocket configuration
        • TLS - Securing Memphis Client with TLS
        • Installing Memphis with an External Metadata Database
    • Docker
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
    • Open-source Support
  • Client Libraries
    • REST (Webhook)
    • Node.js / TypeScript / NestJS
    • Go
    • Python
    • Kotlin (Community)
    • .NET
    • Java
    • Rust (Community)
    • NATS
    • Scala
  • 🔌Integrations Center
    • Index
    • Processing
      • Zapier
    • Change data Capture (CDC)
      • Debezium
    • Monitoring
      • Datadog
      • Grafana
    • Notifications
      • Slack
    • Storage tiering
      • S3-Compatible Object Storage
    • Source code
      • GitHub
    • Other platforms
      • Argo
  • 🗒️Release notes
    • KB
    • Releases
      • v1.4.3 - latest/stable
      • v1.4.2
      • v1.4.1
      • v1.4.0
      • v1.3.1
      • v1.3.0
      • v1.2.0
      • v1.1.1
      • v1.1.0
      • v1.0.3
      • v1.0.2
      • v1.0.1
      • V1.0.0 - GA
      • v0.4.5 - beta
      • v0.4.4 - beta
      • v0.4.3 - beta
      • v0.4.2 - beta
      • v0.4.1 - beta
      • v0.4.0 - beta
      • v0.3.6 - beta
      • v0.3.5 - beta
      • v0.3.0 - beta
      • v0.2.2 - beta
      • v0.2.1 - beta
      • v0.2.0 - beta
      • v0.1.0 - beta
Powered by GitBook
LogoLogo

Legal

  • Terms of Service
  • Privacy Policy

All rights reserved to Memphis.dev 2023

On this page
  • Introduction
  • Initial state
  • Scenarios
  • 1. Memphis brokers
  • 2. Kubernetes Workers
  • 3. Kubernetes volumes (PVC)

Was this helpful?

  1. Memphis Broker
  2. Key concepts

Failover Scenarios

Last updated 2 years ago

Was this helpful?

Introduction

This section describes the exact behavior of Memphis in different failure scenarios based on three parameters:

  • Ability to read

  • Ability to write

  • Potential data loss

This section also aims to educate and explain Memphis's cluster mode's internals so its users can make design decisions that suit their workloads perfectly.

Initial state

The test environment is based on a station with three replicas (mirrors) over three memphis brokers in cluster mode.

(*) The small circle within the brokers emphasizes if they are a

Scenarios

1. Memphis brokers

Single broker failure, leader or follower. Produce / Consume is working usually. No data loss.

Two out of three replicas/brokers are down. Produce/Consume will be stopped until at least one replica/follower is available. No data loss.

The entire cluster is down.

Produce/Consume will be stopped until at least one leader and replica/follower are available. No data loss.

2. Kubernetes Workers

The kubernetes worker that holds one of the "followers" is down. The cluster will wait for the broker to be "Ready." Produce / Consume is working usually. No data loss.

The kubernetes worker that holds the "leader" is down. The "leader" role has been taken over by broker 2. Producers / Consumers will might require a reconnect. No data loss.

The kubernetes workers that hold the "followers"/"Leader and a follower" are down. No quorum for stream. No cluster leader. The station is temporarily unavailable. Produce/Consume will be stopped until at least one replica/follower is available. No data loss.

The Kubernetes cluster is down; therefore, the Memphis cluster is down.

Produce/Consume will be stopped until at least one leader and replica/follower are available. No data loss.

3. Kubernetes volumes (PVC)

One of the K8S worker's PVs owned by one of the followers is down.

The affected broker and PV require a manual restart.

The leader's volume is down. Data may not be available during the new leader election process. Producing messages is suspended until a new leader is elected. The affected broker and PV require a manual restart.

Both followers' volumes are down. No quorum for stream. No cluster leader. The station is temporarily unavailable. Produce/Consume will be stopped until at least one replica/follower is available. No data loss.

Both leader and follower volumes are down. No quorum for stream. No cluster leader. The station is temporarily unavailable. Produce/Consume will be stopped until at least one replica/follower is available. No data loss.

All volumes are down; therefore, the Memphis cluster is down.

Produce/Consume will be stopped until at least one leader and replica/follower are available. No data loss.

⭐
Broker 2 is down
Broker 3 is down
Broker (Leader) 1 is down
"Leader" or "Follower."