LogoLogo
CloudDiscordGitHub
  • 👉Getting Started
    • Introduction
    • Quick start
    • Learn by example
    • Case studies
    • How to contribute?
  • ⭐Memphis Broker
    • Architecture
    • Key concepts
      • Message broker
      • Station
      • Producer API
      • Consumer API
      • Consumer Group
      • Storage and Redundancy
      • Security/Authentication
      • Scaling
      • Ordering
      • Dead-letter Station (DLS)
      • Delayed messages
      • Data exchange
      • Idempotency (Duplicate processing)
      • Failover Scenarios
      • Troubleshooting process
      • Connectors
    • Best practices
      • Producer optimization
      • Compression
    • Memphis configuration
    • Comparisons
      • NATS Jetstream vs Memphis
      • RabbitMQ vs Memphis
      • AWS SQS vs Memphis
      • Apache Kafka vs Memphis
      • Apache Pulsar vs Memphis
      • ZeroMQ vs Memphis
      • Apache NiFi vs Memphis
    • Privacy Policy
  • ⭐Memphis Schemaverse
    • Overview
    • Getting started
      • Management
      • Produce/Consume
        • Protobuf
        • JSON Schema
        • GraphQL
        • Avro
    • Comparison
    • KB
  • 📦Open-Source Installation
    • Kubernetes
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
      • Terraform
        • Deploy on AWS
        • Deploy on GCP
        • Deploy on DigitalOcean
      • Guides
        • Deploy/Upgrade Memphis utilizing predefined secrets
        • Monitoring/Alerts Recommendations
        • Production Best Practices
        • NGINX Ingress Controller and Cloud-Agnostic Memphis Deployments
        • Migrate Memphis storage between storageClass's
        • Expanding Memphis Disk Storage
        • Scale-out Memphis cluster
        • TLS - Deploy Memphis with TLS Connection to Metadata Frontend
        • TLS - Memphis TLS websocket configuration
        • TLS - Securing Memphis Client with TLS
        • Installing Memphis with an External Metadata Database
    • Docker
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
    • Open-source Support
  • Client Libraries
    • REST (Webhook)
    • Node.js / TypeScript / NestJS
    • Go
    • Python
    • Kotlin (Community)
    • .NET
    • Java
    • Rust (Community)
    • NATS
    • Scala
  • 🔌Integrations Center
    • Index
    • Processing
      • Zapier
    • Change data Capture (CDC)
      • Debezium
    • Monitoring
      • Datadog
      • Grafana
    • Notifications
      • Slack
    • Storage tiering
      • S3-Compatible Object Storage
    • Source code
      • GitHub
    • Other platforms
      • Argo
  • 🗒️Release notes
    • KB
    • Releases
      • v1.4.3 - latest/stable
      • v1.4.2
      • v1.4.1
      • v1.4.0
      • v1.3.1
      • v1.3.0
      • v1.2.0
      • v1.1.1
      • v1.1.0
      • v1.0.3
      • v1.0.2
      • v1.0.1
      • V1.0.0 - GA
      • v0.4.5 - beta
      • v0.4.4 - beta
      • v0.4.3 - beta
      • v0.4.2 - beta
      • v0.4.1 - beta
      • v0.4.0 - beta
      • v0.3.6 - beta
      • v0.3.5 - beta
      • v0.3.0 - beta
      • v0.2.2 - beta
      • v0.2.1 - beta
      • v0.2.0 - beta
      • v0.1.0 - beta
Powered by GitBook
LogoLogo

Legal

  • Terms of Service
  • Privacy Policy

All rights reserved to Memphis.dev 2023

On this page
  • Key components
  • Connectivity Diagram
  • Ports list
  • Network architecture diagram
  • Topology and ordering
  • Mirroring and Replications
  • Communication types
  • Deployment sequence
  • Self-hosted requirements
  • Delivery Guarantee

Was this helpful?

  1. Memphis Broker

Architecture

This section describes Memphis' architecture

Last updated 1 year ago

Was this helpful?

Key components

Memphis platform is comprised of three main components:

  1. Memphis. The broker itself which acts as the data storage layer. That is the component that stores and controls the ingested messages and their entire lifecycle management.

  2. Metadata store. Responsible for storing the platform metadata only, such as general information, monitoring, GUI state, and pointers to dead-letter messages. The metadata store uses Postgres.

  3. REST Gateway. Responsible for exposing Memphis management and data ingestion through REST requests.

Connectivity Diagram

Ports list

Name
Port
TCP/UDP
Inter/External
Description

Dashboard/CLI

9000

TCP

External

External port that serve CLI clients and Web UI dashboard

Client connections

6666

TCP

Internal/External

Port for TCP-based client connections with memphis SDKs

REST Gateway

4444

TCP

External

REST gateway endpoint

Websocket

7770

TCP

External

Websocket port

Metrics

8222

TCP

Internal

Memphis monitor port

Cluster connectivity

6222

TCP

Internal

Internal port for connectiovity between brokers in the cluster

Exporter

7777

TCP

Internal

Memphis metrics exporter port for Prometheus

Meta-data

5432

TCP

Internal

Meta-data storage port

Network architecture diagram

The diagram below depicts a full Kubernetes-based deployment.

Topology and ordering

Producers and consumers are the main entities in Memphis. The role of the producers is to produce data into a Memphis station, and the consumers to consume the ingested data or messages.

A single client or application can run multiple producers and consumers, and connect them to various stations. Within the context of a single station, consumer groups encapsulate multiple consumers. Each consumer group will consume all of the stored and unconsumed messages within the station and split the consumed messages within its internal members, similar to thread pool behavior. That means that a single produced message will be consumed by all of the consumer groups, but within the consumer group itself, it will only be read by a single consumer.

Each consumer is bound to a unique consumer group, and cannot be shared across multiple CGs.

The stored messages within the stations are ordered based on First-In-First-Out (Fifo) manner and will be consumed in the same order as they are produced.

Mirroring and Replications

Memphis brokers should run on different nodes to ensure data consistency and zero loss within complete broker’s reboots. To comply with RAFT requirements which are ½ cluster size + 1 an odd number of Memphis brokers shall be deployed. The minimum number of brokers is one, and the next scale would be 3, 5, and so forth.

Communication types

Deployment sequence

Self-hosted requirements

Minimum Requirements (Without high availability)

Resource
Quantity

K8S Nodes

1

CPU

2 CPU

Memory

4GB RAM

Storage

12GB PVC

Recommended Requirements for production (With high availability)

Resource
Minimum Quantity

K8S Nodes

3

CPU

4 CPU

Memory

8GB RAM

Storage

12GB PVC Per node

Requirements (No HA)

Resource
Quantity

OS

Mac / Windows / Linux

CPU

1 CPU

Memory

4GB

Storage

6GB

Delivery Guarantee

  • At least once

This is achieved by the combination of published messages being persisted to the station as well as the consumer tracking delivery and acknowledgment of each message as clients receive and process them.

Searched terms: connectivity, cluster, ordering, mirror, mirroring, deployment, protocols, requirements, delivery guarantee

Memphis is designed to run as a distributed cluster for a highly available and scalable system. The consensus algorithm responsible for atomicity within Memphis is called RAFT and does not require a witness or a standalom Qorum, unlike others such as Apache ZooKeeper which is widely used by projects like Kafka. RAFT is also equivalent to in fault tolerance and performance.

⭐
Paxos
SDKs
REST
WebSockets
Exactly once (Idempotence)