LogoLogo
CloudDiscordGitHub
  • 👉Getting Started
    • Introduction
    • Quick start
    • Learn by example
    • Case studies
    • How to contribute?
  • ⭐Memphis Broker
    • Architecture
    • Key concepts
      • Message broker
      • Station
      • Producer API
      • Consumer API
      • Consumer Group
      • Storage and Redundancy
      • Security/Authentication
      • Scaling
      • Ordering
      • Dead-letter Station (DLS)
      • Delayed messages
      • Data exchange
      • Idempotency (Duplicate processing)
      • Failover Scenarios
      • Troubleshooting process
      • Connectors
    • Best practices
      • Producer optimization
      • Compression
    • Memphis configuration
    • Comparisons
      • NATS Jetstream vs Memphis
      • RabbitMQ vs Memphis
      • AWS SQS vs Memphis
      • Apache Kafka vs Memphis
      • Apache Pulsar vs Memphis
      • ZeroMQ vs Memphis
      • Apache NiFi vs Memphis
    • Privacy Policy
  • ⭐Memphis Schemaverse
    • Overview
    • Getting started
      • Management
      • Produce/Consume
        • Protobuf
        • JSON Schema
        • GraphQL
        • Avro
    • Comparison
    • KB
  • 📦Open-Source Installation
    • Kubernetes
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
      • Terraform
        • Deploy on AWS
        • Deploy on GCP
        • Deploy on DigitalOcean
      • Guides
        • Deploy/Upgrade Memphis utilizing predefined secrets
        • Monitoring/Alerts Recommendations
        • Production Best Practices
        • NGINX Ingress Controller and Cloud-Agnostic Memphis Deployments
        • Migrate Memphis storage between storageClass's
        • Expanding Memphis Disk Storage
        • Scale-out Memphis cluster
        • TLS - Deploy Memphis with TLS Connection to Metadata Frontend
        • TLS - Memphis TLS websocket configuration
        • TLS - Securing Memphis Client with TLS
        • Installing Memphis with an External Metadata Database
    • Docker
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
    • Open-source Support
  • Client Libraries
    • REST (Webhook)
    • Node.js / TypeScript / NestJS
    • Go
    • Python
    • Kotlin (Community)
    • .NET
    • Java
    • Rust (Community)
    • NATS
    • Scala
  • 🔌Integrations Center
    • Index
    • Processing
      • Zapier
    • Change data Capture (CDC)
      • Debezium
    • Monitoring
      • Datadog
      • Grafana
    • Notifications
      • Slack
    • Storage tiering
      • S3-Compatible Object Storage
    • Source code
      • GitHub
    • Other platforms
      • Argo
  • 🗒️Release notes
    • KB
    • Releases
      • v1.4.3 - latest/stable
      • v1.4.2
      • v1.4.1
      • v1.4.0
      • v1.3.1
      • v1.3.0
      • v1.2.0
      • v1.1.1
      • v1.1.0
      • v1.0.3
      • v1.0.2
      • v1.0.1
      • V1.0.0 - GA
      • v0.4.5 - beta
      • v0.4.4 - beta
      • v0.4.3 - beta
      • v0.4.2 - beta
      • v0.4.1 - beta
      • v0.4.0 - beta
      • v0.3.6 - beta
      • v0.3.5 - beta
      • v0.3.0 - beta
      • v0.2.2 - beta
      • v0.2.1 - beta
      • v0.2.0 - beta
      • v0.1.0 - beta
Powered by GitBook
LogoLogo

Legal

  • Terms of Service
  • Privacy Policy

All rights reserved to Memphis.dev 2023

On this page
  • Scaling Up vs Scaling Out / Vertical vs Horizontal Scaling
  • Scaling up / Vertical Scaling
  • Scaling out / Horizontal Scaling

Was this helpful?

  1. Memphis Broker
  2. Key concepts

Scaling

Last updated 1 year ago

Was this helpful?

The following section outlines the Memphis scaling process and provides instructions on how to implement it.

Scaling Up vs Scaling Out / Vertical vs Horizontal Scaling

Memphis utilize the same scaling paradigm as most database or application solutions which is called Scaling Up vs Scaling Out aka Vertical vs Horizontal Scaling.

Scaling up an application refers to increasing the capacity of the current system by adding more resources such as CPU, memory, or storage. This is also known as vertical scaling.

On the other hand, scaling out an application involves distributing the workload across multiple systems or instances to increase capacity. This is also known as horizontal scaling.

Both approaches can be used to increase the capacity of an application to handle a larger workload, but they have different implications and trade-offs. Scaling up a single system can be more cost-effective, but it may reach a physical limit in terms of the resources that can be added. Scaling out requires more infrastructure and may be more complex to set up and manage, but it allows for virtually unlimited scaling potential.

Below, you'll find examples, instructions and best practices on how to scale up/scale out your Memphis cluster.

Scaling up / Vertical Scaling

Addition of CPU / memory / storage to every broker itself (can be only one). As mentioned in the hardware requirement table, Memphis can be installed with only one memphis broker over a Kubernetes cluster with a single worker equipped with 2 CPUs, 4GB of RAM, and 16GB of storage. Note that these are the minimum requirements. You can go below them at your own risk but may experience instability in your Memphis ecosystem.

As the demand for computing resources increases with the growth of workload and data transfer through Memphis, it may be necessary to allocate additional resources. One approach to doing this is to increase the number of CPUs in the Kubernetes nodes and, depending on the storage requirements of the system, adding more memory and/or storage capacity.

As long as it is implemented correctly, using Kubernetes for the production-level of Memphis will not impact the functioning of Memphis and there will be no downtime during the scaling process.

Strengths

  • Relative speed: Upgrading a resource, such as replacing a single processor with a dual processor, doubles the processing speed of the CPU. Similarly, upgrading dynamic random access memory (DRAM) can enhance the dynamic memory performance of the system.

  • Simplicity: Expanding the size of a current system does not alter network connectivity or software configuration, making the scaling process more efficient and requiring less time and effort compared to the alternative of scaling out the architecture.

Scaling out / Horizontal Scaling

Scale-out is a concept that exists in distributed applications only.

In such a scale type, each cluster node act as a stateless worker of the cluster, and when more power is needed, we add more cluster workers. In Memphis's case, this means adding more Memphis Brokers to handle the increasing workload. Please refer down below for an example of how to add more Memphis brokers.

A step-by-step guide to adding more Memphis brokers

Step 1: Add more StatefulSets

kubectl scale statefulsets memphis --replicas=<new amount of replicas> -n memphis

Step 2: Edit Memphis ConfigMap route table

kubectl get cm memphis-config -o yaml -n memphis > memphis-config.yaml
vi memphis-config.yaml

Add the new StatefulSet in the marked line with the following pattern -

, nats://memphis-X.memphis.memphis.svc.cluster.local:6222

Relevant for Memphis

For further information on upgrading Kubernetes nodes with zero downtime
cluster-mode
⭐
Page cover image