LogoLogo
CloudDiscordGitHub
  • 👉Getting Started
    • Introduction
    • Quick start
    • Learn by example
    • Case studies
    • How to contribute?
  • ⭐Memphis Broker
    • Architecture
    • Key concepts
      • Message broker
      • Station
      • Producer API
      • Consumer API
      • Consumer Group
      • Storage and Redundancy
      • Security/Authentication
      • Scaling
      • Ordering
      • Dead-letter Station (DLS)
      • Delayed messages
      • Data exchange
      • Idempotency (Duplicate processing)
      • Failover Scenarios
      • Troubleshooting process
      • Connectors
    • Best practices
      • Producer optimization
      • Compression
    • Memphis configuration
    • Comparisons
      • NATS Jetstream vs Memphis
      • RabbitMQ vs Memphis
      • AWS SQS vs Memphis
      • Apache Kafka vs Memphis
      • Apache Pulsar vs Memphis
      • ZeroMQ vs Memphis
      • Apache NiFi vs Memphis
    • Privacy Policy
  • ⭐Memphis Schemaverse
    • Overview
    • Getting started
      • Management
      • Produce/Consume
        • Protobuf
        • JSON Schema
        • GraphQL
        • Avro
    • Comparison
    • KB
  • 📦Open-Source Installation
    • Kubernetes
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
      • Terraform
        • Deploy on AWS
        • Deploy on GCP
        • Deploy on DigitalOcean
      • Guides
        • Deploy/Upgrade Memphis utilizing predefined secrets
        • Monitoring/Alerts Recommendations
        • Production Best Practices
        • NGINX Ingress Controller and Cloud-Agnostic Memphis Deployments
        • Migrate Memphis storage between storageClass's
        • Expanding Memphis Disk Storage
        • Scale-out Memphis cluster
        • TLS - Deploy Memphis with TLS Connection to Metadata Frontend
        • TLS - Memphis TLS websocket configuration
        • TLS - Securing Memphis Client with TLS
        • Installing Memphis with an External Metadata Database
    • Docker
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
    • Open-source Support
  • Client Libraries
    • REST (Webhook)
    • Node.js / TypeScript / NestJS
    • Go
    • Python
    • Kotlin (Community)
    • .NET
    • Java
    • Rust (Community)
    • NATS
    • Scala
  • 🔌Integrations Center
    • Index
    • Processing
      • Zapier
    • Change data Capture (CDC)
      • Debezium
    • Monitoring
      • Datadog
      • Grafana
    • Notifications
      • Slack
    • Storage tiering
      • S3-Compatible Object Storage
    • Source code
      • GitHub
    • Other platforms
      • Argo
  • 🗒️Release notes
    • KB
    • Releases
      • v1.4.3 - latest/stable
      • v1.4.2
      • v1.4.1
      • v1.4.0
      • v1.3.1
      • v1.3.0
      • v1.2.0
      • v1.1.1
      • v1.1.0
      • v1.0.3
      • v1.0.2
      • v1.0.1
      • V1.0.0 - GA
      • v0.4.5 - beta
      • v0.4.4 - beta
      • v0.4.3 - beta
      • v0.4.2 - beta
      • v0.4.1 - beta
      • v0.4.0 - beta
      • v0.3.6 - beta
      • v0.3.5 - beta
      • v0.3.0 - beta
      • v0.2.2 - beta
      • v0.2.1 - beta
      • v0.2.0 - beta
      • v0.1.0 - beta
Powered by GitBook
LogoLogo

Legal

  • Terms of Service
  • Privacy Policy

All rights reserved to Memphis.dev 2023

On this page
  • Step 1: Verify Dynamic Resize Availability
  • Step 2: Increase PVC Size
  • Step 3: Update StatefulSet Configuration
  • Step 4: Modify "updateStrategy"
  • Step 5: Update memphis-config ConfigMap
  • Step 6: Delete the Current StatefulSet with the Orphan Option
  • Step 7: Deploy Edited StatefulSet
  • Step 8: Delete and Validate Pod Updates

Was this helpful?

  1. Open-Source Installation
  2. Kubernetes
  3. Guides

Expanding Memphis Disk Storage

Last updated 1 year ago

Was this helpful?

Kubernetes is commonly employed to deploy stateful applications like Memphis within your cluster. Each Pod in the retains access to its even after rescheduling, ensuring the individual state is maintained, separate from neighboring Pods within the set.

However, these volumes have a notable limitation: Kubernetes does not offer a straightforward way to resize them through the StatefulSet object. The spec.resources.requests.storage property within the volumeClaimTemplates field of the StatefulSet and restricts you from making necessary capacity increases. This article will demonstrate how to work around this limitation.

Step 1: Verify Dynamic Resize Availability

Start by confirming if dynamic resizing is available in the storage class. If not, add the following line to enable it:

allowVolumeExpansion: true

Step 2: Increase PVC Size

To increase the size of the attached PVC (repeat for all PVC instances), execute the following command:

kubectl patch pvc  memphis-js-pvc-memphis-0 -p '{"spec":{"resources":{"requests":{"storage":"600Gi"}}}}' -n memphis

Step 3: Update StatefulSet Configuration

Export the Memphis StatefulSet configuration to a YAML file and adjust the size of the PVC accordingly:

k get sts memphis -n memphis -o yaml > sts.yaml

Edit the file to reflect the new PVC size within the spec section:

    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 600Gi
      volumeMode: Filesystem

Step 4: Modify "updateStrategy"

Change the "updateStrategy" type to "OnDelete" within the sts.yaml file:

  updateStrategy:
    type: OnDelete

Step 5: Update memphis-config ConfigMap

Update the memphis-config ConfigMap with the new size for the storageEngine space. Note that the reload feature will not work, so you will need to restart all the brokers one by one manually. Modify the storageEngine section as follows:

storageEngine {
  max_mem: 8Gi
  store_dir: /data

  max_file:600Gi
}

Step 6: Delete the Current StatefulSet with the Orphan Option

Execute the following command to delete the current StatefulSet, ensuring the use of the --cascade=orphan option:

kubectl delete statefulset --cascade=orphan memphis -n memphis

Step 7: Deploy Edited StatefulSet

Deploy the edited StatefulSet to the cluster using the sts.yaml file:

kubectl apply -f sts.yaml -n memphis

Step 8: Delete and Validate Pod Updates

Delete the Pods individually and confirm that they have the new configuration and retain the previous data.

Important!!! Deleting and validating stateful resources one by one is crucial for a smooth migration process.

📦
StatefulSets
local persistent volumes
is immutable