LogoLogo
CloudDiscordGitHub
  • 👉Getting Started
    • Introduction
    • Quick start
    • Learn by example
    • Case studies
    • How to contribute?
  • ⭐Memphis Broker
    • Architecture
    • Key concepts
      • Message broker
      • Station
      • Producer API
      • Consumer API
      • Consumer Group
      • Storage and Redundancy
      • Security/Authentication
      • Scaling
      • Ordering
      • Dead-letter Station (DLS)
      • Delayed messages
      • Data exchange
      • Idempotency (Duplicate processing)
      • Failover Scenarios
      • Troubleshooting process
      • Connectors
    • Best practices
      • Producer optimization
      • Compression
    • Memphis configuration
    • Comparisons
      • NATS Jetstream vs Memphis
      • RabbitMQ vs Memphis
      • AWS SQS vs Memphis
      • Apache Kafka vs Memphis
      • Apache Pulsar vs Memphis
      • ZeroMQ vs Memphis
      • Apache NiFi vs Memphis
    • Privacy Policy
  • ⭐Memphis Schemaverse
    • Overview
    • Getting started
      • Management
      • Produce/Consume
        • Protobuf
        • JSON Schema
        • GraphQL
        • Avro
    • Comparison
    • KB
  • 📦Open-Source Installation
    • Kubernetes
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
      • Terraform
        • Deploy on AWS
        • Deploy on GCP
        • Deploy on DigitalOcean
      • Guides
        • Deploy/Upgrade Memphis utilizing predefined secrets
        • Monitoring/Alerts Recommendations
        • Production Best Practices
        • NGINX Ingress Controller and Cloud-Agnostic Memphis Deployments
        • Migrate Memphis storage between storageClass's
        • Expanding Memphis Disk Storage
        • Scale-out Memphis cluster
        • TLS - Deploy Memphis with TLS Connection to Metadata Frontend
        • TLS - Memphis TLS websocket configuration
        • TLS - Securing Memphis Client with TLS
        • Installing Memphis with an External Metadata Database
    • Docker
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
    • Open-source Support
  • Client Libraries
    • REST (Webhook)
    • Node.js / TypeScript / NestJS
    • Go
    • Python
    • Kotlin (Community)
    • .NET
    • Java
    • Rust (Community)
    • NATS
    • Scala
  • 🔌Integrations Center
    • Index
    • Processing
      • Zapier
    • Change data Capture (CDC)
      • Debezium
    • Monitoring
      • Datadog
      • Grafana
    • Notifications
      • Slack
    • Storage tiering
      • S3-Compatible Object Storage
    • Source code
      • GitHub
    • Other platforms
      • Argo
  • 🗒️Release notes
    • KB
    • Releases
      • v1.4.3 - latest/stable
      • v1.4.2
      • v1.4.1
      • v1.4.0
      • v1.3.1
      • v1.3.0
      • v1.2.0
      • v1.1.1
      • v1.1.0
      • v1.0.3
      • v1.0.2
      • v1.0.1
      • V1.0.0 - GA
      • v0.4.5 - beta
      • v0.4.4 - beta
      • v0.4.3 - beta
      • v0.4.2 - beta
      • v0.4.1 - beta
      • v0.4.0 - beta
      • v0.3.6 - beta
      • v0.3.5 - beta
      • v0.3.0 - beta
      • v0.2.2 - beta
      • v0.2.1 - beta
      • v0.2.0 - beta
      • v0.1.0 - beta
Powered by GitBook
LogoLogo

Legal

  • Terms of Service
  • Privacy Policy

All rights reserved to Memphis.dev 2023

On this page
  • What is AWS SQS?
  • What is Memphis.dev?
  • Messaging
  • Data Flow
  • Data Usage
  • Message retention
  • Multi-tenancy
  • Observability
  • Features
  • Performance comparison
  • TCO comparison
  • Defining a cost model for data streaming
  • Implementation costs
  • Implementation costs
  • Infrastructure costs - 15K RPS or 60MB/s
  • Summary

Was this helpful?

  1. Memphis Broker
  2. Comparisons

AWS SQS vs Memphis

This section describes the differences between AWS SQS and Memphis

Last updated 2 years ago

Was this helpful?

What is AWS SQS?

Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and cost allocation tags. It provides a generic web services API that you can access using any programming language that the AWS SDK supports.

What is Memphis.dev?

Memphis is a next-generation messaging queue.

A simple, robust, and durable cloud-native message broker wrapped with an entire ecosystem that enables fast and reliable development of next-generation event-driven use cases.

Memphis.dev enables building next-generation applications that require large volumes of streamed and enriched data, modern protocols, zero ops, rapid development, extreme cost reduction, and a significantly lower amount of dev time for data-oriented developers and data engineers.

Messaging

Parameter
Memphis
AWS SQS

Benchmark

300K messages per second per station (queue).

60K-100K messages per second

Message Retention

Policy-based (e.g., 30 days)

Acknowledgment based

Data Type

Transactional, Operational

Transactional

Consumer Mode

Smart broker/Smart consumer

Smart broker/dumb consumer

Topology

Publish/subscribe based

Not supporting broadcasting. 1:1 producer to subscriber

Payload Size

Up to 15M

Max of 256KB

Batch size

No limitation

Max of 10 messages

Use Cases

Massive data/high throughput cases | Simple use cases

Simple use cases

Delivery Guarantee

At least once, Exactly once

At least once

Message ordering

Message ordering is provided via consumer groups. By message key, messages are sent to stations.

FIFO is optional

Message priorities

Unavailable

Unavailable

Message lifetime

Since station messages are kept on file/memory. This can be controlled by defining a retention policy.

Because SQS is a queue, messages are discarded after being read, and an acknowledgment is given.

Clustering

Active-Active

Unavailable

Performance

Scale-up, Scale-out

No control

Multi-region

Supported*

No

Multi-tenancy

Supported*

No

Read-replicas

Supported*

No

Data striping across nodes

Supported

Unavailable

*Available for Memphis cloud users

Data Flow

SQS uses a distinct, bounded data flow. Messages are created and sent by the producer and received by the consumer.

Memphis uses an unbounded data flow, with the key-value pairs continuously streaming to the assigned station.

Data Usage

AWS SQS is best for transactional data, such as order formation, placement, and user requests.

Memphis works great for transactional and operational data like process operations, auditing and logging statistics, and system activity.

Message retention

AWS SQS pushes messages to consumers. These messages are removed from the queue once they are processed and acknowledged.

Memphis is a log. It uses continuous messages, which stay in the station (queue) until the retention period expires.

Multi-tenancy

AWS SQS doesn't support multi-tenancy but through a lambda function, required to be code and managed by the user that acts as a router.

Memphis supports multi-tenancy using namespaces which offers a complete separation from connections, producers, consumers, security, dedicated dashboard, including node selection.

Observability

Some level of observability can be received by using 3rd party apps like Cloudwatch/Datadog/New Relic. To understand the full path of a message, it is required to use AWS X-Ray and add some headers to each client. Notifications can be achieved by building a dedicated event queue with lambda triggers. Some alarms and triggers must be defined over 3rd party apps to enable lag identifications and latency in real-time.

Memphis offers full Infra-to-cluster-to-data GUI-based observability, monitoring, real-time message tracing, and notifications embedded inside the management layer, including self-healing policies based on the defined events.

Features

Parameter
Memphis.dev
AWS SQS

GUI

Yes

Yes

Schema Management

Yes

No

Wildcard consume

No

Yes

Stream Enrichment

Yes

Yes

Ready-to-use source/sinks connectors

Yes

No

Stream lineage

Yes

No

Data-Level Observability

Yes

Yes

Self-healing

Yes + Managed service

Managed service

Deduplication

Yes. Modified bloom filter

Deduplication interval of 5 minutes

Delayed queues

Yes. Atomic per message.

Yes. Not atomic, and per entire queue.

Dead-letter

Yes

Yes

REST Gateway

Yes

No

Consumer internal communication

Experimental

No

Production deployment environment

Kubernetes, Docker, Managed service

Managed service

Storage tiering

Disk, Memory, S3 for Archiving

Disk

Notifications

Slack, Email, More

With SNS and Cloudwatch

SDK support

Node js, Python, Go, .NET, Java, NestJS, and Typescript

C++, Go, Java, .NET, Python, node.js, Rust, Ruby, PHP

Performance comparison

AWS SQS Client node: 1 x m4.2xlarge / 50 threads

Memphis Client node: 1 x m5n.8xlarge / 20 threads

1KB Messages
Memphis
AWS SQS

100K messages = 100MB

0.16951 sec

5.88 sec

500K messages = 500MB

2.74 sec

29 sec

1M messages = 1GB

9.419 sec

58 sec

10M messages = 10GB

106.576 sec

588 sec

TCO comparison

Defining a cost model for data streaming

In this economic climate, costs are top of mind for everyone.

Total Cost of Ownership (TCO) should be a primary consideration when evaluating the Return on Investment (ROI) of adopting a new software platform. TCO is the blended cost of deploying, configuring, securing, productionizing, and operating the software over its expected lifetime, including all infrastructure, personnel, training, and subscription costs.

For this comparison, we define TCO as a combination of the following components:

  1. Implementation: The cost of implementing a new streaming technology

  2. Infrastructure: The cost of computing and storage, in this case on AWS

For the infrastructure cost comparison, we ran benchmarks to compare the performance of AWS SQS against Memphis.

Implementation costs

Oftentimes, there is a misconception that cloud services are a turnkey solution. Here are some of the missing components that will need to be constructed when using AWS SQS -

Implementation costs

Feature
Memphis
AWS SQS

Performance

Built-in. Automatic.

0 dev hours

Required to use threads.

130 dev hours

Multi-Cloud

Built-in. By design.

0 dev hours

Required to build abstraction to different cloud queues and APIs. 378 dev hours

Multi-tenancy

Built-in. By design.

0 dev hours

Required to build. Using different queues and/or tagging data.

63 dev hours

Monitoring and Notifications

Built-in. Ready-to-use slack notifications / Grafana / Datadog / prometheus.

12 dev hours + 2 DevOps hours

Required to build + use 3rd party open-source/paid tools.

126 DevOps hours + 49 dev hours

Runtime DLQ consumption

Built-in.

0 dev hours

Required to build.

30 dev hours

Delayed consumers

Built-in.

0 dev hours

Required to build.

20 dev hours

Cost

$1,000

Based on average dev hourly rate of $70.

$55,720 ($54,720 difference)

Infrastructure costs - 15K RPS or 60MB/s

Metric
Memphis
AWS SQS

Requests per month

30,000 messages per second = 77,760 million messages per month

30,000 messages per second = 155,520 million requests per month

*Every Amazon SQS action counts as a request, including DLQ, publish, subscribe

EKS

Organizational.

None.

Nodes

3 x i3en.large (On-demand). $547

Included.

Client nodes

1 X M4.2xlarge.

$324.12

12 X M4.2xlarge

$3,889.44

Retention (Storage)

3 days X 10 kb message X 77,760M = 6Tb.

$1,737.52

3 days

Data Transfer between AZs

$2,866

$1,433.60

Licensing

Memphis Self-hosted enterprise licensing for partners. Flat.

$1,100 – $2,000

Included.

Cost

$6,574

Based on average dev hourly rate of $70 $56,655.60 ($50,081 difference)

Summary

15K RPS
30K RPS
60K RPS

Memphis.

Implementation

$100

$100

$100

Memphis.

Infrastructure

$4,330

$6,574

$9,909

Memphis.

Total

$4,430

$6,674

$10,009

AWS SQS.

Implementation

$4,643

$4,643

$4,643

AWS SQS.

Infrastructure

$35,709

$56,655

$99,516

AWS SQS.

Total

$40,352

$61,298

$104,159

Summary.

Implementation

x46

x46

x46

Summary. Infrastructure

x8.2

x8.6

x10

Summary. TCO

x9.1

x9.1

x10.4

Performance heavily relies on the client’s threads

What if you required to run on GCP for specific customer?

Not built for SaaS.

No multi-tenancy

Consumer-side delay queues

Monitoring and notification

Consumption from DLQ

⭐
AWS SQS
Memphis namespaces
Memphis GUI
Troubleshooting process
Notification Center
Page cover image