Comment on page
AWS SQS vs Memphis
This section describes the differences between AWS SQS and Memphis
Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues and cost allocation tags. It provides a generic web services API that you can access using any programming language that the AWS SDK supports.
Memphis is a next-generation messaging queue.
A simple, robust, and durable cloud-native message broker wrapped with an entire ecosystem that enables fast and reliable development of next-generation event-driven use cases.
Memphis.dev enables building next-generation applications that require large volumes of streamed and enriched data, modern protocols, zero ops, rapid development, extreme cost reduction, and a significantly lower amount of dev time for data-oriented developers and data engineers.
Parameter | Memphis | AWS SQS |
|---|---|---|
Benchmark | 300K messages per second per station (queue). | 60K-100K messages per second |
Message Retention | Policy-based (e.g., 30 days) | Acknowledgment based |
Data Type | Transactional, Operational | Transactional |
Consumer Mode | Smart broker/Smart consumer | Smart broker/dumb consumer |
Topology | Publish/subscribe based | Not supporting broadcasting. 1:1 producer to subscriber |
Payload Size | Up to 15M | Max of 256KB |
Batch size | No limitation | Max of 10 messages |
Use Cases | Massive data/high throughput cases | Simple use cases | Simple use cases |
Delivery Guarantee | At least once, Exactly once | At least once |
Message ordering | Message ordering is provided via consumer groups. By message key, messages are sent to stations. | FIFO is optional |
Message priorities | Unavailable | Unavailable |
Message lifetime | Since station messages are kept on file/memory. This can be controlled by defining a retention policy. | Because SQS is a queue, messages are discarded after being read, and an acknowledgment is given. |
Clustering | Active-Active | Unavailable |
Performance | Scale-up, Scale-out | No control |
Multi-region | Supported* | No |
Multi-tenancy | Supported* | No |
Read-replicas | Supported* | No |
Data striping across nodes | Supported | Unavailable |
*Available for Memphis cloud users
SQS uses a distinct, bounded data flow. Messages are created and sent by the producer and received by the consumer.
Memphis uses an unbounded data flow, with the key-value pairs continuously streaming to the assigned station.
AWS SQS is best for transactional data, such as order formation, placement, and user requests.
Memphis works great for transactional and operational data like process operations, auditing and logging statistics, and system activity.
AWS SQS pushes messages to consumers. These messages are removed from the queue once they are processed and acknowledged.
Memphis is a log. It uses continuous messages, which stay in the station (queue) until the retention period expires.
AWS SQS doesn't support multi-tenancy but through a lambda function, required to be code and managed by the user that acts as a router.
AWS SQS
Memphis supports multi-tenancy using namespaces which offers a complete separation from connections, producers, consumers, security, dedicated dashboard, including node selection.
Memphis namespaces
Some level of observability can be received by using 3rd party apps like Cloudwatch/Datadog/New Relic. To understand the full path of a message, it is required to use AWS X-Ray and add some headers to each client. Notifications can be achieved by building a dedicated event queue with lambda triggers. Some alarms and triggers must be defined over 3rd party apps to enable lag identifications and latency in real-time.
Memphis offers full Infra-to-cluster-to-data GUI-based observability, monitoring, real-time message tracing, and notifications embedded inside the management layer, including self-healing policies based on the defined events.
Memphis GUI
Troubleshooting process
Notification Center
Parameter | Memphis.dev | AWS SQS |
|---|---|---|
GUI | Yes | Yes |
Schema Management | Yes | No |
Wildcard consume | No | Yes |
Stream Enrichment | Yes | Yes |
Ready-to-use source/sinks connectors | Yes | No |
Stream lineage | Yes | No |
Data-Level Observability | Yes | Yes |
Self-healing | Yes + Managed service | Managed service |
Deduplication | Yes. Modified bloom filter | Deduplication interval of 5 minutes |
Delayed queues | Yes. Atomic per message. | Yes. Not atomic, and per entire queue. |
Dead-letter | Yes | Yes |
REST Gateway | Yes | No |
Consumer internal communication | Experimental | No |
Production deployment environment | Kubernetes, Docker, Managed service | Managed service |
Storage tiering | Disk, Memory, S3 for Archiving | Disk |
Notifications | Slack, Email, More | With SNS and Cloudwatch |
SDK support | Node js, Python, Go, .NET, Java, NestJS, and Typescript | C++, Go, Java, .NET, Python, node.js, Rust, Ruby, PHP |
AWS SQS Client node: 1 x m4.2xlarge / 50 threads
Memphis Client node: 1 x m5n.8xlarge / 20 threads
1KB Messages | Memphis | AWS SQS |
|---|---|---|
100K messages = 100MB | 0.16951 sec | 5.88 sec |
500K messages = 500MB | 2.74 sec | 29 sec |
1M messages = 1GB | 9.419 sec | 58 sec |
10M messages = 10GB | 106.576 sec | 588 sec |
In this economic climate, costs are top of mind for everyone.
Total Cost of Ownership (TCO) should be a primary consideration when evaluating the Return on Investment (ROI) of adopting a new software platform. TCO is the blended cost of deploying, configuring, securing, productionizing, and operating the software over its expected lifetime, including all infrastructure, personnel, training, and subscription costs.
For this comparison, we define TCO as a combination of the following components:
- 1.Implementation: The cost of implementing a new streaming technology
- 2.Infrastructure: The cost of computing and storage, in this case on AWS
For the infrastructure cost comparison, we ran benchmarks to compare the performance of AWS SQS against Memphis.
Oftentimes, there is a misconception that cloud services are a turnkey solution.
Here are some of the missing components that will need to be constructed when using AWS SQS -
Performance heavily relies on the client’s threads
What if you required to run on GCP for specific customer?
Not built for SaaS.
No multi-tenancy
Consumer-side delay queues
Monitoring and notification
Consumption from DLQ
Feature | Memphis | AWS SQS |
|---|---|---|
Performance | Built-in. Automatic. 0 dev hours | Required to use threads. 130 dev hours |
Multi-Cloud | Built-in. By design. 0 dev hours | Required to build abstraction to different cloud queues and APIs.
378 dev hours |
Multi-tenancy | Built-in. By design. 0 dev hours | Required to build. Using different queues and/or tagging data. 63 dev hours |
Monitoring and Notifications | Built-in. Ready-to-use slack notifications / Grafana / Datadog / prometheus. 12 dev hours + 2 DevOps hours | Required to build + use 3rd party open-source/paid tools. 126 DevOps hours + 49 dev hours |
Runtime DLQ consumption | Built-in. 0 dev hours | Required to build. 30 dev hours |
Delayed consumers | Built-in. 0 dev hours | Required to build. 20 dev hours |
Cost | $1,000 | Based on average dev hourly rate of $70. $55,720 ($54,720 difference) |
Metric | Memphis | AWS SQS |
|---|---|---|
Requests per month | 30,000 messages per second = 77,760 million messages per month | 30,000 messages per second = 155,520 million requests per month *Every Amazon SQS action counts as a request, including DLQ, publish, subscribe |
EKS | Organizational. | None. |
Nodes | 3 x i3en.large (On-demand). $547 | Included. |
Client nodes | 1 X M4.2xlarge. $324.12 | 12 X M4.2xlarge $3,889.44 |
Retention (Storage) | 3 days X 10 kb message X 77,760M = 6Tb. $1,737.52 | 3 days |
Data Transfer between AZs | $2,866 | $1,433.60 |
Licensing | Memphis Self-hosted enterprise licensing for partners. Flat. $1,100 – $2,000 | Included. |
Cost | $6,574 | Based on average dev hourly rate of $70 $56,655.60 ($50,081 difference) |
Text | 15K RPS | 30K RPS | 60K RPS |
|---|---|---|---|
Memphis. Implementation | $100 | $100 | $100 |
Memphis. Infrastructure | $4,330 | $6,574 | $9,909 |
Memphis. Total | $4,430 | $6,674 | $10,009 |
AWS SQS. Implementation | $4,643 | $4,643 | $4,643 |
AWS SQS. Infrastructure | $35,709 | $56,655 | $99,516 |
AWS SQS. Total | $40,352 | $61,298 | $104,159 |
Summary. Implementation | x46 | x46 | x46 |
Summary.
Infrastructure | x8.2 | x8.6 | x10 |
Summary.
TCO | x9.1 | x9.1 | x10.4 |
Last modified 7mo ago