Search…
⌃K
Links

Consumer API

This section describes Memphis consumer API

What is a consumer?

A consumer is a client that reads data or messages from the broker or more specifically from the station.
As the user configures a client connection to Memphis, it comprises several objects
  • Connection - An open socket between the client to Memphis. Only required once as the client/application gets initialized for the first time.
  • Consumer - A consumer entity must be declared to read data/messages from Memphis.
  • (And/Or) Producer - A producer entity must be declared to write data/messages into Memphis.

Broker Data Format

Memphis started from NATS which receives, stores, and sends data in binary format for performance, format alignment, and efficient memory allocations.
When consumers consume messages to Memphis station, they should be converted from binary to string or any other type needed.
An example from the node.js SDK using .getData().toString() -
consumer.on('message', (message) => {
let msg = message.getData().toString();
message.ack();
});

Parameters

(*) Names might be a bit different from one SDK to another. Meanings are the same.
Connection
  • host: Memphis URL
  • port: Memphis port
  • username: Can be root or any other application-type user
  • connectionToken: The token received when the user created. Will change in the future to more robust credentials and authentication system
  • reconnect: The connection entity will try to reconnect to Memphis in case of a disconnection
  • maxReconnect: Amount of time the client will try to reconnect before backing off
  • reconnectIntervalMs: Time window between one retry to another
  • timeoutMs: Ability to kill a dead connection after explicit time
Consumer
  • stationName: The name of the station to be connected
  • consumerName: In a station resolution, each connected consumer must have a unique identity
  • consumerGroup: Explained in detail here. Consumers are grouped under an object called "Consumer group." If not specified, a default CG will be created using the consumerName
  • pullIntervalMs: Configured in milliseconds, this parameter defines the intervals of each consume operation. For example, if the value is set to 1000, it means that every 1000 ms, the consumer will try to pull new messages
  • batchSize: Defines how many messages will be collected per pull operation
  • batchMaxTimeToWaitMs: Defines how much time (in milliseconds) the consumer should wait for the entire required batch to be collected
  • maxAckTimeMs: For the consumer to receive the next message, the current one must be acknowledged, meaning the consumer is ready to consume and handle the next message. Oftentimes, the consumer gets crashed/throws an exception / not able to handle the message. The maxAckTimeMs ensure that until X millisecond Memphis has not received ACK, it will automatically retransmit the message. If not configured correctly, it can result in a duplicate processing
  • maxMsgDeliveries: The number of times Memphis will retransmit the same message to the same consumer
For more information about how to create and connect a consumer to Memphis,
please head here

Supported Protocols

  • NATS Protocol (Client SDKs)
  • HTTP * Soon *
  • WebSockets * Soon *
  • gRPC * Soon *
  • WASM * Soon *
  • MQTT * Soon *