Ask or search…

Use cases and best practices Well-Architected

Learn, get inspired, and build better pipelines using architectural best practices.

Use case: Apply and enforce schemas over Kafka topics

Memphis Functions works great next to Kafka and simplifies the needed code around it. In this scenario, it operates as a consumer group for a specific Kafka topic, potentially targeting specific partitions.
  • Functions CG consumes each received event and subjects it to schema validation as defined in Schemaverse.
  • After validation, approved events proceed to the next Topic through Functions. Unapproved ones are discarded, directed to the Memphis Dead-letter, and trigger user notifications via email or Slack.
  • Despite introducing an additional step, both the movement and validation occur rapidly and efficiently. This segregation of raw, unvalidated data and validated data into separate topics adds an extra protective layer for consumers.

Use case: Prepare and enrich traces before entering BigQuery

Preparing real-time data presents a significant challenge when inserting it into BQ due to the necessary transformations before aligning with the specified schema.
One approach involves crafting a substantial client that consumes events, parses their types, and performs corresponding transformations. However, this method lacks scalability, and any code modifications may disrupt others due to its monolithic nature. Moreover, the constant changes in schemas, tables, and requirements exacerbate these challenges.
Memphis Functions enable the distribution of logic, enabling the creation of dedicated pipelines for each event type across multiple stations.
  • Memphis producers can produce the same message to multiple stations simultaneously.
  • A dedicated function sequence will be attached for each station.
  • Individual handling for each event type
  • Fully distributed and highly decoupled
  • Processing issues won't impact all pipelines
Last modified 1mo ago