LogoLogo
CloudDiscordGitHub
  • 👉Getting Started
    • Introduction
    • Quick start
    • Learn by example
    • Case studies
    • How to contribute?
  • ⭐Memphis Broker
    • Architecture
    • Key concepts
      • Message broker
      • Station
      • Producer API
      • Consumer API
      • Consumer Group
      • Storage and Redundancy
      • Security/Authentication
      • Scaling
      • Ordering
      • Dead-letter Station (DLS)
      • Delayed messages
      • Data exchange
      • Idempotency (Duplicate processing)
      • Failover Scenarios
      • Troubleshooting process
      • Connectors
    • Best practices
      • Producer optimization
      • Compression
    • Memphis configuration
    • Comparisons
      • NATS Jetstream vs Memphis
      • RabbitMQ vs Memphis
      • AWS SQS vs Memphis
      • Apache Kafka vs Memphis
      • Apache Pulsar vs Memphis
      • ZeroMQ vs Memphis
      • Apache NiFi vs Memphis
    • Privacy Policy
  • ⭐Memphis Schemaverse
    • Overview
    • Getting started
      • Management
      • Produce/Consume
        • Protobuf
        • JSON Schema
        • GraphQL
        • Avro
    • Comparison
    • KB
  • 📦Open-Source Installation
    • Kubernetes
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
      • Terraform
        • Deploy on AWS
        • Deploy on GCP
        • Deploy on DigitalOcean
      • Guides
        • Deploy/Upgrade Memphis utilizing predefined secrets
        • Monitoring/Alerts Recommendations
        • Production Best Practices
        • NGINX Ingress Controller and Cloud-Agnostic Memphis Deployments
        • Migrate Memphis storage between storageClass's
        • Expanding Memphis Disk Storage
        • Scale-out Memphis cluster
        • TLS - Deploy Memphis with TLS Connection to Metadata Frontend
        • TLS - Memphis TLS websocket configuration
        • TLS - Securing Memphis Client with TLS
        • Installing Memphis with an External Metadata Database
    • Docker
      • 1 - Installation
      • 2 - Access
      • 3 - Upgrade
    • Open-source Support
  • Client Libraries
    • REST (Webhook)
    • Node.js / TypeScript / NestJS
    • Go
    • Python
    • Kotlin (Community)
    • .NET
    • Java
    • Rust (Community)
    • NATS
    • Scala
  • 🔌Integrations Center
    • Index
    • Processing
      • Zapier
    • Change data Capture (CDC)
      • Debezium
    • Monitoring
      • Datadog
      • Grafana
    • Notifications
      • Slack
    • Storage tiering
      • S3-Compatible Object Storage
    • Source code
      • GitHub
    • Other platforms
      • Argo
  • 🗒️Release notes
    • KB
    • Releases
      • v1.4.3 - latest/stable
      • v1.4.2
      • v1.4.1
      • v1.4.0
      • v1.3.1
      • v1.3.0
      • v1.2.0
      • v1.1.1
      • v1.1.0
      • v1.0.3
      • v1.0.2
      • v1.0.1
      • V1.0.0 - GA
      • v0.4.5 - beta
      • v0.4.4 - beta
      • v0.4.3 - beta
      • v0.4.2 - beta
      • v0.4.1 - beta
      • v0.4.0 - beta
      • v0.3.6 - beta
      • v0.3.5 - beta
      • v0.3.0 - beta
      • v0.2.2 - beta
      • v0.2.1 - beta
      • v0.2.0 - beta
      • v0.1.0 - beta
Powered by GitBook
LogoLogo

Legal

  • Terms of Service
  • Privacy Policy

All rights reserved to Memphis.dev 2023

On this page
  • Introduction
  • Using Memphis as an Argo workflows event source
  • Getting started
  • Step 1: Create event source YAML file
  • Step 2: Create memphis-creds secret in argo-evens namespace
  • Step 3: Create Event Source resource
  • Step 4: Create sensor resource YAML file
  • Step 5: Create sensor resource
  • Step 6: Wrap the "foo" subject with a stream

Was this helpful?

  1. Integrations Center
  2. Other platforms

Argo

Argo and Memphis using NATS Event source

Last updated 2 years ago

Was this helpful?

Introduction

is a collection of open-source tools for Kubernetes to run workflows,manage clusters, and do GitOps easily.

is an open-source next-generation alternative to traditional message brokers.

Among Argo tools, Argo Workflows can be found. Kubernetes-native workflow engine supporting DAG and step-based workflows. Some of Argo Workflows features -

  • Define workflows where each step in the workflow is a container.

  • Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a graph (DAG).

  • Easily run compute-intensive jobs for machine learning or data processing in a fraction of the time using Argo Workflows on Kubernetes.

  • Run CI/CD pipelines natively on Kubernetes without configuring complex software development products.

Argo-defined workflows can be triggered by incoming events which arrive from a component called "Event Source." Multiple event sources can be used simultaneously.

Using Memphis as an Argo workflows event source

Memphis provides multiple features that enhance the experience with Argo, like:

  • Full GUI with real-time observability

  • Schema management and transformation

  • REST Webhooks

  • Dead-Letter Queue with automatic message retransmit

  • Serverless stream enrichments

  • SDKs: Node.JS, Go, Python, TypeScript, NestJS

One of the key advantages of using Memphis is the ability to troubleshoot complicated async processes using message tracing, data-level observability, self-healing policies, and real-time notifications, which can be a great help, especially for auditing and logging when events and triggers are fired from multiple places towards your Argo and potentially can create highly expensive resources and workflows.

Memphis can trigger Argo workflows via Argo Event Source.

An Event Source defines the configurations required to consume events from external sources like Memphis, NATS, AWS SNS, SQS, GCP PubSub, Webhooks, etc.

Getting started

Step 1: Create event source YAML file

memphis_eventsource.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: memphis
spec:
  nats:
    example:
      # url of the nats service
      url: nats://<>:6666
      # jsonBody specifies that all event body payload coming from this
      # source will be JSON
      jsonBody: true
      # subject name = memphis station
      subject: argo_workflows
      auth:
        token:
          name: 
          key: CONNECTION_TOKEN
      # optional backoff time for connection retries.
      # if not provided, default connection backoff time will be used.
      connectionBackoff:
        # duration in nanoseconds, or strings like "4s", "1m". following value is 10 seconds
        duration: 10s
        # how many backoffs
        steps: 5
        # factor to increase on each step.
        # setting factor > 1 makes backoff exponential.
        factor: 2
        jitter: 0.2

Step 2: Create memphis-creds secret in argo-evens namespace

kubectl create secret generic memphis-creds \
--from-literal=CONNECTION_TOKEN=<APPLICATION_USER>::<CONNECTION_TOKEN> \
-n argo-events

Step 3: Create Event Source resource

kubectl apply -f memphis_eventsource.yaml

Step 4: Create sensor resource YAML file

memphis_sensor.yaml
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: memphis
spec:
  template:
    serviceAccountName: operate-workflow-sa
  dependencies:
    - name: test-dep
      eventSourceName: memphis
      eventName: example
  triggers:
    - template:
        name: memphis-workflow-trigger
        k8s:
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: memphis-workflow-
              spec:
                entrypoint: whalesay
                arguments:
                  parameters:
                  - name: message
                    value: hello world
                templates:
                - name: whalesay
                  inputs:
                    parameters:
                    - name: message
                  container:
                    image: docker/whalesay:latest
                    command: [cowsay]
                    args: ["{{inputs.parameters.message}}"]
          parameters:
            - src:
                dependencyName: test-dep
              dest: spec.arguments.parameters.0.value

Step 5: Create sensor resource

kubectl apply -f memphis_sensor.yaml

Step 6: Wrap the "foo" subject with a stream

nats stream add  -s <MEMPHIS_BROKER_URL>:6666 --user=<MEMPHIS_APPLICATION_USER>::<MEMPHIS_CONNECTION_TOKEN> 
? Stream Name argo_workflows
? Subjects argo_workflows
? Storage file
? Replication 3
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Total Stream Size -1
? Message TTL -1
? Max Message Size -1
? Duplicate tracking time window 2m0s
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
Stream argo_workflows was created

As Memphis works with streams, wrapping the subject will enable Memphis to control the subject. To complete that step, is needed.

Argo
Memphis
NATS cli
🔌
Integrating Memphis as a NATS Event Source of Argo
Page cover image