Comment on page

Argo

Argo and Memphis using NATS Event source

Introduction
​Argo is a collection of open-source tools for Kubernetes to run workflows,manage clusters, and do GitOps easily. 
​Memphis is an open-source next-generation alternative to traditional message brokers.
Among Argo tools, Argo Workflows can be found.
Kubernetes-native workflow engine supporting DAG and step-based workflows.
Some of Argo Workflows features - 
Define workflows where each step in the workflow is a container.
Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a graph (DAG).
Easily run compute-intensive jobs for machine learning or data processing in a fraction of the time using Argo Workflows on Kubernetes.
Run CI/CD pipelines natively on Kubernetes without configuring complex software development products.
Argo-defined workflows can be triggered by incoming events which arrive from a component called "Event Source." 
Multiple event sources can be used simultaneously.
Using Memphis as an Argo workflows event source
Memphis provides multiple features that enhance the experience with Argo, like:
Full GUI with real-time observability
Schema management and transformation
REST Webhooks
Dead-Letter Queue with automatic message retransmit
Serverless stream enrichments
SDKs: Node.JS, Go, Python, TypeScript, NestJS
One of the key advantages of using Memphis is the ability to troubleshoot complicated async processes using message tracing, data-level observability, self-healing policies, and real-time notifications, which can be a great help, especially for auditing and logging when events and triggers are fired from multiple places towards your Argo and potentially can create highly expensive resources and workflows.
Memphis can trigger Argo workflows via Argo Event Source.
An Event Source defines the configurations required to consume events from external sources like Memphis, NATS, AWS SNS, SQS, GCP PubSub, Webhooks, etc.
Integrating Memphis as a NATS Event Source of Argo
Getting started
Step 1: Create event source YAML file
memphis_eventsource.yaml
1
apiVersion: argoproj.io/v1alpha1
2
kind: EventSource
3
metadata:
4
  name: memphis
5
spec:
6
  nats:
7
    example:
8
      # url of the nats service
9
      url: nats://<MEMPHIS_BROKER_URL>:6666
10
      # jsonBody specifies that all event body payload coming from this
11
      # source will be JSON
12
      jsonBody: true
13
      # subject name = memphis station
14
      subject: argo_workflows
15
      auth:
16
        token:
17
          name: memphis-creds​
18
          key: CONNECTION_TOKEN
19
      # optional backoff time for connection retries.
20
      # if not provided, default connection backoff time will be used.
21
      connectionBackoff:
22
        # duration in nanoseconds, or strings like "4s", "1m". following value is 10 seconds
23
        duration: 10s
24
        # how many backoffs
25
        steps: 5
26
        # factor to increase on each step.
27
        # setting factor > 1 makes backoff exponential.
28
        factor: 2
29
        jitter: 0.2
Step 2:  Create memphis-creds secret in argo-evens namespace
kubectl create secret generic memphis-creds \
--from-literal=CONNECTION_TOKEN=<APPLICATION_USER>::<CONNECTION_TOKEN> \
-n argo-events
Step 3: Create Event Source resource
kubectl apply -f memphis_eventsource.yaml
Step 4: Create sensor resource YAML file
memphis_sensor.yaml
1
apiVersion: argoproj.io/v1alpha1
2
kind: Sensor
3
metadata:
4
  name: memphis
5
spec:
6
  template:
7
    serviceAccountName: operate-workflow-sa
8
  dependencies:
9
    - name: test-dep
10
      eventSourceName: memphis
11
      eventName: example
12
  triggers:
13
    - template:
14
        name: memphis-workflow-trigger
15
        k8s:
16
          operation: create
17
          source:
18
            resource:
19
              apiVersion: argoproj.io/v1alpha1
20
              kind: Workflow
21
              metadata:
22
                generateName: memphis-workflow-
23
              spec:
24
                entrypoint: whalesay
25
                arguments:
26
                  parameters:
27
                  - name: message
28
                    value: hello world
29
                templates:
30
                - name: whalesay
31
                  inputs:
32
                    parameters:
33
                    - name: message
34
                  container:
35
                    image: docker/whalesay:latest
36
                    command: [cowsay]
37
                    args: ["{{inputs.parameters.message}}"]
38
          parameters:
39
            - src:
40
                dependencyName: test-dep
41
              dest: spec.arguments.parameters.0.value
Step 5: Create sensor resource
kubectl apply -f memphis_sensor.yaml
Step 6: Wrap the "foo" subject with a stream
As Memphis works with streams, wrapping the subject will enable Memphis to control the subject. To complete that step, NATS cli is needed.
nats stream add  -s <MEMPHIS_BROKER_URL>:6666 --user=<MEMPHIS_APPLICATION_USER>::<MEMPHIS_CONNECTION_TOKEN> 
? Stream Name argo_workflows
? Subjects argo_workflows
? Storage file
? Replication 3
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Total Stream Size -1
? Message TTL -1
? Max Message Size -1
? Duplicate tracking time window 2m0s
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
Stream argo_workflows was created

Last modified 8mo ago