The Neon.Cadence class library is built for NETStandard 2.0 and is capable of connecting to a Uber Cadence workflow cluster to manage and implement workflows.
Neon.Cadence currently supports AMD 64-bit Windows, OS/X, and Linux runtime environments only. 32-bit and ARM platforms are not currently supported.
Uber Cadence is a wonderful open source solution for developers who need to deploy reliable workflows for small to highly scaled applications and services. This isn't the first workflow product, but it may be the first one tailored specifically for developers who want to code workflows in a popular language as opposed to drawing flow-charts, editing crazy XML workflow definitions, and/or jumping through multi-step build hoops.
Neon.Cadence combined with Cadence Server lets you implement workflows in any supported .NET language with little friction. You can even set breakpoints and debug your workflows directly in Visual Studio.
This page describes the key concepts. Subsequent pages will show you how to get started.
Cadence is a general purpose platform that can handle workflows that exist for just a moment or two or workflows that run for days months or even years. Cadence persists the state of running workflows using the Cassandra or MySQL database to ensure that workflows will execute correctly even in the face of hardware failures. Cadence and Cassandra can be easily scaled horizontally for fault tolerance and to handle very high loads. Uber has Cadence deployments that can handle millions of simultaneous workflows.
Here's a video by Maxim Fateev, the creator of Cadence, describing what Cadence is and how it works:
and here's a link to the Neon Cadence client namespace documentation if you want to dive in immediately:
To supplement Maxim's video, we'll discuss how this works at a high-level below so you'll have some understanding of the key concepts and how they fit together:
Responsible for coordinating and tracking workflows and activities. From the perspective of a workflow developer, this includes the Cadence Server itself as well as the database where Cadence perisits its state. Cadence supports Cassandra for data storage as well as MySQL, Postgres, as well as a plugin architecture for custom database integrations. Cassandra is free, scalable, and battle tested and is what we've been using for the Neon related projects.
When we use the term Cadence Server this typically includes the associated database as well. Production deployments may include multiple Cadence Servers for reliability and scalability. This may be referred to as a Cadence Cluster or perhaps as the Cadence Server since the cluster-ness is mostly abstracted from a workflow author's perspective.
Provides the APIs for accessing a Cadence Server or Cluster. This is implemented by CadenceClient and you'll establish a connection to a Cadencer server by calling ConnectAsync, passing settings that specify the IP address or host name for one or more of the Cadence servers in the cluster.
Once you have a connection, you can use it to deploy, call, as well as manage workflows and activities.
A fault-oblivious stateful function that orchestrates activities. A Workflow has full control over which activities are executed, and in which order. A Workflow must not affect the external world directly, only through activities. What makes workflow code a Workflow is that its state is preserved by Cadence. Therefore any failure of a worker process that hosts the workflow code does not affect the workflow execution. The Workflow continues as if these failures did not happen. At the same time, activities can fail any moment for any reason. Because workflow code is fully fault-oblivious, it is guaranteed to get notifications about activity failures or timeouts and act accordingly. There is no limit on potential workflow duration.
A business-level function that implements your application logic such as calling a service or transcoding a media file. An activity usually implements a single well-defined action; it can be short or long running. An activity can be implemented as a synchronous method or fully asynchronously involving multiple processes. An activity can be retried indefinitely according to the provided exponential retry policy. If for any reason an activity is not completed within the specified timeout, an error is reported to the workflow and the workflow decides how to handle it. There is no limit on potential activity duration.
An activity that is invoked directly in the same process by a workflow code. It consumes much less resources than a normal activity, but imposes a lot of limitations like low duration and lack of rate limiting. These are commonly used for things like making a REST call or performing a database operation.
Workers are responsible for actually implementing workflows and activities. This is code that a workflow developer will write and deploy. You'll need to establish a connection a Cadence cluster anmd then register your workflow and activity implementations with the cluster. Then, Cadence will schedule workflows and activities on these workers by instantiating and invoking your workflow and activity classes.
A decision is any action taken by a workflow implementation. This may include calling an activity or child workflow as well as other built-in operations like sleeping for a period of time or generating a UUID or random number. Typically, the workflow code consists of just the logic around coordinating activities and child workflows that actually perform the workflow operations.
In legacy workflow platforms, a decision task would be modeled as conditional nodes in a flow chart along with nodes that do something like call a REST API, do a database operation, send an email, etc. Cadence is code centric, so a decision task is more of a virtual concept. It's essentially the bit of logic in your workflows between activity calls as well as the built-in APIs.
Cadence maintains a durable history of the results of all activities and built-in APIs so that any workflow can be rescheduled on a different worker instance and end up in the same state as it had before. This is central to how Cadence works.
Note that workflow logic must never access or modify global or external state directly. This includes things like performing database or REST operations but also includes things like getting the time, generating UUIDs and random numbers, or sleeping for a period of time. You must use activities, child workflows or the built-in APIs to ensure that the workflow will end up in the same state after being rescheduled. Workflow logic has a limited amount of time to make decisions between activity, child workflow, or built-in API calls. By default, a workflow has 10 seconds to decide what to do next. This can be increased to a maximum of 60 seconds via configuration.
Cadence is a multi-tenant service and uses domains to isolate each tenant from any others. A domain is simply a unique namespace where workflows and activities as well as their worker implementations are registered.
Workflows are identifyed by Workflow ID. This can be the name of a business entity like order-00001234 that identifies a product order in a database or firstname.lastname@example.org that identifies a workflow that periodically computes the life-time-value (LTV) of the user with ID email@example.com. Workflow IDs can be any string including a UUID.
Workflow Run IDs are UUIDs generated by Cadence for each run of a workflow. Workflows are allowed (via configuration) to be able to scheduled multiple times using the same Workflow ID. This is handy when workflow IDs map to business entities. In this case, you can use the Run ID to identify exactly which run of the workflow you're interested in. The most workflow recent run is assumed when you don't specify a Run ID.
The diagram above depicts how these components fit together. The two RED boxes show the Cadence Cluster (one or more Cadence server instances) along with its backing database. You'll need to deploy and configure these as outlined here: Cadence Deployment
You'll be responsible for implementing and deploying the GREEN boxes including one or more workers with your workflow and activity implementations as well as any applications that need to run or monitor workflows. We'll do a very high level workflow of how this works:
Deploy and start the Cadence cluster along with its database.
Author your workflows and activities as a deployable process. This is typically just a .NET Console application that runs on its own or perhaps as a Docker image in a Kubernetes cluster. These workers will register their workflow and activity implementations with Cadence under a task list (Acme-PROD in this case) to isolate these registrations from other cluster users.
For this example, we each worker registers a workflow that is designed to send an email and the underlying activity that actually sends the email. The workflow essentially consists of a single decision task that just calls the activity or perhaps something including more complex that customizes retry handling.
Once one or more workers have registered their workflow/activity implementations and started, the Cadence cluster will be ready to schedule workflow and activity executions on them.
The last GREEN box represents your application code. This can be anything such as a website, REST API backend, console application, etc. This is typically where workflows will be invoked.
Let's say the application needs to send an email. The application will need to connect to the Cadence cluster with the Acme-PROD task list and invoke the Send() workflow, passing the target address, subject, message, and such.
Cadence receives the workflow call and persists it to the database.
Cadence then looks for any workers that have registered the workflow implementation. It chooses one and invokes the workflow there.
Workflows are registered on workers as interfaces and classes. From the workflow developer's perspective, a workflow invocation means that an instance of your workflow class will be constructed and your workflow entry point method will be called.
For this example, the Send() workflow simply calls the Send() activity to actually send the email.
Cadence receives the Send() activity call, persists it to the database and then looks for any workers that have registered the activity, chooses one and invokes the activity there. Activity implementations work essentially the same as workflows: an activity class will be instantiated and its entry point method will be called.
The activity sends the message and returns any results to Cadence. Cadence server persists the result into the workflow history and then returns the reslt to the workflow.
Cadence server returns the activity result to the workflow when then returns its result back to Cadence which records it to the database and then returns the result back to the original calling application.