Click or drag to resize

Overview

Caution note Caution

Neon.Temporal is still under construction and is not ready for any kind of use.

Temporal is a fork of the popular Uber Cadence workflow platform. This fork is maintained by the creators of Uber Cadence as well as Amazon Simple Workflow Foundation (SWF) and we expect that Temporal will be where we'll see continuted advances. We plan on deprecating Neon.Cadence in the near future in favor of Temporal.

The Neon.Temporal class library is built for NETStandard 2.0 and is capable of connecting to a Temporal workflow cluster to manage and implement workflows.

Caution note Caution

Neon.Temporal currently supports AMD 64-bit Windows, OS/X, and Linux runtime environments only. 32-bit and ARM platforms are not currently supported.

Temporal is a wonderful open source solution for developers who need to deploy reliable workflows for small to highly scaled applications and services. This isn't the first workflow product, but it may be the first one tailored specifically for developers who want to code workflows in a popular language as opposed to drawing flow-charts, editing crazy XML workflow definitions, and/or jumping through multi-step build hoops.

Neon.Temporal combined with Temporal Server lets you implement workflows in any supported .NET language with little friction. You can even set breakpoints and debug your workflows directly in Visual Studio.

This page describes the key concepts. Subsequent pages will show you how to get started.

Overview

Temporal is a general purpose platform that can handle workflows that exist for just a moment or two or workflows that run for days months or even years. Temporal persists the state of running workflows using the Cassandra or MySQL database to ensure that workflows will execute correctly even in the face of hardware failures. Temporal and Cassandra can be easily scaled horizontally for fault tolerance and to handle very high loads. Uber has Temporal deployments that can handle millions of simultaneous workflows.

Here's a video by Maxim Fateev, the creator of Temporal, describing what Temporal is and how it works:

Temporal: The Only Workflow Platform You'll Ever Need

and here's a link to the Neon Temporal client namespace documentation if you want to dive in immediately:

Neon.Temporal

To supplement Maxim's video, we'll discuss how this works at a high-level below so you'll have some understanding of the key concepts and how they fit together:

Temporal Server

Responsible for coordinating and tracking workflows and activities. From the perspective of a workflow developer, this includes the Temporal Server itself as well as the database where Temporal perisits its state. Temporal supports Cassandra for data storage as well as MySQL, Postgres, as well as a plugin architecture for custom database integrations. Cassandra is free, scalable, and battle tested and is what we've been using for the Neon related projects.

When we use the term Temporal Server this typically includes the associated database as well. Production deployments may include multiple Temporal Servers for reliability and scalability. This may be referred to as a Temporal Cluster or perhaps as the Temporal Server since the cluster-ness is mostly abstracted from a workflow author's perspective.

Temporal Client

Provides the APIs for accessing a Temporal Server or Cluster. This is implemented by TemporalClient and you'll establish a connection to a Temporalr server by calling ConnectAsync, passing settings that specify the IP address or host name for one or more of the Temporal servers in the cluster.

Once you have a connection, you can use it to deploy, call, as well as manage workflows and activities.

Workflows

A fault-oblivious stateful function that orchestrates activities. A Workflow has full control over which activities are executed, and in which order. A Workflow must not affect the external world directly, only through activities. What makes workflow code a Workflow is that its state is preserved by Temporal. Therefore any failure of a worker process that hosts the workflow code does not affect the workflow execution. The Workflow continues as if these failures did not happen. At the same time, activities can fail any moment for any reason. Because workflow code is fully fault-oblivious, it is guaranteed to get notifications about activity failures or timeouts and act accordingly. There is no limit on potential workflow duration.

Activities

A business-level function that implements your application logic such as calling a service or transcoding a media file. An activity usually implements a single well-defined action; it can be short or long running. An activity can be implemented as a synchronous method or fully asynchronously involving multiple processes. An activity can be retried indefinitely according to the provided exponential retry policy. If for any reason an activity is not completed within the specified timeout, an error is reported to the workflow and the workflow decides how to handle it. There is no limit on potential activity duration.

Local Activities

An activity that is invoked directly in the same process by a workflow code. It consumes much less resources than a normal activity, but imposes a lot of limitations like low duration and lack of rate limiting. These are commonly used for things like making a REST call or performing a database operation.

Workers

Workers are responsible for actually implementing workflows and activities. This is code that a workflow developer will write and deploy. You'll need to establish a connection a Temporal cluster anmd then register your workflow and activity implementations with the cluster. Then, Temporal will schedule workflows and activities on these workers by instantiating and invoking your workflow and activity classes.

Decisions and History

A decision is any action taken by a workflow implementation. This may include calling an activity or child workflow as well as other built-in operations like sleeping for a period of time or generating a UUID or random number. Typically, the workflow code consists of just the logic around coordinating activities and child workflows that actually perform the workflow operations.

In legacy workflow platforms, a decision task would be modeled as conditional nodes in a flow chart along with nodes that do something like call a REST API, do a database operation, send an email, etc. Temporal is code centric, so a decision task is more of a virtual concept. It's essentially the bit of logic in your workflows between activity calls as well as the built-in APIs.

Temporal maintains a durable history of the results of all activities and built-in APIs so that any workflow can be rescheduled on a different worker instance and end up in the same state as it had before. This is central to how Temporal works.

Note that workflow logic must never access or modify global or external state directly. This includes things like performing database or REST operations but also includes things like getting the time, generating UUIDs and random numbers, or sleeping for a period of time. You must use activities, child workflows or the built-in APIs to ensure that the workflow will end up in the same state after being rescheduled. Workflow logic has a limited amount of time to make decisions between activity, child workflow, or built-in API calls. By default, a workflow has 10 seconds to decide what to do next. This can be increased to a maximum of 60 seconds via configuration.

Namespaces and Task Queues

Temporal is a multi-tenant service and uses namespaces to isolate each tenant from any others. A namespace is simply a unique namespace where workflows and activities as well as their worker implementations are registered.

Workflow and Run IDs

Workflows are identifyed by Workflow ID. This can be the name of a business entity like order-00001234 that identifies a product order in a database or ltv-jeff@lilltek.com that identifies a workflow that periodically computes the life-time-value (LTV) of the user with ID jeff@lilltek.com. Workflow IDs can be any string including a UUID.

Workflow Run IDs are UUIDs generated by Temporal for each run of a workflow. Workflows are allowed (via configuration) to be able to scheduled multiple times using the same Workflow ID. This is handy when workflow IDs map to business entities. In this case, you can use the Run ID to identify exactly which run of the workflow you're interested in. The most workflow recent run is assumed when you don't specify a Run ID.

overview-components

The diagram above depicts how these components fit together. The two RED boxes show the Temporal Cluster (one or more Temporal server instances) along with its backing database. You'll need to deploy and configure these as outlined here: Temporal Deployment

You'll be responsible for implementing and deploying the GREEN boxes including one or more workers with your workflow and activity implementations as well as any applications that need to run or monitor workflows. We'll do a very high level workflow of how this works:

  1. Deploy and start the Temporal cluster along with its database.

  2. Author your workflows and activities as a deployable process. This is typically just a .NET Console application that runs on its own or perhaps as a Docker image in a Kubernetes cluster. These workers will register their workflow and activity implementations with Temporal under a task queue (Acme-PROD in this case) to isolate these registrations from other cluster users.

    For this example, we each worker registers a workflow that is designed to send an email and the underlying activity that actually sends the email. The workflow essentially consists of a single decision task that just calls the activity or perhaps something including more complex that customizes retry handling.

  3. Once one or more workers have registered their workflow/activity implementations and started, the Temporal cluster will be ready to schedule workflow and activity executions on them.

  4. The last GREEN box represents your application code. This can be anything such as a website, REST API backend, console application, etc. This is typically where workflows will be invoked.

  5. Let's say the application needs to send an email. The application will need to connect to the Temporal cluster with the Acme-PROD task queue and invoke the Send() workflow, passing the target address, subject, message, and such.

  6. Temporal receives the workflow call and persists it to the database.

  7. Temporal then looks for any workers that have registered the workflow implementation. It chooses one and invokes the workflow there.

  8. Workflows are registered on workers as interfaces and classes. From the workflow developer's perspective, a workflow invocation means that an instance of your workflow class will be constructed and your workflow entry point method will be called.

  9. For this example, the Send() workflow simply calls the Send() activity to actually send the email.

  10. Temporal receives the Send() activity call, persists it to the database and then looks for any workers that have registered the activity, chooses one and invokes the activity there. Activity implementations work essentially the same as workflows: an activity class will be instantiated and its entry point method will be called.

  11. The activity sends the message and returns any results to Temporal. Temporal server persists the result into the workflow history and then returns the reslt to the workflow.

  12. Temporal server returns the activity result to the workflow when then returns its result back to Temporal which records it to the database and then returns the result back to the original calling application.

See Also