Workflow Engine (Temporal, Cadence, AWS Step Functions)

Why it exists

Some processes are hard: 'Charge the card, then update the shipping order, then send a confirmation email — and if any step fails, roll back the previous ones.' Doing this in application code means writing retry logic, compensation logic, state persistence, timeout handling, and observability yourself — every time. Workflow engines factor all of this out. You write the business logic; the engine handles durability, retries, and compensation.

How it works

A workflow is defined as code (Temporal, Cadence) or as a JSON state machine (Step Functions). Each step is an activity — a call to a service or a piece of business logic. The engine persists the workflow's state after every step. On worker failure, the engine reruns the workflow from the last checkpoint. Because activities may re-execute, they must be idempotent. Workflows can wait for external signals (user confirmation), sleep for days, and roll back via explicit compensation actions.

Scaling characteristics

Temporal / Cadence handle millions of concurrent workflows in production; state size per workflow is bounded (~100 KB) but total state can be terabytes. Latency per step is a few ms of engine overhead + the activity itself. Throughput scales with worker fleet + engine cluster; typical clusters run 100K workflow starts/sec.

When to use it

• Multi-service transactions (order → payment → inventory → shipping)
• Long-running processes (KYC, onboarding, subscription lifecycle)
• Human-in-the-loop flows (approval workflows)
• Compensating transactions across services (Saga pattern implementation)
• Retryable ML pipelines (data prep → training → deployment)
• Batch processing with complex step dependencies

When NOT to use it

• Simple synchronous requests — the overhead isn't worth it
• High-throughput low-latency work — the engine adds latency; a direct queue is faster
• Two-step processes where a queue + retry is enough
• You're already using Airflow and it works — don't switch just for the sake of it

Failure modes

• Non-idempotent activities cause corruption on retry — always design activities to be idempotent
• Workflow history grows unbounded on long-running workflows — use continueAsNew to reset
• Poison workflow: a bug in the code causes infinite retry — set retry limits + alerts
• Engine cluster overloaded from too many workflows — capacity plan carefully
• Debugging is harder than synchronous code — invest in observability early

Alternatives

• Choreographed sagas via events — no central engine; more moving parts to reason about
• A queue with retry — good for 2-step workflows; falls apart at 5+ steps
• Airflow — better for scheduled DAGs (batch); worse for event-triggered long-running flows
• Custom code + database state — reinventing the wheel; usually a mistake at any scale

Interview questions

• Explain the Saga pattern. When would you use a workflow engine to implement it vs. choreographed events?
• How do you handle a workflow that fails halfway through — resume from where it stopped or start over?
• Design a subscription-renewal workflow that runs monthly, retries on payment failure, and grace-periods before cancellation.
• What does 'idempotent activity' mean in Temporal, and why is it critical?
• Your workflow started before you deployed a code change. What happens when it resumes?
• How would you monitor the health of 10M concurrent workflows?