Scheduler (cron, Airflow, Temporal timers)

Why it exists

Some work must happen periodically: daily reports, weekly billing runs, hourly cache warmups, monthly cleanup. cron on a single machine works for small scale — but at scale, you need distributed schedulers that survive node failure, guarantee at-least-once (or exactly-once) execution, retry on failure, and give you observability (did the job run? how long? what did it produce?).

How it works

A scheduler reads job definitions (schedule + task) and triggers execution when the time arrives. For distributed schedulers, one node holds the 'is scheduler' lease and is the sole trigger (avoids double-fire). Complex schedulers (Airflow, Prefect) support DAGs — jobs that depend on other jobs. Advanced schedulers (Temporal, Cadence) support durable execution: the workflow's state is checkpointed to a database so it survives worker crashes and can wait days for a subsequent step.

Scaling characteristics

Simple schedulers (cron on one box) handle any single-node workload; the limit is worker fan-out, not the scheduler itself. Distributed schedulers scale by adding workers, keeping the scheduler thin. Airflow can trigger millions of jobs/day; Temporal handles millions of long-running workflows concurrently. Cost: scheduler state (DB) is small; worker pools scale linearly with job load.

When to use it

• Recurring reports and rollups
• Nightly ETL jobs
• Warmup or precomputation tasks (cache warming, feature computation)
• Delayed messages (send a follow-up email in 24 hours)
• Timeouts (a workflow that pauses for a user response, gives up after 7 days)
• Backfills of historical data

When NOT to use it

• Event-triggered work — use a queue, not a scheduler
• Sub-second recurrence — cron and most schedulers are minute-granular
• Ad-hoc one-off scripts — engineer running them manually is fine

Failure modes

• Double execution: two nodes both think they're the scheduler. Mitigate with distributed lock + heartbeat.
• Missed execution: scheduler node was down when the trigger time passed. Mitigate with catch-up mode (Airflow) or run-once semantics.
• Long-running job blocks the next scheduled run — always set a timeout + concurrency limit per job
• Job depends on data that isn't ready yet — Airflow has sensors; Temporal has waitFor
• Cron drift on a single machine (system clock issues) causes silent misfires

Alternatives

• cron on a single box — simplest for tiny scale
• Kubernetes CronJob — cron with pod isolation and retry; great for containerized workloads
• Airflow — for DAG-based ETL; heavy but powerful
• Temporal — for long-running, stateful workflows with retries built in
• AWS EventBridge / GCP Cloud Scheduler — managed cron for cloud native

Interview questions

• How do you prevent a scheduled job from running twice when you have 3 scheduler nodes?
• A daily job takes 26 hours some days and overlaps with the next run. What's your fix?
• Design a system to send 'follow-up in 24 hours' emails. Where does the scheduling state live?
• How would you retry a failed job — immediately, with backoff, or from a specific checkpoint?
• Explain how Airflow handles a DAG where one task fails.
• What's the difference between an idempotent job and one that just retries safely?