Design Job Scheduler

1. ⏰ What is a distributed job scheduler?

A distributed job scheduler receives instructions such as "run this task every day at 02:00 UTC" and ensures that the task is triggered at the intended time. On a single host, this is relatively simple. With multiple scheduler and worker machines sharing responsibility, the system must survive failures without losing a scheduled run or committing the same run twice.

2. Requirements

Design a scheduler that supports immediate, one-time, and recurring jobs. Throughout the design, consider the following central problem: if multiple machines determine that one job is ready, how can the system create exactly one durable record for that scheduled occurrence and recover it safely when a failure happens?

Functional requirements

• Users must be able to create immediate, one-time, or recurring UTC jobs, including parameters, timeouts, and retry policies.
• For every occurrence that becomes due, the platform must create one durable slot and recreate missed slots after downtime.
• Each slot must retain its claims, attempts, timeouts, and final outcome.
• The system must identify failed workers, reclaim their work, and enforce cancellation or retry policy for jobs that run for a long time.
• Users and operators must be able to examine job definitions, execution history, lag, and final outcomes.

Non-functional requirements

• During nominal operation, a due occurrence should begin within approximately 2 seconds of its scheduled fire time.
• Following an outage or restart, missed occurrences must be backfilled from durable state instead of being discarded.
• Every (job_id, scheduled_fire_time) must correspond to one durable slot and no more than one committed success.
• Exactly-once effects in external systems still depend on idempotent handlers or transactional integration beyond this core.
• The design should support roughly 10k execution starts per second on a sustained basis, with 5-10x burst skew at minute or hour boundaries.
• Scheduler nodes and worker nodes must scale independently, using coordinated shard ownership and concurrency limits driven by lag.
• Losing schedulers, workers, queues, or individual hosts must neither discard accepted schedule intent nor allow work to remain permanently stuck without escalation.
• Queue lag, missed-fire backlog, retry counts, lease timeouts, and dead-letter volume must be observable so that correctness and capacity can be investigated.

Back-of-the-envelope estimation

Treat these figures as planning assumptions. They provide concrete inputs for reasoning about partitioning, queue depth, and recovery behavior in the sections that follow.

[!IMPORTANT] The central tension is that many scheduler nodes can independently discover that a job is due, while only one durable record for that scheduled occurrence should reach a final result. Keep separate the description of what should execute, the identity of the machine currently scheduling it, and the worker authorized to complete it.

Out of scope (below the line)

• User-facing pause, resume, or arbitrary reschedule operations after a job has been created. Internal cancellation caused by timeouts remains in scope.
• Complete directed acyclic graph (DAG) workflow orchestration, including dependencies, human approval, or conditional branching.
• Uploading arbitrary user code and designing its sandbox. Tasks are pre-registered handlers from a catalog.
• Global active-active multi-region scheduling without a single-region authority boundary.