Scheduler Overview
Architecture
Section titled “Architecture”┌─────────────────────────────┐│ Scheduler Process ││ - Checks schedules ││ - Calculates next run ││ - Enqueues via broker │└──────────────┬──────────────┘ │ enqueue_async() ▼┌─────────────────────────────┐│ PostgreSQL ││ - tasks table ││ - schedule_state table │└──────────────┬──────────────┘ │ NOTIFY ▼┌─────────────────────────────┐│ Workers ││ - Execute scheduled tasks │└─────────────────────────────┘The scheduler:
- Runs as a separate process/dyno from workers
- Checks configured schedules at regular intervals
- Enqueues tasks when schedules are due
- Tracks state in database to prevent duplicates
Configuration
Section titled “Configuration”Add a ScheduleConfig to your AppConfig:
from horsies.core.models.schedule import ScheduleConfig, TaskSchedule, DailySchedulefrom datetime import time
config = AppConfig( broker=PostgresConfig(...), schedule=ScheduleConfig( enabled=True, check_interval_seconds=1, schedules=[ TaskSchedule( name="daily-cleanup", task_name="cleanup_old_data", pattern=DailySchedule(time=time(3, 0, 0)), timezone="UTC", ), ], ),)Running the Scheduler
Section titled “Running the Scheduler”horsies scheduler myapp.instance:app --loglevel=INFORun separately from workers. One scheduler per cluster is sufficient.
Key Concepts
Section titled “Key Concepts”Schedule Name
Section titled “Schedule Name”Each schedule has a unique name used for state tracking:
TaskSchedule( name="daily-report", # Must be unique task_name="generate_report", ...)Task Name
Section titled “Task Name”The task_name must match a registered @app.task():
@app.task("generate_report")def generate_report() -> TaskResult[str, TaskError]: ...
TaskSchedule( name="...", task_name="generate_report", # Must match decorator ...)Check Interval
Section titled “Check Interval”How often the scheduler checks for due schedules:
ScheduleConfig( check_interval_seconds=1, # Check every second ...)Range: 1-60 seconds. Lower values provide better precision but more database queries.
State Tracking
Section titled “State Tracking”The schedule_state table tracks:
| Field | Purpose |
|---|---|
schedule_name |
Schedule identifier |
last_run_at |
When schedule last executed |
next_run_at |
When schedule should run next |
last_task_id |
ID of most recent enqueued task |
run_count |
Total execution count |
config_hash |
Detects configuration changes |
This prevents duplicate executions when:
- Scheduler restarts
- Multiple schedulers run (advisory locks serialize)
- Network issues cause delays
A slot that already has a task
Section titled “A slot that already has a task”Each slot has a deterministic task id, and the row stores a fingerprint of the enqueue. A scheduler that stops after the enqueue of a slot and before it saves the state fires that slot again at its next start. With the same command, the enqueue returns the existing task. With a different command, the broker returns PAYLOAD_MISMATCH. The command is different when the next start runs another horsies version, or when the schedule’s kwargs or the task’s options changed.
The slot already has its task, so the scheduler counts the slot as fired:
- It logs one warning that names the schedule, the slot and the task id.
- It records the run with that task id and moves to the next slot.
- The existing task keeps the command that it was enqueued with.
Self-healing state rows
Section titled “Self-healing state rows”State rows are created at scheduler startup. If a schedule’s row
creation fails there (for example a transient database error during
start()), the schedule is logged as failed and the tick loop
recreates the missing row on a later tick — each tick checks for
enabled schedules without a state row and re-runs their initialization.
The same check recovers a row deleted externally. The recreated row’s
next_run_at is computed from the current time, so no missed-slot
backlog is replayed.
Catch-Up Logic
Section titled “Catch-Up Logic”When catch_up_missed=True, missed runs are executed:
TaskSchedule( name="hourly-sync", task_name="sync_data", pattern=IntervalSchedule(hours=1), catch_up_missed=True, # Execute missed runs)If the scheduler was down for 3 hours, it will enqueue 3 tasks on restart
(bounded by max_catch_up_runs).
When catch_up_missed=False (default), missed runs are dropped: after
downtime the scheduler fires at most one run — the most recent due slot —
logs how many older slots were skipped, and resumes at the next future
slot. Slot alignment is preserved, so task ids stay deterministic.
Timezone Support
Section titled “Timezone Support”Each schedule can have its own timezone:
TaskSchedule( name="morning-report", task_name="send_report", pattern=DailySchedule(time=time(9, 0, 0)), timezone="America/New_York", # 9 AM Eastern)Uses Python’s zoneinfo module. Default is “UTC”.
Validation
Section titled “Validation”Schedules are validated at startup:
- Task must be registered
- Queue must be valid (CUSTOM mode)
- Required arguments must be provided
- Kwargs must encode against the task’s types and fit
payload.reject_bytes
# This will fail at scheduler start:TaskSchedule( name="bad", task_name="nonexistent_task", # Not registered ...)The scheduler encodes the kwargs of each schedule once at startup and sends that text for every slot until the next start. A kwargs model field with a default_factory, or a validator that reads the clock, runs once per start, not once per fire. If such a validator refuses the value later, the task is enqueued and fails at the worker with WORKER_SERIALIZATION_ERROR.
Task Options
Section titled “Task Options”A scheduled run stores the options of its task, as a plain .send() of the task does. The worker reads timeout_ms, retry_policy, execution and cancel_grace_ms from the row, so a scheduled run gets the same timeout, retries and execution class as a sent one. A schedule has no options of its own: set them on @app.task().
@app.task() encodes the options once, when the task is defined. The scheduler takes that text at startup and sends it for every slot until the next start. If the options do not encode, @app.task() raises ValueError with the task name and the cause, and the task is not registered. horsies scheduler then exits 1 at startup: the startup check reports the module import with MODULE_EXEC_ERROR, and Scheduler.start() refuses the schedule with TASK_NOT_REGISTERED.
Graceful Shutdown
Section titled “Graceful Shutdown”The scheduler handles SIGTERM/SIGINT for clean shutdown:
# Sends SIGTERMkill <scheduler_pid>Current schedule check completes, then scheduler exits.
