Recovery Config
Overview
Section titled “Overview”Tasks can become stale when:
- Worker process crashes mid-execution
- Network partition prevents heartbeats
- Worker machine goes down
Horsies automatically detects and recovers these tasks.
Basic Usage
Section titled “Basic Usage”from horsies.core.models.recovery import RecoveryConfig
config = AppConfig( broker=PostgresConfig(...), recovery=RecoveryConfig( auto_requeue_stale_claimed=True, claimed_stale_threshold_ms=120_000, auto_fail_stale_running=True, running_stale_threshold_ms=300_000, finalizing_stale_threshold_ms=300_000, ),)Fields
Section titled “Fields”| Field | Type | Default | Description |
|---|---|---|---|
auto_requeue_stale_claimed |
bool |
True |
Requeue tasks stuck in CLAIMED |
claimed_stale_threshold_ms |
int |
120,000 | Ms before CLAIMED task is stale |
auto_fail_stale_running |
bool |
True |
Fail tasks stuck in RUNNING |
running_stale_threshold_ms |
int |
300,000 | Ms before RUNNING task is stale |
finalizing_stale_threshold_ms |
int |
300,000 | Ms a completed child may remain in finalization handoff before recovery |
crashed_worker_recovery_grace_ms |
int |
10,000 | Grace before recovering a workflow task whose underlying task is terminal but whose workflow progression was not applied; 0 disables |
check_interval_ms |
int |
30,000 | How often to check for stale tasks |
runner_heartbeat_interval_ms |
int |
30,000 | How often the worker process writes the heartbeats of its RUNNING tasks |
claimer_heartbeat_interval_ms |
int |
30,000 | CLAIMED task heartbeat frequency |
worker_state_snapshot_interval_ms |
int |
30,000 | How often each worker persists a monitoring snapshot row to horsies_worker_states (1s–1min). The cap is half the 120 s live window, so a running worker stays live when a snapshot write is late by up to half the window. A larger value fails with HRS-204 |
auto_terminate_orphaned_workflow_tasks |
bool |
True |
Terminate claimed tasks whose workflow node linkage no longer exists |
phase2_quarantine_after_attempts |
int |
25 | Recovery passes an unresolvable crashed-worker progression row may retain before its evidence moves to the quarantine table and discovery stops retrying it (3–1,000) |
workflow_recovery_failure_limit |
int |
25 | Reaper passes in which the recovery of a workflow may raise a persistent error before the workflow fails closed (3–1,000) |
All time values for thresholds and intervals are in milliseconds. Retention windows are in hours; the sweep interval is in seconds.
Removed in 0.5.0 (setting either fails validation naming the successor):
heartbeat_retention_hours — heartbeats live in hourly partitions and old
partitions drop whole; a row-delete window no longer exists.
queue_terminal_record_retention_hours — terminal task records age by the
retention class assigned at enqueue; per-queue row-delete windows no longer
exist.
Retention and partition-coverage fields moved to
AppConfig.retention; this table lists only what
RecoveryConfig still owns.
Recovery Behaviors
Section titled “Recovery Behaviors”Stale CLAIMED Tasks
Section titled “Stale CLAIMED Tasks”When a task is CLAIMED but the claimer heartbeat stops:
- Safe to requeue: User code never started executing
- Task is reset to PENDING for another worker to claim
- Original worker may have crashed before dispatching
Stale RUNNING Tasks
Section titled “Stale RUNNING Tasks”When a regular task is RUNNING but the runner heartbeat stops:
- Not safe to blindly requeue: User code was executing, could have partial side effects
- If the task has a retry policy with
WORKER_CRASHEDinauto_retry_forand retries remaining: scheduled for retry (returns to PENDING withnext_retry_at) - Otherwise: marked as FAILED with
WORKER_CRASHEDerror
running_stale_threshold_ms is based on missing runner heartbeats, not total
task duration. Long tasks remain healthy as long as the worker writes their
heartbeats. With auto_fail_stale_running on, a worker that cannot write them
for half of running_stale_threshold_ms kills its tasks and exits non-zero, so
the reaper does not fail a task that still runs. That half must hold one
runner_heartbeat_interval_ms plus 2 s for a heartbeat to land, so the
threshold is at least 2 × (runner_heartbeat_interval_ms + 2_000). With it
off, the worker logs the failed writes and keeps running.
After user code returns, the child marks the task as finalizing before the
parent writes the terminal result. The reaper will not recover that task until
finalizing_stale_threshold_ms has also elapsed.
For workflow tasks, terminalization records the owed workflow progression in a transactional outbox as it moves the task to history, and the reaper consumes those records. Task finalization is two transactions — the task is terminalized first, then the workflow DAG is advanced — so a progression can be owed for a brief moment while its finalizer is still in flight. The reaper waits crashed_worker_recovery_grace_ms (default 10s) before consuming such a record, so it does not race a healthy finalizer; only a genuine crash in that gap is recovered (after the grace plus one reaper sweep). This grace is independent of the heartbeat-coupled thresholds. See Heartbeats & Recovery for details.
Unresolvable progression evidence quarantines
Section titled “Unresolvable progression evidence quarantines”A progression record whose disposition keeps refusing to resolve is retried for
phase2_quarantine_after_attempts recovery passes (default 25, bounds 3–1,000),
then moved to a quarantine table with its evidence preserved; discovery stops
retrying it. Retaining dispositions are either transient races, which resolve
within a pass or two, or structural conflicts, which never do — the bound gives
a transient an order of magnitude more passes than it needs while capping a
structural row’s error-log noise. Quarantined counts, rows over the attempt
bound, and the quarantine function’s refusals are published on the worker
health surface beside the phase-2 pass summary.
A progression that raises a persistent error (see the next section) counts
toward the same bound. At the bound the record moves to the quarantine table,
and the workflow fails closed with WORKFLOW_RECOVERY_LIMIT_REACHED, because
discovery stops and no other path ends it. A transient raise does not count.
Workflow recovery limit
Section titled “Workflow recovery limit”The reaper repairs a stuck workflow by running the repair again. A repair that
raises rolls back, so the next pass selects it again. Without a bound, a raise
that repeats would repeat on every pass, and the workflow would stay RUNNING.
A workflow fails closed after workflow_recovery_failure_limit reaper passes in
which its recovery raised a persistent error (default 25, bounds 3–1,000). At
the default check_interval_ms and the default limit, the limit is reached 24
intervals after the first pass, which is about 720 seconds.
- A transient raise does not count. A connection, lock, deadlock or timeout
error clears by itself, so a database outage cannot fail a workflow. The rule
is the type of the error:
OperationalError,InterfaceError, a pool timeout, a disconnect, and the error that follows a lost connection in the same session. A transient raise that repeats stays in the log and in the healtherrorscount. - A persistent raise counts once per pass. This covers a defect, stored data
that the engine cannot use, and a SQL error outside
OperationalError. Many poison nodes in one workflow add one count per pass. - The count is stored. It is in
horsies_workflow_recovery_failures, so a worker restart keeps it, and all workers share it. A pass that is lost also loses its count, so the bound moves later, never sooner. - The count has no reset, so a later unrelated raise after a repair reaches the limit sooner. Delete the row of a workflow to start its count again. The retention sweep deletes a row after its workflow is gone.
- The resume pass does not count. It logs the raise, and the reaper counts the same candidate.
At the limit, the write runs SQL only, because engine code is what raised:
- The node that raised goes
FAILEDwithWORKFLOW_RECOVERY_LIMIT_REACHED. - Other
PENDINGandREADYnodes goSKIPPED. - The workflow keeps a stored first error. If it has none, the limit error is stored.
- With no
ENQUEUEDorRUNNINGnode, the workflow goesFAILEDat once. - With a live node, the workflow stays
RUNNINGuntil the last live node ends, as foron_error=OnError.FAIL. If asuccess_policyis then met by nodes that completed, the workflow can endCOMPLETED. - The write at once does not evaluate
success_policy, because that evaluation is one of the paths that raise. on_error=OnError.PAUSEdoes not apply. A limit must end the workflow, and aPAUSEDworkflow does not end by default.
If the limit write fails, the count stays, the workflow stays RUNNING, and a
CRITICAL line names the workflow and the cause. The next raise tries the write
again. over_failure_limit on the worker health surface counts such workflows
and the ones with live nodes.
Heartbeat System
Section titled “Heartbeat System”Two heartbeat types:
- Claimer heartbeat: Sent by worker for CLAIMED tasks (not yet running)
- Runner heartbeat: Written by the worker process, from its own thread, for the RUNNING tasks that its children hold
The reaper (running in each worker) checks for missing heartbeats.
Threshold Guidelines
Section titled “Threshold Guidelines”| Threshold | Constraint |
|---|---|
Running stale, auto_fail_stale_running on |
Must be >= 2 × (runner heartbeat interval + 2 s) |
Running stale, auto_fail_stale_running off |
Must be >= 2x runner heartbeat interval |
| Finalizing stale | Must be >= 2x runner heartbeat interval |
| Claimed stale | Must be >= 2x claimer heartbeat interval |
| Claimed stale | 1 second to 1 hour |
| Running stale | 1 second to 2 hours |
| Check interval | 1 second to 10 minutes |
| Heartbeat intervals | 1 second to 2 minutes |
For CPU-Heavy Tasks
Section titled “For CPU-Heavy Tasks”A task that holds the GIL does not delay its runner heartbeats, because the worker process writes them. CPU-bound tasks need no extra margin.
For Quick Tasks
Section titled “For Quick Tasks”Fast tasks can use tighter thresholds:
RecoveryConfig( runner_heartbeat_interval_ms=10_000, # Heartbeat every 10s running_stale_threshold_ms=40_000, # 40s before considered stale)The worker stops when no heartbeat lands for 20 s, so a heartbeat may take up to 10 s to land.
Validation
Section titled “Validation”The config validates that thresholds are safe:
# This raises ConfigurationError (HRS-204):RecoveryConfig( runner_heartbeat_interval_ms=30_000, running_stale_threshold_ms=60_000, # Must be >= 64_000: 2 x (30s + 2s))Retention
Section titled “Retention”Retention moved to its own configuration section:
AppConfig.retention. It decides how long records
live and how their storage is prepared and reclaimed, which is a
different question from the one this page answers — recovery is about
work that went wrong, retention about work that went right and is now
history.
Setting a retention field on RecoveryConfig fails at construction and
names its new home; there is no alias.
Disabling Recovery
Section titled “Disabling Recovery”To disable automatic recovery (not recommended):
RecoveryConfig( auto_requeue_stale_claimed=False, auto_fail_stale_running=False,)Tasks will remain stuck until manually resolved.
Manual Recovery
Section titled “Manual Recovery”Query stale tasks via broker:
broker = app.get_broker()
# Find stale tasksstale = await broker.get_stale_tasks(stale_threshold_minutes=5)
# Manually fail stale RUNNING tasksfailed_count = await broker.mark_stale_tasks_as_failed(stale_threshold_ms=300_000)
# Manually requeue stale CLAIMED tasksrequeued_count = await broker.requeue_stale_claimed(stale_threshold_ms=120_000)