Worker Architecture
Overview
Section titled “Overview”┌─────────────────────────────────────────────────────────┐│ Worker Process (Main) ││ - Subscribes to NOTIFY channels ││ - Claims tasks from PostgreSQL ││ - Assigns tasks to child slots ││ - Writes runner heartbeats (own thread, not the loop) ││ - Writes results back ││ ││ ┌─────────────────────────────────────────────────┐ ││ │ Slots (one child, one socket each) │ ││ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ ││ │ │ Child 1 │ │ Child 2 │ │ Child N │ ... │ ││ │ │ Task A │ │ Task B │ │ (idle) │ │ ││ │ └─────────┘ └─────────┘ └─────────┘ │ ││ └─────────────────────────────────────────────────┘ │└─────────────────────────────────────────────────────────┘Process Model
Section titled “Process Model”A worker owns one child process per slot:
- Main process: Coordinates claiming, dispatching, result handling, and writes the runner heartbeats of the running tasks
- Isolated children: Execute task code in isolation, one task at a time. Set by
IsolatedChildren(count=N)in the worker profile - Shared async children: Execute up to
seatsSHARED_ASYNCtasks at once, as coroutines on one event loop. Set bySharedAsyncChildren(count=N, seats=K). See Shared Async Children - Shared thread children: Execute up to
seatsSHARED_THREADtasks at once, each on a thread of its own. Set bySharedThreadChildren(count=N, seats=K). See Shared Thread Children
Each child:
- Starts from the preload parent on Linux, and with
execon other platforms. It inherits no socket or lock from the worker. - Has its own session, so a kill also reaches the subprocesses of its task.
- Has its own socket to the worker. Tasks and results move over it as JSON frames.
Benefits of isolated children:
- GIL doesn’t block concurrent task execution
- A crash, a timeout, or an exit in one child affects only the task of that child
- Memory isolation between tasks
Shared children trade these for more tasks per process. See Choosing an Execution Mode.
Failure Isolation
Section titled “Failure Isolation”Each isolated child holds one task, so a failure stays with that task. A shared child holds the tasks of its seats, so a crash or a kill of it ends every task in it; see Shared Async Children and Shared Thread Children.
| Case | Result |
|---|---|
A task exceeds timeout_ms |
Its child and the subprocesses in its session are killed. After the child exits, the task fails with TASK_TIMEOUT. Other tasks continue |
A child crashes (segfault, out-of-memory kill, os._exit()) |
The worker sees the exit at once and recovers that task. Other tasks continue |
| The worker dies | Each child stops at once, also while C code holds the GIL. A subprocess that a task started keeps running, because the child cannot act after a SIGKILL |
| A supervisor sends SIGTERM to every process | Running tasks continue, and the worker drains them |
| A task forks a helper, then its child dies | The worker sees the death at once. The helper does not hide it |
| A timeout fires as a task ends | The kill is bound to that task, so it cannot reach the next task. The killed child takes no next task |
An async def task returns while asyncio tasks that it started are still on the loop |
The child waits up to 1 s for them, then cancels them in three rounds 0.1 s apart. The worker writes what the task returned after that work ended, and replaces the child. The deadline no longer applies. Work still alive after the rounds, or no report within 26.3 s of the return: the worker kills the child, and writes what the task returned after the exit. See Async Tasks |
| A task holds the GIL in C code | Its runner heartbeats continue: the worker process writes them, so the reaper does not fail the task |
The worker cannot write runner heartbeats for half of running_stale_threshold_ms, or its heartbeat thread dies |
With auto_fail_stale_running on, the worker kills every child that holds a task, stops, and exits non-zero. With it off, the worker logs it and runs on. See When the Worker Cannot Write Heartbeats |
Execution Flow
Section titled “Execution Flow”1. Startup
Section titled “1. Startup”# In CLIhorsies worker myapp.instance:app --profile default- Load app from module
- Import task modules (for registry population)
- Start the preload parent (Linux)
- Start N children (each loads app independently). A child whose app has an
async deftask creates its event loop before it reports ready - Wait until each child reports ready
- Subscribe to NOTIFY channels
2. Main Loop
Section titled “2. Main Loop”while not stopped: await claim_and_dispatch_all() await wait_for_any_notify()- Claim pass: Query for available tasks, claim them
- Dispatch: Assign claimed tasks to child slots, oldest first
- Wait: Block on NOTIFY until new tasks arrive
- Repeat
3. Claiming
Section titled “3. Claiming”One stream per queue and execution class, simplified:
WITH next AS ( SELECT id FROM tasks WHERE queue_name = :queue AND execution = :class AND status = 'PENDING' AND enqueued_at <= now() AND (good_until IS NULL OR good_until > now()) ORDER BY priority ASC, enqueued_at ASC, id ASC FOR UPDATE SKIP LOCKED LIMIT :take)UPDATE tasks SET status='CLAIMED', claimed_at=now()FROM next WHERE tasks.id = next.idRETURNING tasks.id, tasks.executionKey features:
FOR UPDATE SKIP LOCKED: Non-blocking concurrent claiming- One budget per execution class that the worker runs children for; see Budgets per Execution Class
- Priority ordering within one class: Lower priority number = higher priority
- FIFO within same priority
- Expiry check: Skip tasks past
good_until - Class check: a claimed row of a shared class whose registered task has
another class goes back to the queue as
ISOLATED
4. Task Execution
Section titled “4. Task Execution”In child process:
- Set the row RUNNING, with the first runner heartbeat in the same statement. A workflow task also sets its node RUNNING, in one transaction with the row
- Resolve task from registry by name
- Deserialize arguments
- Call task function. An
async deftask runs on the event loop that the child created at boot. The child keeps that loop until it stops, and the loop runs only while an async task runs - Serialize the result
- Mark the row finalizing, and report the result. When the body of an
async deftask returned with asyncio tasks still on the loop, the child reports the return with the result, waits up to 1 second for that work, and cancels the work that is still alive. When that work has ended, it marks the row finalizing and reports how the work ended; work that did not stop gets no marker. The worker writes the result only after that report, and replaces the child
The child runs no heartbeat thread. While the child holds the task, a thread of the worker process writes its runner heartbeats. See Heartbeats & Recovery.
5. Result Handling
Section titled “5. Result Handling”Main process receives result from child:
- Parse TaskResult (ok or err)
- Check for retry eligibility
- Update task status (COMPLETED or FAILED)
- Send NOTIFY for result waiters
A plain task that succeeded is written in a batch. Its finalizer hands the
outcome to the finalize batcher of the worker process and waits for the answer
of its own row. One statement in autocommit writes up to 64 rows
(horsies_complete_tasks_fused; a batch of one row runs
horsies_complete_task_fused itself). For each row, in task-id order, it
checks the claim generation, writes the attempt, moves the row to the history,
and wakes the workers of the queue. That is one database round trip per batch
with the defaults (PostgresConfig.worker_pool_pre_ping=False).
- The batcher has no timer. Two flushers write whenever rows wait, each on a connection of the worker pool. A lone row is written at once. The rows that come while both flushers write form the next batch.
- When a full batch of 64 rows waits while both flushers write, one more flusher starts, up to the size of the worker pool. Such a flusher writes full batches only, so it never makes a batch smaller.
- A finalizer ends, and frees its seat, only after the statement of its batch returned. So a seat is free only after the commit of its row.
- A row that the server refuses (for example a row whose retention class has no history partition) refuses the whole statement, and no row of it commits. The batcher then writes each row of that batch alone, once: the other rows commit, and the refused row takes a phase-1 error.
- A lost reply is a retryable error for each row of the batch. The run that retries a row finds it already applied when the batch committed.
- At a stop, the batcher closes after the finalizers: it writes the queued rows, for at most 10 seconds.
A workflow task is written in two phases, each with its own commit, so an error in phase 2 never undoes phase 1.
- Phase 1, a success: one statement in autocommit
(
horsies_finish_workflow_success). It takes the task advisory lock, then the row lock under the claim generation, writes the attempt, and moves the row to the history with the phase-2 record of its node. It sends no wake. A failure, a timeout or a crash is the read of the retry facts and one call ofhorsies_finish_failed_attempt. - Phase 2: one transaction that holds the workflow row lock from its first statement to its COMMIT. The first statement writes the node, deletes its phase-2 record, sends the capacity wake of the task’s queue, which the COMMIT delivers, and returns whether a node of the workflow waits for this node. The engine then applies the failure policy, promotes the dependents and checks the completion of the workflow. A node that no node waits for does not read its dependents. One update moves the promoted dependents from PENDING to ENQUEUED, and the task of each one is inserted and linked in one statement per node, all in one round trip.
- A lost reply in phase 1 is a retryable error. The run that retries it finds the task already applied, reads the stored result, and runs phase 2. A worker that stops between the phases leaves the phase-2 record, and the recovery pass runs phase 2.
Round trips with the defaults (worker_pool_pre_ping=False; with it on, one
more per phase whose connection waited in the pool for more than 1 s):
| Finalize of a workflow task | Phase 1 | Phase 2 |
|---|---|---|
| Success, promotes its dependents | 1 | 8 |
| Success, the last node of the workflow | 1 | 7 |
Failure, its dependent is skipped (on_error=FAIL) |
2 | 12 |
Failure, the last node of the workflow (on_error=FAIL) |
2 | 8 |
Shared Async Children
Section titled “Shared Async Children”A shared async child runs SHARED_ASYNC tasks of the claimed rows. It has four threads:
- The control thread reads the frames of the worker. It hands each frame to the event loop, so EOF still means the worker is gone, whatever the loop does.
- The main thread runs the event loop from the end of the start steps to the stop frame. A run frame starts its task at once on a free seat, as an asyncio task, while other tasks run.
- The writer thread writes the frames of the child. The loop queues a frame and returns, so a worker that reads slowly never stops the loop.
- The watchdog thread watches the event loop. When the loop has not turned for 5 seconds, it names the task that holds the loop, with the source line, and reports it to the worker through the writer thread.
The start steps, after the import of the app and the on_child_process_start hooks: the child opens an async connection pool on its loop, runs the on_shared_async_child_start hooks, and sends its ready frame. Each task then runs on the loop: the RUNNING update with the first runner heartbeat, the task body, and the finalizing marker. Each write goes through the async pool, so no write of the child blocks the loop.
The worker places each SHARED_ASYNC task on the open shared child with the fewest tasks in flight and a free seat. Each result frame names its run, and a free seat takes the next queued task at once. The runner heartbeats of every task of the child come from the worker process, as for an isolated child.
| Case | Result |
|---|---|
A task reaches timeout_ms |
The child cancels it. When its coroutine and the work that it started have ended, the worker writes TASK_TIMEOUT, the retry and the next workflow step. Other tasks of the child continue |
| A task returns a value after its deadline fired | The outcome is TASK_TIMEOUT |
A task is still alive cancel_grace_ms after its cancel |
The child cancels it a second time, with the asyncio tasks that it started, in up to three rounds 0.1 s apart. The child takes no new task, and a new child replaces it when its tasks end |
| A task is still alive after the cancel rounds | The task is stuck and keeps its seat. The worker kills the child when every task left in it is stuck; each fails with TASK_TIMEOUT, and its retry runs in an isolated child |
| The event loop has not turned for 5 s | The worker kills the child at once. The task that held the loop fails with TASK_BLOCKED_LOOP and retries isolated; each other task before its deadline fails with NEIGHBOUR_BLOCKED_LOOP; a task past its deadline with TASK_TIMEOUT |
The child reports neither the second cancel nor the end of a task by timeout_ms + cancel_grace_ms + 10 s |
C code holds the GIL in the child. The worker kills it: each task past its deadline fails with TASK_TIMEOUT, and every other task takes the crash path. A task that the child reported stuck retries in an isolated child |
The child reported the second cancel of a task, or the cancel of the work that a task left, then reports neither the task stuck nor its end by timeout_ms + cancel_grace_ms + 20.3 s |
The child froze after its report. The worker kills it, with the same outcomes as in the row above |
| The child crashes | Every task of the child takes the crash path, after the worker saw the exit. A task that returned before its deadline, and whose left work the child reported stuck, ends with what it returned. A new child starts |
| The child reaches a recycle limit | It takes no new task, its tasks finish, then it stops and a new child starts |
At a stop the worker closes each shared child: it takes no new task, and it exits by stop frame when its tasks have ended. At the drain deadline the worker kills a shared child that still runs tasks: each of them past its deadline fails with TASK_TIMEOUT, and every other one takes the crash path. A task that the worker already named as the cause of a stuck stop (a task that a shared async child reported stuck, or the task whose deadline closed a shared thread child) retries in an isolated child at this kill too.
Shared Thread Children
Section titled “Shared Thread Children”A shared thread child runs SHARED_THREAD tasks of the claimed rows. Its threads:
- The control thread reads the frames of the worker and hands each one to the main thread.
- The main thread starts the deadline of each run and hands the run to a free seat.
seatsseat threads, started at boot. Each one runs one task at a time, and takes the next run when its task has ended.- The deadline thread reports each deadline at its time.
- The writer thread writes the frames of the child.
The child boots as an isolated child does: the import of the app, the on_child_process_start hooks, and one sync connection pool. Then it starts its threads and sends its ready frame. A refused thread fails the boot (ThreadStartRefused), and that cause can clear. Each task runs on its seat thread: the RUNNING update with the first runner heartbeat, the task body, and the finalizing marker. The seat threads share the sync pool of the child.
The worker places each SHARED_THREAD task on the open shared thread child with the fewest tasks in flight and a free seat. The runner heartbeats of every task of the child come from the worker process.
| Case | Result |
|---|---|
A task reaches timeout_ms |
The child reports the deadline, and the task runs on: a thread cannot be cancelled. The child closes: it takes no new task |
| No task in a closed child has time left | The worker kills the child with its session. After the exit, each task that was still in it fails with TASK_TIMEOUT, with the retry and the next workflow step; the task whose deadline closed the child retries in an isolated child. A new child starts |
| A task ends by itself after its deadline | The outcome is TASK_TIMEOUT |
| A task in the closed child still has time left | It runs to its end or to its own deadline |
The child does not report a deadline by timeout_ms + 10 s |
C code holds the GIL, and no thread of the child runs. The worker kills it: each task past its deadline fails with TASK_TIMEOUT, and every other task of the child takes the crash path. The task whose deadline closed the child, when one did, retries in an isolated child |
| The child crashes or is killed | Every task of the child takes the crash path, after the worker saw the exit. A new child starts |
| The child reaches a recycle limit | It takes no new task, its tasks finish, then it stops and a new child starts |
See Shared Thread Tasks.
Child Process Initialization
Section titled “Child Process Initialization”Each child process:
- Reads its settings from the worker over its socket. The database URL is not in
argv, becausepsshowsargv. - Loads app via
_child_initializer() - Imports task modules
- Populates local task registry
- Suppresses sends during import (prevents side effects)
def _child_initializer(app_locator, imports): app = _locate_app(app_locator) set_current_app(app) app.suppress_sends(True) for module in imports: import_module(module) app.suppress_sends(False)A child differs from a multiprocessing process:
multiprocessing.parent_process()returnsNone.multiprocessing.current_process().nameishorsies-child-<slot>.__main__is the child program, not your script. The worker refuses a__main__:applocator at startup.
Preload Parent
Section titled “Preload Parent”On Linux, a worker starts one extra process: the preload parent. It imports code once, then forks each child. A child then starts in milliseconds and shares those memory pages with the other children.
WorkerProfile.preload selects what it imports:
| Mode | Imports | Use it when |
|---|---|---|
library (default) |
Horsies and its dependencies | Always. Each child still imports your app |
app |
Horsies, its dependencies, and your app | Your app has no side effect at import |
none |
Nothing, and there is no preload parent | You must rule the preload parent out. Every child starts with exec |
Conditions: 8 children, Linux, tasks that run pydantic, SQLAlchemy, and pandas (pandas app only), Postgres in the loop. Memory is the summed PSS of the children and the preload parent.
| App | preload |
Next child, p50 | Memory |
|---|---|---|---|
| Small | none |
618 ms | 461 MiB |
| Small | library |
27 ms | 169 MiB |
| Small | app |
20 ms | 151 MiB |
| With pandas | none |
1,174 ms | 715 MiB |
| With pandas | library |
486 ms | 472 MiB |
| With pandas | app |
26 ms | 236 MiB |
With app:
- Your app’s module code runs once, in the preload parent, before any child exists. Put per-child setup in an
on_child_process_starthook, which still runs in every child. - The worker refuses to start if the import leaves a Python thread or a socket. A fork copies a lock in whatever state another thread left it, and all children would share one socket.
With 4 children, the whole worker at start, including its main process, is measured as summed PSS:
| App | none |
library |
app |
|---|---|---|---|
| Small | 290 MiB | 159 MiB | 146 MiB |
| With pandas | 445 MiB | 342 MiB | 212 MiB |
Throughput is on the performance page.
The worker stays the real parent of each child, so exit codes, kills, and parent-death behavior do not change.
If the preload parent dies, running children continue, and new children start with exec. The worker replaces it after 5 seconds, up to 5 times. In ps, a forked child shows the command line of the preload parent.
Other platforms have no preload parent, and every child starts with exec. Linux also falls back to exec and logs a warning if the kernel has no pidfd_open (below 5.3) or blocks prctl.
Choosing a Mode
Section titled “Choosing a Mode”Keep library unless memory is your limit. Most of the gain from library is Horsies and its dependencies, so an app with heavy imports gains little from it. With 4 children, an app that imports boto3, openai, google-genai, firebase-admin, and lxml uses 651 MiB on 0.4.6, 575 MiB with library, and 298 MiB with app.
Before you use app in production, run it on staging with real calls to every client library inside tasks. An import that passes the guard does not prove that a library works after a fork.
Limits of Each Mode
Section titled “Limits of Each Mode”Both preload modes:
| Limit | Effect |
|---|---|
| Library code loads before your app | Anything your app sets first arrives too late. See Things to Avoid |
| A recycled child does not reload library code | After an in-place package upgrade, restart the worker |
| Copied pages grow as tasks run | Python reference counts write to shared pages, so memory rises after start |
| One more process | The preload parent uses about 20–30 MiB. The tables above include it |
app only:
| Limit | Effect |
|---|---|
| Your module code runs once | Anything built at import is shared by all children |
| A recycled child does not reload your code | New app code needs a worker restart. Recycling still returns memory |
| The guard does not see native threads | A BLAS pool or a gRPC client built at import passes the guard. It may hang or fail in a child |
| Native libraries may not be fork-safe | Build their clients in a hook, after the fork |
Worker Configuration
Section titled “Worker Configuration”horsies worker builds the internal WorkerConfig from the app config and the chosen worker profile:
| Field | Source |
|---|---|
dsn |
PostgresConfig.database_url |
session_dsn |
PostgresConfig.session_database_url, or database_url |
pgbouncer_transaction_mode |
PostgresConfig.pgbouncer_transaction_mode. Disables child-process prepared statements for PgBouncer transaction pooling |
queues |
The queues of the app |
processes |
IsolatedChildren.count: the number of isolated children, 0 with NoIsolatedChildren() |
max_tasks_per_child |
IsolatedRecycle.after |
max_memory_per_child_mb |
IsolatedRecycle.above |
shared_async |
WorkerProfile.shared_async |
shared_thread |
WorkerProfile.shared_thread |
max_claim_batch |
ClaimLimits.per_queue_per_pass |
max_claim_per_worker |
ClaimLimits.claimed_per_worker |
drain_timeout_s |
WorkerProfile.drain_timeout |
child_preload |
WorkerProfile.preload |
parent_pool_size |
WorkerProfile.pool, resolved. The pool of the worker process has no overflow |
isolated_pool |
IsolatedChildren.pool, resolved; the default IsolatedPool() with NoIsolatedChildren(). The pool of a shared child comes from shared_async.pool or shared_thread.pool |
isolated_task_send_pool |
IsolatedChildren.task_send_pool; the default with NoIsolatedChildren(). The task-send pool of a shared child comes from shared_async.task_send_pool or shared_thread.task_send_pool |
app_locator |
The app locator of the command |
imports |
Task module paths |
Workers require PostgreSQL LISTEN for dispatch. When using transaction-pooled
PgBouncer, configure PostgresConfig.session_database_url with a direct or
session-capable URL. Workers fail startup instead of silently switching to
poll-only dispatch if LISTEN is unavailable.
Child database pools are initialized inside the child process. Psycopg
connections are not process-safe. A child starts with exec, so it inherits no
parent database socket. Child DB connections remain lazy by default: the min_size of each child
pool is 0. Each connection of a child pool is in autocommit: the RUNNING update
of a plain task and the finalizing mark are one statement each, and one round
trip each with the checkout check off (the default). The RUNNING update of a
workflow task and the start of its node run in one transaction. An isolated
child uses one connection at a time: for the RUNNING
update, and for the finalizing mark. Its pool is IsolatedChildren.pool, 0 to
2 connections by default; see
Connection Pools and the Default Executor. The seats of a
shared child share the pool of the child for the same writes. See
Connections. The worker process holds one more
connection, for the runner heartbeats.
Graceful Shutdown
Section titled “Graceful Shutdown”On SIGTERM/SIGINT the worker:
- Stops new claims
- Returns tasks that wait for a child or a seat to
PENDING, because they have not started - Waits up to the
drain_timeoutof the profile for running tasks (default 300 seconds). A shared child takes no new task in this time - Kills the children that still run a task. Each of these tasks takes the child crash path; for a shared child, every task in it
- Waits for pending result writes
- Closes database connections
Set drain_timeout below the stop window of your supervisor, and leave a few seconds for result writes. If the supervisor kills the worker first, its running tasks wait for the reaper. At the deadline, the worker records each killed task at once. A task whose child does not exit within 30 seconds of the kill is left to stale-task recovery, or, with stale RUNNING recovery off, written when the worker sees the exit before it stops.
| Supervisor | Default stop window | Signal goes to |
|---|---|---|
| Fly.io | 5 s (kill_timeout), SIGINT |
Main process |
| Docker | 10 s (docker stop --time) |
Main process |
| Kubernetes | 30 s (terminationGracePeriodSeconds) |
Main process |
| Amazon ECS | 30 s (stopTimeout) |
Main process |
| Heroku | 30 s | Every process |
| systemd | 90 s (TimeoutStopSec) |
Every process, or the main process with KillMode=mixed |
Children ignore SIGTERM and SIGINT. The worker decides when a child stops, so a task continues when a supervisor signals every process. A subprocess that a task starts does not ignore these signals. To keep such a subprocess running on systemd, set KillMode=mixed.
If the worker process dies, each child stops at once. A child that continues has no way to write its result, and a retry could overlap it. The reaper recovers these rows. See Recovery Time After a Worker Stops.
Resilience
Section titled “Resilience”Workers recover from transient infrastructure failures without operator intervention. Configure via WorkerResilienceConfig in AppConfig.
What It Handles
Section titled “What It Handles”- Database outages: Worker retries on connection loss during startup, claiming, and result writes. No crash, no manual restart.
- Silent NOTIFY channels: If the PostgreSQL NOTIFY mechanism stops delivering (broken listener connection, dropped trigger), the worker falls back to periodic polling so tasks are still picked up.
- Child loss: If a child dies, the worker starts a new child in that slot. Other children continue. A task still in CLAIMED status is requeued back to PENDING. A task already in RUNNING status is recovered through the same retry policy used by the reaper: if
WORKER_CRASHEDis listed inauto_retry_forand retries remain, the task is scheduled for retry; otherwise it is marked FAILED with aWORKER_CRASHEDresult. - Child start failure: The cause of the failure decides what the worker does. See Child Start Failures.
Child Start Failures
Section titled “Child Start Failures”A child that cannot start reports the cause of the failure to the worker, before it exits. The worker sees the causes that the child cannot report: a spawn that the OS refuses, a child that dies with no report, and a child that is not ready in time.
| Cause | Class | What the worker does |
|---|---|---|
The app or a task module does not import (AppLoadFailed) |
Repeats | Counts an attempt |
The child connection pool cannot be set up, for example a URL that does not parse (ChildPoolSetupFailed) |
Repeats | Counts an attempt |
The child dies with no report: a crash in native code, os._exit() in app code (LostBeforeReady) |
Repeats | Counts an attempt |
No ready report within 120 seconds (NotReadyInTime) |
Repeats | Counts an attempt |
The child cannot create its event loop for async def tasks for a cause other than the errno values below, for example an event loop policy that raises (TaskLoopCreateFailed) |
Repeats | Counts an attempt |
The OS refuses to start the child process with another errno, for example ENOENT or EACCES for the interpreter, or a forked child does not reach the worker in 5 seconds (SpawnFailed) |
Repeats | Counts an attempt |
An on_child_process_start hook fails |
Repeats | Stops the worker at once |
A start step of the child or of the worker (the socket pair, the spawn, the event loop) fails with OSError EMFILE, ENFILE, ENOMEM or EAGAIN (ResourceShortage) |
Can clear | Takes the slot down |
The OS refuses a thread that the child starts at boot: the control thread, the watchdog of a start hook, a thread of the connection pool (ThreadStartRefused) |
Can clear | Takes the slot down |
A child that dies with no report names no cause, so the worker counts it as a cause that repeats. The worker cannot tell a crash that clears from a crash in the code. Counting it stops a worker whose code crashes at every start, instead of keeping a slot down with no end.
- After three counted attempts in a row, the worker exits non-zero, so that its supervisor restarts it. This also applies when other children serve tasks.
- A slot that is down takes no task, and the claim budget leaves it out: a worker with 4 processes and 1 slot down claims as a worker with 3 processes. The worker starts a new child for that slot after 1 second, then 2, 4, up to 60 seconds between attempts. Each failed attempt logs one line with the slot, the cause, and the next wait. The slot returns to service when a child starts.
- A cause that can clear counts an attempt only when every other slot is down too. A worker that serves no task must fail where its supervisor can see it. A failed attempt that does not count, because another slot is in service, starts the count again. A worker with 1 process has no other slot, so each of its failed attempts counts.
- At the first start, the worker starts when each slot has a ready child or is down, and at least one child is ready. Its worker state names the slots that are down. When no child starts, the worker exits non-zero.
- An outside SIGKILL of a booting replacement child, such as an out-of-memory kill, does not count an attempt while another child is serving: the worker waits up to 5 seconds and starts another. A kill during the first start counts, and so does a kill when no other child is serving.
Configuration
Section titled “Configuration”from horsies import WorkerResilienceConfig
config = AppConfig( broker=broker, resilience=WorkerResilienceConfig( db_retry_initial_ms=500, # initial backoff on DB errors db_retry_max_ms=30_000, # backoff cap db_retry_max_attempts=0, # 0 = retry forever notify_poll_interval_ms=5_000, # poll fallback when NOTIFY is silent ),)| Field | Type | Default | Description |
|---|---|---|---|
db_retry_initial_ms |
int |
500 |
Initial backoff delay (100–60,000ms) |
db_retry_max_ms |
int |
30000 |
Maximum backoff delay (500–300,000ms) |
db_retry_max_attempts |
int |
0 |
Max retry attempts; 0 = infinite (0–10,000) |
notify_poll_interval_ms |
int |
5000 |
Poll interval when NOTIFY is silent (1,000–300,000ms) |
Backoff uses exponential growth (initial_ms * 2^attempt) with ±25% jitter, capped at db_retry_max_ms. Backoff resets after each successful loop iteration.
Module Discovery
Section titled “Module Discovery”Workers need to know where tasks are defined:
# In instance.py# Dotted module paths (recommended)app.discover_tasks(["myapp.tasks", "myapp.jobs.worker_tasks"])
# Or file pathsapp.discover_tasks(["tasks.py", "more_tasks.py"])This list is passed to child processes for import.
Error Handling
Section titled “Error Handling”| Error Type | Handling |
|---|---|
| Task exception | Wrapped in TaskResult(err=…) |
| Serialization error | WORKER_SERIALIZATION_ERROR |
| Task not found | WORKER_RESOLUTION_ERROR |
| Child crash | Detected at once, WORKER_CRASHED for that task only |
Multiple Workers
Section titled “Multiple Workers”Run multiple worker processes for horizontal scaling:
# On machine 1horsies worker myapp.instance:app --profile default
# On machine 2horsies worker myapp.instance:app --profile default- Each worker claims independently
SKIP LOCKEDprevents duplicate claims- Advisory locks serialize certain operations
cluster_wide_caplimits total concurrency
Things to Avoid
Section titled “Things to Avoid”Don’t build a client at import with preload=ChildPreload.APP. The client exists before the fork, and a client with native threads, such as gRPC, can hang in a child.
# Wrong: this runs once, in the preload parentclient = firestore.Client()
# Correct: this runs in every child, after the forkclient = None
@app.on_child_process_startdef build_client() -> None: global client client = firestore.Client()Don’t set a library’s import-time setting in .env that your app loads. library and app both import psycopg, SQLAlchemy, and pydantic before your app runs, so load_dotenv() in your app is too late for PSYCOPG_IMPL. Set such a value in the process environment.
# Wrong: in a .env file that the app loadsPSYCOPG_IMPL=binary
# Correct: in the environment of the worker processPSYCOPG_IMPL=binary horsies worker myapp.instance:app --profile defaultMonkeypatching has the same limit: a patch that must run before those imports, such as gevent, is too late. Use preload=ChildPreload.NONE.
Don’t expect a recycle to load new code with preload=ChildPreload.APP. A recycled child is a new fork of the same preload parent. Restart the worker.
Don’t count children by process name. A forked child shows the command line of the preload parent, and there is one more process than the profile has children. Read processes in worker health.
Don’t add children_memory_mb to get the real footprint. It sums RSS, so with preload it counts each shared page once per child. Read the container or cgroup total.
