Changelog
The format is based on Keep a Changelog. horsies is pre-1.0: breaking changes may land in minor or patch releases, and there is no migration contract between pre-1.0 versions.
Unreleased
Section titled “Unreleased”0.7.0 — 2026-11-15
Section titled “0.7.0 — 2026-11-15”0.7.0 runs a task in one of three execution modes: in an isolated child,
in a shared async child, or in a shared thread child. A task names its
mode, and a worker profile sets the children of each kind; a worker starts
with --profile, which replaces the execution flags. A task takes fewer
round trips, and plain successes are written in batches. The schema moves
from v52 to v61, and the first 0.7.0 process that connects upgrades it.
The release is breaking: read the migration guide to 0.7.0 before you
upgrade.
-
WorkerProfile.parent_cpugives the worker process a core of its own.ParentCpu.OWN_CORE(Linux) puts every thread of the worker process on one physical core of the CPUs that the worker may use, both SMT threads, and every child, recycled children too, on the other CPUs. The worker sets the CPUs of each child before the child’s first frame. The worker start fails on another platform or with fewer than 2 physical cores.ParentCpu.SHARED(the default) changes nothing. The children lose the CPUs of that core: on 8 CPUs (4 physical cores with 2 SMT threads each), 8 isolated children that ran pure-Python work went from 52 to 39 tasks/s, and one shared thread child with 8 seats that ran C work that releases the GIL went from 71 to 56. Use it when the worker process is the limit. -
A task can be an
async deffunction.@app.task()registers it and returns anAsyncTaskFunction. A worker child runs it as it runs adeftask: one task at a time, with the same RUNNING update, heartbeats,timeout_mskill, recovery and retry policy. Each child keeps one event loop until the child stops, and an async task runs on that loop, so an async client that a module keeps stays open across the tasks of one child. The loop runs only while an async task runs. Adeftask still runs with no running event loop and can callasyncio.run(). In an async task, send withsend_async(): the syncsend()returnsErr(TaskSendAsyncContext)there.- A direct call of an async task returns a coroutine of the
TaskResult. The return check and the exception mappers are the same as for adeftask. In a worker, aCancelledErrorof a body that cancelled its own task, and aGeneratorExitthat the body raises, go through the exception mappers too; the direct call lets them pass. TaskNode.fn,.node()and the task registry take both kinds.- An
async deffunction that usesyieldis refused withHRS-101, as a generator function is. - A child whose app has an
async deftask creates its event loop at boot, before it reports ready. A child whose app has none creates no loop. When the loop cannot be created, the child fails its boot, and no task reaches it. AnOSErrorwithEMFILE,ENFILE,ENOMEMorEAGAINcan clear, so only that slot is down (see the change to child start failures below). The log names the slot and the cause, for exampleSlot 1 is down: the OS ran short of a resource (EMFILE): OSError: [Errno 24] Too many open files. Any other failure, such as an event loop policy that raises, repeats (TaskLoopCreateFailed) and counts one of three attempts. - The task loop of a worker child does not count against the broker’s
limit of 5 open caller loops (
EVENT_LOOP_LIMIT), as the broker’s sync runner loop does not. An async task that calls the broker gives the task loop a resource set, which it keeps until the child stops. - A task must await the work that it starts. When the body of an async
task returns (a value, or an exception that its exception mapper turns
into a result) and asyncio tasks that it started are still on the loop,
the child waits up to 1 second for them (
ISOLATED_LEFTOVER_GRACE_S; an isolated task has no grace option), then cancels the work that is alive in three rounds 0.1 s apart. The worker writes what the task returned only after that work ended, and only then the retry and the next workflow step, so they never run next to that work. The worker then replaces the child: cancelled code can leave broken state in it. The return ends the deadline:timeout_msbounds the body, and the outcome is neverTASK_TIMEOUTin this wait. Work still alive 0.3 s after the first cancel is stuck: the worker kills the child and writes what the task returned after it saw the exit. A child that reports neither the end of that work nor that it is stuck within 26.3 s of the return, and a child that ends in the wait, also end with what the task returned once the exit was seen; no exit seen within 30 s of a kill follows the rule of a timeout kill. The child writes the finalizing marker of the row only after that work ended, and not at all for work that did not stop, so the worker beats the row while the work runs and the reaper does not fail it, also with small stale thresholds. A client that keeps its own background tasks on the loop delays the outcome of each async task by about 1 second and causes a child replacement after it. The worker logs one line for each task that left work: the coroutine of each task left, up to 10, and how that work ended. A task that cancels a helper task infinallymust also await it. The check sees asyncio tasks only: a callback thatloop.call_soon()orloop.call_later()scheduled, andloop.run_in_executor()work that the task did not await, are not waited for, cause no replacement, and can still run during a later task of the same child. - When the worker stops a child, the child cancels what is left on its
loop and closes the loop, in at most 7 seconds. A thread of
asyncio.to_thread()that still runs after that time does not hold the child: the child exits without waiting for it, and itsatexithandlers do not run. The close of the async generators and the shutdown of the default executor each get at least 1 second, also when the cancel of left tasks used the first 5 seconds. So thefinallyof an open async generator runs, and an idle executor does not count as a running thread.
- A direct call of an async task returns a coroutine of the
-
The worker state names the slots that are down.
WorkerStateSnapshot.slot_healthisAllSlotsServing(),SlotsDown(slots), orSlotHealthNotRecorded()for a snapshot that a worker without this field wrote. EachDownSlotholds the slot index, the cause (ResourceShortage(errno_name)orThreadStartRefused()), the error text of the last failed start (at most 2400 characters),down_since, andfailed_boots.list_worker_states(),get_worker_state()andget_worker_state_history()return it. A stored value of another shape fails the read withMONITORING_QUERY_FAILED. Schema v53 adds the columnhorsies_worker_states.slots_down(jsonb, nullable, no backfill). -
The worker state reads the claim budgets, and a fleet check finds the pending classes that no live worker can run. No schema change.
WorkerStateSnapshot.class_capacityisClassBudgets(budgets), with oneClassBudget(budget, capacity, serves)per budget inExecutionorder, orClassCapacityNotRecorded()for a snapshot without the column value. A stored value of another shape (an unknown class name, a capacity that is not a whole number of 0 or more, aserveslist without its own class, or a class in two budgets) fails the read withMONITORING_QUERY_FAILED, and the message names the worker.get_class_coverage()gives each (queue, class) withPENDINGrows one state:StreamServed(workers, capacity),StreamWithoutCapacity(workers)(every child or seat of the serving budgets is down) orStreamUnserved(). A worker is live while its latest snapshot is at most 120 seconds old (WORKER_STATE_LIVE_WINDOW_S, the threshold at which the web UI already marked a worker stale). A live snapshot withoutclass_capacity, from a worker version that does not record its budgets, leaves that worker out: it serves no stream, is not inlive_workers, andnot_recordednames it (ClassesNotRecorded(worker_id, snapshot_at), by worker id). A live snapshot whose value is unreadable fails the read and names the worker.- Each reaper pass runs the check, one check interval after the worker
starts. An unserved (queue, class) logs an error, one without capacity a
warning; each worker logs one (queue, class) at most once per 5 minutes.
The workers left out log one warning per 5 minutes that names up to 10 of
them.
This covers a shared task that went back to the queue as
ISOLATEDwhile no live worker has isolated children. get_class_report()adds the wait of each stream’s oldest due row (OldestWaitingorNoneDue) and, per live worker, each budget’s capacity, rows in flight and free capacity (BudgetLoad). It names the workers left out innot_recordedtoo;GET /api/workers/classesreturns them innot_recorded.- Both reads probe
idx_horsies_tasks_claim_pending_class: the check once per pending (queue, class), the report once per (queue, class, priority). Each read has a bound and fails closed at it. - The web UI shows the pending rows per execution class with fleet
warnings, each live worker’s budgets with free capacity, and each
worker’s down slots (
GET /api/workers/classes;WorkerStateInfogainsslot_healthanddown_slots).
-
horsies check --livecompares every worker profile with the server connection limit. One worker opens the pool of the worker process (the resolvedWorkerProfile.pool), 2 listener connections, 1 runner heartbeat connection and the resolvedpool.max_sizeof its mode for each child. A profile whose worker opens more thanmax_connectionslesssuperuser_reserved_connectionsandreserved_connectionsfails withHRS-219(CHECK_WORKER_CONNECTIONS). Task code that sends opens the app’s broker in its child, thetask_send_poolof its child kind for each event loop it sends from. The comparison does not count it; the error and the boot log state it, with one sending loop per child. Withpgbouncer_transaction_mode=Truenothing is compared. At boot, before any child starts, a worker logs the same comparison as a warning. It also warns when the per-child memory limits of its profile add up to more than the memory that it can use: the container (cgroup v2 or v1) memory limit when that is below the host RAM, else the host RAM. The cgroup is the one that/proc/self/cgroupnames, and its limit is the smallest on the path from it up to the mount point, so a systemdMemoryMaxand a container seen with--cgroupns=hostcount. The warning names the source. It starts in both cases. -
Each task declares its execution class, and each task row stores it.
@app.task(..., execution=...)takes anExecutionmember:Execution.ISOLATED(the default: one task at a time in its own child process),Execution.SHARED_ASYNC(manyasync deftasks as coroutines in one child process) orExecution.SHARED_THREAD(manydeftasks on threads in one child process).Executionis exported fromhorsies.- The decorator checks the rules when the task is defined, one code per
rule:
SHARED_ASYNCneeds anasync deffunction (HRS-107),SHARED_THREADneeds adeffunction (HRS-108), both shared classes needtimeout_ms(HRS-109), onlySHARED_ASYNCtakescancel_grace_ms(HRS-110), andcancel_grace_msis from 100 to 10,000 ms (HRS-111).horsies checkreports them. Withexecutionwritten as a member, a type checker refuses the same declarations. A class known only at run time needstimeout_msand takescancel_grace_ms; the decorator checks it at run time. cancel_grace_msis the time that aSHARED_ASYNCtask gets to stop after its cancel. A task that sets none gets 1,000 ms. The stored task options hold the class ("execution") and the grace in effect at send ("cancel_grace_ms", null for the other classes).- Schema v54 adds
horsies_tasks.execution(TEXT, NOT NULL, default'ISOLATED', a CHECK on the three values) and buildsidx_horsies_tasks_claim_pending_classon(queue_name, execution, priority, enqueued_at, id)over PENDING rows, concurrently. A send, a delayed send, a workflow node and a rerun write the column from the stored task options. Rows written before v54, and rows that a process of an earlier version writes, readISOLATED. TaskInfo.executionand the monitoringLeafTaskInfo.executiongive the class of a live task. Both areNonefor a task read from history, which does not keep the class.- The stored task options have two more keys, so the command fingerprint and the enqueue digest of a send change. A keyed resend of a request that an earlier version sent is a conflict, not a replay.
- Enqueue refuses, with
INVALID_TASK_OPTIONS, task options that are not a JSON object, and anexecutionor acancel_grace_msthat does not fit the rules above.
- The decorator checks the rules when the task is defined, one code per
rule:
-
Shared async children run
SHARED_ASYNCtasks, many at once.WorkerProfile.shared_asynctakesSharedAsyncChildren(count, seats, recycle=SharedRecycle(after_per_seat, above))orNoSharedAsyncChildren()(the default). A shared async child is one process that runs up toseatsSHARED_ASYNCtasks at once, as coroutines on one event loop, from its start to its stop; a free seat takes the next task at once.countis 1 to 1024 andseatsis 1 to 1000. A profile with only shared async children is valid.SharedAsyncChildren,NoSharedAsyncChildren,SharedRecycleandSharedAsyncChildStartHookare exported fromhorsies.- A worker with shared async children claims
SHARED_ASYNCrows into aSHARED_ASYNCbudget, whose capacity is the seats of its shared children in service (a closing child counts only the tasks it still runs), and runs each on a seat; its isolated children never run one. A worker with none runsSHARED_ASYNCrows in isolated children. A new task goes to the open shared child with the fewest tasks in flight. - An attempt ends when its coroutine has ended and no work that it
started is alive: the asyncio tasks that it created, itself or through a
library (found by a task factory and by a scan of the live tasks, so a
task built with
asyncio.Task(...)counts too), and its calls in the default executor. The child reports the end only then, so a retry never runs next to work of the attempt that it replaces. The end checks of one loop turn share one scan of the live tasks. - A task that returns before its deadline (a value, or an exception that
its exception mapper turns into a result) while such work is alive
keeps what it returned: its deadline no longer applies. The child waits
up to
cancel_grace_msfor that work. Work still alive then is cancelled in the cancel rounds below, and the child closes (the newCloseReason.LEFTOVER_WORK): it takes no new task and is replaced when its tasks end. Work still alive 0.3 s after that cancel is stuck: the task keeps its seat, the worker kills the child when every task left in it is stuck, and then writes what the task returned, with noTASK_TIMEOUTand no retry in an isolated child. The report of that cancel carries what the task returned, so from then on any kill of the child whose exit the worker saw writes what the task returned. Each task that opens a client on first use, outsideon_shared_async_child_start, so costs one child restart. The child logs one line for each task that left work: the coroutine names of that work, up to 10, and whether it ended within the grace, was cancelled, or was stuck. - The child holds the deadline of each task. At
timeout_msit cancels the task; when the attempt has ended, the worker writesTASK_TIMEOUT, the retry and the workflow step, and the other tasks of the child continue. A task that returns a value after its deadline fired still fails withTASK_TIMEOUT. Attimeout_ms + cancel_grace_ms, when the attempt is alive, the child sends a second cancel to the task and to its asyncio tasks, then two more rounds 0.1 s apart, and the child closes: it takes no new task and is replaced when its tasks end. An attempt still alive 0.3 s after the second cancel is stuck: it keeps its seat, and the worker kills the child when every task left in it is stuck. Those tasks fail withTASK_TIMEOUT, and their retry runs in an isolated child (the retry write setsexecution = 'ISOLATED'on the row). - A thread in each shared child watches its event loop. A loop that has
not turned for 5 s is reported with the task that holds it and the
source line, and the worker kills the child at once. That task fails
with the new
TASK_BLOCKED_LOOPand retries in an isolated child; each other task before its deadline fails with the newNEIGHBOUR_BLOCKED_LOOP, and runs again only whenauto_retry_fornames that code; a task past its deadline fails withTASK_TIMEOUT. - A call in the default executor stays the work of its task when an
on_shared_async_child_starthook set a default executor of its own: after the hooks the child wraps that executor, which still runs the calls. A default executor that is not aThreadPoolExecutorafter the hooks (set pastloop.set_default_executor()) ends the child with the exit code of a failed hook.anyio.to_thread.run_sync()work is not seen as the work of its task. - The watching thread names a task only when that task held the loop for the whole block: once the loop has not turned for 0.2 s it reads the task that runs every 0.1 s, and at the report, and names it only when every reading saw the same task. Many short steps of several tasks in one turn name no task: each task before its deadline then takes the crash path, as for code of no task.
- The worker kills a child that has reported neither the second cancel
nor the end of a task
timeout_ms + cancel_grace_ms + 10 safter its placement: C code holds the GIL in it. A child that reported that cancel, or the cancel of the work that a task left after its return, and then reports neither the task stuck nor its end bytimeout_ms + cancel_grace_ms + 20.3 safter its placement (the first check, then 0.3 s for the cancel rounds and the stuck check, and the same 10 s margin) froze after the report, and is killed as well. A child that still holds a refusal 145 s after its placement is killed as well: the writes of a refusal end within 120 s on a shared async child and 135 s on a shared thread child (two attempts of the RUNNING update and the finalizing marker, each at most 30 s for a connection, 10 s for its statements, and 5 to 10 s for a cancelled statement to end), and the block limit with its margin comes after them. At these kills a task past its deadline fails withTASK_TIMEOUT, and every other task takes the crash path. A task that the child reported stuck keeps its retry in an isolated child at every kill by the worker whose exit it saw: this backstop, the drain deadline of a stop, and a blocked loop of another task. No other retry moves to an isolated child. - A crash of a shared child that the worker did not cause ends every task
in it through crash recovery (
WORKER_CRASHED, or its retry), after the worker saw the exit. A task that returned before its deadline, and whose left work the child reported in its cancel rounds or stuck, ends with what it returned at every end of its child whose exit the worker saw, also at an out-of-memory kill or a crash. - A task that cancels its own asyncio task and returns before an
awaittakes the cancel ends as that cancel, through its exception mapper, as a cancel that reaches the body at anawaitdoes. The request is cleared, so the seat does not end cancelled, and the child serves on. - The child writes the finalizing marker of a task when its attempt ends:
after its coroutine and the work that it started, and only when the
task returned before its deadline and that work ended within the grace.
The child reports the end after the marker, and waits for it at most
until the deadline of the task. A task that timed out, or whose work
needed a cancel, gets no marker: the child reports its end at once, and
the runner heartbeats of the worker cover the row until the worker reads
that end. So the heartbeats of a row never stop while work of its
attempt still runs, a marker that waits for a pooled connection never
turns a returned value into
TASK_TIMEOUT, and no marker uses the cancel grace of a task. - A
SHARED_ASYNCrow without a usable storedtimeout_msorcancel_grace_msfails withWORKER_SERIALIZATION_ERRORon every worker, after its RUNNING update and before the task resolves; the data of the error names the field. It never runs, and it does not return to the queue. - The RUNNING update with the first runner heartbeat and the finalizing
marker of a shared task run on an async connection pool that the child
opens on its loop, sized by
SharedAsyncChildren.pool. An attempt of the RUNNING update that does not end within 40 s (30 s for a connection, 10 s for its statements, for example an UPDATE that waits on a row lock) ends withOWNERSHIP_UNCONFIRMED: the task does not start, and its row staysCLAIMEDuntil its lease ends. The child writes its frames from a thread, so a slow worker never stops the loop. A shared child runs onlyasync deftasks; a row that reaches one with adeffunction fails withWORKER_RESOLUTION_ERROR. @app.on_shared_async_child_startregisters anasync defhook that runs on the loop of each shared async child before its first task, for long-lived async clients. Same 10 s bound and fail-closed stop ason_child_process_start. It also accepts an object whose__call__isasync def, a function with an awaitable return annotation, and a lambda; it refuses another callable withHRS-214.- Recycling counts per seat: a shared child closes after
after_per_seat.tasks × seatstasks (default 100 per seat) or when its RSS after a task reachesabove. A closing child takes no new task, its tasks finish, then a new child starts. WorkerProfile.seat_count()is one per isolated child plusseatsper shared child. The claim limit check (HRS-218) compares with it plusprefetch_buffer; the connection check (HRS-219) and the memory warning count the shared children; the boot memory guard compares each mode’s limit with the baseline of a child of that mode. A mode whose children were all down at the start is compared when its first child becomes ready, and a breach then stops the worker non-zero.
- A worker with shared async children claims
-
Shared thread children run
SHARED_THREADtasks, many at once.WorkerProfile.shared_threadtakesSharedThreadChildren(count, seats, recycle=SharedRecycle(after_per_seat, above))orNoSharedThreadChildren()(the default). A shared thread child is one process that runs up toseatsdeftasks at once, each on a seat thread that the child starts at boot; a free seat takes the next task at once.countis 1 to 1024 andseatsis 1 to 256 (each seat is an OS thread with its own stack). A profile with only shared thread children is valid.SharedThreadChildrenandNoSharedThreadChildrenare exported fromhorsies.- A worker with shared thread children claims
SHARED_THREADrows into aSHARED_THREADbudget, whose capacity is the seats of its thread children in service (a closing child counts only the tasks it still runs); its other children never run one. A worker with none runsSHARED_THREADrows in isolated children. A new task goes to the open thread child with the fewest tasks in flight. - A thread cannot be cancelled. The child holds the deadline of each task:
at
timeout_msit reports the deadline and closes (it takes no new task), and the task runs on. When every task still in the child is past its own deadline, the worker kills the child with its session; after the exit, each of those tasks fails withTASK_TIMEOUT, with its retry and its workflow step. The task whose deadline closed the child retries in an isolated child. A task alone in its child is killed at its deadline; a neighbour with time left runs to its end. A task that ends by itself after its deadline fails withTASK_TIMEOUT. A child that has not reported a deadline bytimeout_ms + 10 sis frozen (C code holds the GIL): the worker kills it, each task past its deadline fails withTASK_TIMEOUT, and every other task of the child takes the crash path. The task whose deadline closed the child keeps its retry in an isolated child at every kill by the worker whose exit it saw, also at this one and at the drain deadline of a stop. - A crash or a kill of a thread child ends every task in it, each through crash recovery after the worker saw the exit.
- A
SHARED_THREADrow without a usable storedtimeout_msfails withWORKER_SERIALIZATION_ERRORon every worker, also on an isolated child, after its RUNNING update and before the task resolves. - A thread child boots as an isolated child does and opens one sync pool,
sized by
SharedThreadChildren.pool; its seat threads share it for the RUNNING update and the finalizing marker, and wait up to 30 s for a connection. A thread child runs onlydeftasks; a row that reaches one with anasync deffunction fails withWORKER_RESOLUTION_ERROR. - The outcome of a thread task is fixed when it returns, and a task that
returned has no deadline left. The seat then writes the finalizing
marker, only for a task that returned before its deadline, and waits for
a pooled connection for it at most until that deadline. A task that
timed out gets no marker: the child reports its end at once, and the
runner heartbeats of the worker cover the row until the worker reads
it. A marker that waits for a connection never turns a returned value
into
TASK_TIMEOUT. - A seat thread whose task reaches its deadline while the thread waits
for a pooled connection does not start the task: right before the
RUNNING update it reads the deadline of the run, and the row stays
CLAIMED. The child reports the deadline, and the worker returns the row to the queue with noTASK_TIMEOUT. - Recycling is per seat, as for shared async children. The claim limit
check (
HRS-218), the connection check (HRS-219), the memory warning and the boot memory guard count the thread children. - New docs page “Shared Thread Tasks”, with when threads help (C code that releases the GIL scales on every build; pure-Python code needs a free-threaded build) and the rules for task authors.
- A worker with shared thread children claims
-
A worker claims per execution class. A worker has one claim budget for each execution class that it runs children for. Its capacity is the children (
ISOLATED) or seats (a shared class) of that class in service. The worker builds its claim plan once at start from its profile and logs it:- A class with children of its own is claimed into its own budget.
- A shared class with no children of its own is claimed into the
ISOLATEDbudget, when the worker runs isolated children. Isolated children never take a class that has children of its own. - Any other class is not claimed. The worker still starts, and logs one line that names those classes.
- A profile with isolated children only claims every class into the
ISOLATEDbudget. - Priority holds within one class. Each (queue, class) is its own claim
stream in
(priority, enqueued_at, id)order, and a budget of several classes reads their merge in that order. - The use of each budget is counted from the rows: this worker’s RUNNING and live CLAIMED rows of the classes that the budget claims.
ClaimLimits.per_queue_per_pass,ClaimLimits.claimed_per_worker, the queue caps andcluster_wide_capapply to the worker as a whole. In soft cap mode each budget, and the worker, may holdprefetch_bufferrows more than its capacity. A budget whose capacity in service is 0 (every child or seat of it is down) gets no prefetch, and neither does a worker whose budgets are all at 0: their rows stay for a worker with seats. The capacity thatHRS-218compares with is the sum of the budgets plusprefetch_buffer.- A claimed row of a shared class whose task, as this worker registered it,
has another class (for example
SHARED_ASYNCon the row, and the function is nowdef) is writtenISOLATEDand returned toPENDING, fenced to its claim generation, with one warning. A row of a task that the worker did not register is dispatched as before. - Schema v55 adds
horsies_worker_states.class_capacity(jsonb, nullable, no backfill): each budget class with itscapacityand the classes itserves.
-
Each child kind has its own connection pool, and a shared async child sizes its default executor.
IsolatedChildren.pooltakes anIsolatedPool(min_size, max_size=PoolSize(connections=N), check);SharedAsyncChildren.poolandSharedThreadChildren.pooltake aSharedPool(min_size, max_size, check)whosemax_sizeisPoolSize(connections=N)orPoolSizeFromSeats().SharedAsyncChildren.executortakesExecutorThreads(threads=N)orExecutorThreadsFromSeats(), and sizes the default executor of the event loop of each shared async child (asyncio.to_thread,loop.run_in_executor(None, ...)). Anon_shared_async_child_starthook that sets its own default executor still replaces it.- Defaults: an isolated child opens a pool of 0 to 1 connections, with
no checkout check: its two writes run one after the other. A shared child
derives its pool
max_size, and a shared async child its executor threads, from its seats:min(cap, max(floor, ceil(seats / seats_per_unit))), with constants of each kind inhorsies.core.models.child_sizing: one connection per 10 seats, from 2 to 40 for a shared async child and from 2 to 26 for a shared thread child; one executor thread per seat, from 2 to 32. - The pool sizes are the measured knees with tasks that sleep 0.1 s, 3.5 ms from Postgres 17: a shared async child of 100 seats gives 790-810 tasks/s with 10 connections, as with 20, and 570-590 with 5; a shared async child of 1000 seats gives 1,700-1,760 tasks/s with 40 connections, 1,715-1,720 with 80 and 970 with 10.
- Bounds:
PoolSize.connectionsis 1 to 1000,min_sizeis 0 to 1000, andExecutorThreads.threadsis 1 to 256. Amin_sizeabove the resolvedmax_sizefailshorsies checkwithHRS-218and names the field path, for exampleworker_profiles['batch'].shared_async.pool.min_size. - The connection check (
HRS-219) and the boot warning count each child with the resolvedpool.max_sizeof its mode, one term per mode:parent pool max_size 5 + listener 2 + runner heartbeat 1 + 4 isolated children x isolated.pool max_size 1 + 2 shared async children x shared_async.pool max_size 5 = 22. - The start log line of the worker states the resolved pool of each kind
and the executor threads of the shared async children, with
derived from seatsfor a derived size. IsolatedPool,SharedPool,PoolSize,PoolSizeFromSeats,ExecutorThreadsandExecutorThreadsFromSeatsare exported fromhorsies.
- Defaults: an isolated child opens a pool of 0 to 1 connections, with
no checkout check: its two writes run one after the other. A shared child
derives its pool
-
The worker process sizes its own pool in the profile, and task code in a child opens a task-send pool of its kind.
WorkerProfile.pooltakes aParentPool(max_size)whosemax_sizeisPoolSize(connections=N)orPoolSizeFromSeats(), the default. The worker process claims tasks and writes the outcome of each task through this pool. It opens withpool_size = max_size,max_overflow = 0and the pre-ping ofworker_pool_pre_ping, so the connection check counts it exactly.- The default derives the size from the seats of the whole profile:
min(20, max(5, ceil(seat_count() / 10))), with the constantsPARENT_POOL_CONNECTIONS_CAP,PARENT_POOL_CONNECTIONS_FLOORandPARENT_SEATS_PER_POOL_CONNECTIONinhorsies.core.models.child_sizing. 8 isolated children: 5; 64 isolated children: 7; one shared async child of 100 seats: 10; of 1000 seats: 20.WorkerProfile.resolved_parent_pool()returns the size. - An explicit size has a minimum of 5 (
PARENT_POOL_MIN_CONNECTIONS): while the worker is busy, the claim loop holds 1 connection and the 2 steady flushers of the batched success write hold 1 each, and 2 stay free for the reaper and the writes per task. A smallerPoolSize(connections=N)fails with HRS-218; a size derived from the seats is never below it. - Plain successes are written in batches: a worker of 1000 seats kept at most 3 to 8 of its 20 connections busy at 970-1,760 tasks/s. A failure is written per task: a worker of 1000 seats whose tasks all fail, 3.5 ms from Postgres, gives 1,010 tasks/s with 20 connections and 500 with 5.
IsolatedChildren,SharedAsyncChildrenandSharedThreadChildrentaketask_send_pool: PoolSize, defaultPoolSize(connections=2). Task code that sends through the app (app.get_broker()) opens the app’s broker in its child with that many connections and no overflow, for each event loop that it sends from. That broker keepspool_pre_ping. The child sets it before it imports its task modules, so a broker that an import builds opens it too. The pool ofPostgresConfigdoes not apply in a child. In every other process, the worker process included,app.get_broker()opens the pool ofPostgresConfig.- The connection check (
HRS-219) and the boot warning count the parent pool asparent pool max_size N, and state the task-send pool of each mode as a ceiling for children whose tasks send:8 isolated children x isolated.task_send_pool 2 = 16 more, 32 in all. - The start log line of the worker states the resolved parent pool, with
the seats it derives from, and the
task_send_poolof each kind:pool=20 connections (derived from 1000 seats). ParentPoolis exported fromhorsies.
- The default derives the size from the seats of the whole profile:
Changed
Section titled “Changed”-
The worker process and its children spend less CPU per task. The worker process counts the tasks that each child holds, reads child frames from one buffer and writes a small frame in one call, probes its listener connection at most once per interval of activity, freezes the objects of its start for the garbage collector, and sends a batch of fused completions as one JSON parameter. A child keeps the resolved hints and signature of each task function and reads parent frames through one buffer. One frame parser serves both sides.
-
A sync send of a producer writes its row on the calling thread.
send()ran its INSERT on the broker’s sync runner thread: each send handed the work to that thread and waited for it. On a VM, the hand-offs across CPUs around the database wait cost CPU: with 5,000 sequential sends to one Postgres 18 server, 411-419 sends/s and 1,441-1,479 µs of CPU per send. In a process that runs no task bodies, an unkeyedsend()now prepares the row and runs the same INSERT on the calling thread, through a sync pool of the broker with the same URL, pool sizes, connect args, session settings and idle ping: 637-660 sends/s and 619-672 µs of CPU per send.- The first send of a broker (it checks the schema), a send with an
idempotency key, and a send from task code in a worker child keep the
sync runner. A task id that is taken is compared on the sync runner, as
before. The outcomes and errors are those of
enqueue_async. - The sync pool opens on first use and holds up to
pool_size + max_overflowconnections, next to the pool of the sync runner’s set.close()closes it.
- The first send of a broker (it checks the schema), a send with an
idempotency key, and a send from task code in a worker child keep the
sync runner. A task id that is taken is compared on the sync runner, as
before. The outcomes and errors are those of
-
A stopped child ends without the interpreter teardown. After a stop frame, a child that has no executor threads left waits for its non-daemon threads, runs its
atexithandlers and flushes its output, assys.exit(0)did, then ends withos._exit(0). The teardown freed every object, which wrote to the pages that a forked child shares with the preload parent, so each ending child copied them, and the peak memory of the worker rose while children were replaced. An object that still exists at the stop no longer gets__del__in a child; Python does not promise those calls at exit. Close or flush files in the task: buffered files that task code left open are not flushed at the stop. The child looks up these steps, private CPython functions, at the stop; on a Python without them it logs the missing name and ends throughsys.exit. -
pool_pre_pingpings only a connection that waited in the pool for more than 1 s. SQLAlchemy’spool_pre_pingpinged at every checkout: one round trip more for each send of a producer. The broker now registers its own checkout hook (register_idle_ping), with SQLAlchemy’s ping off: a connection checked in withinPRE_PING_AFTER_IDLE_S(1 s), and a new connection, are used with no ping.worker_pool_pre_pinghas the same meaning for the engine of the worker process.- A ping that finds a dead connection replaces it and every connection checked in before it, and the checkout takes a new one, as before.
- A connection that dies within 1 s of its check-in fails its next
statement with a retryable connection error; for a send,
TaskSendEnqueueFailed(retryable=True). The ping never covered a connection that dies after it.
-
The worker keeps one connection for its claim loop and for each steady finalize flusher. Each claim pass and each batched completion took a connection from the worker pool and gave it back, so each statement paid a checkout, and a ping when
worker_pool_pre_pingis on. The claim loop and the two steady flushers of the finalize batcher now hold one pool connection each, in autocommit, and send each statement on its psycopg connection. A flusher started for a surge takes a connection per statement, as before.- A held connection goes back to the pool after 1 s with no statement,
when it reaches the
pool_recycleage, and at the worker stop. - A connection error on a held connection is a lost outcome
(
OutcomeUnknown). The claim loop retries the pass, and finalize retries and classifies the replay, as before. SQLAlchemy’s disconnect handling then replaces every connection of the pool. A cancelled statement drops its held connection. - A plain success finalize is one round trip, with
worker_pool_pre_pingon or off: the held connection needs no checkout. - With 20,000 no-op tasks, 3 runs each: 8 isolated children 1,540-1,674 → 1,799-1,925 tasks/s, Postgres CPU per task 1.49-1.62 → 1.32-1.42 ms; 8 shared async children of 100 slots 4,109-4,204 → 4,150-4,288 tasks/s.
- A held connection goes back to the pool after 1 s with no statement,
when it reaches the
-
A task row is read by its id alone (schema v60). The start of a task, its finalizing marker, and the completion and failure functions (
horsies_complete_task_fused,horsies_complete_locked_task,horsies_fail_locked_task,horsies_finish_failed_attempt,horsies_finish_workflow_success) read the row by id and check the claim on the locked row. With the statistics that Postgres takes while every row is PENDING, for example after a burst of sends, the planner served the earlier statements from the status index or a partial index: 113 index and table pages per call for the start, 258 for the marker and 179 for the read of the completion, against 27, 10 and 5 with the statistics of a working queue. Outcomes and locks are the same, except that a start or a marker whose claim changed now locks the row until its transaction ends. -
A task’s start sends no
horsies_task_statusevent (schema v59). The status update trigger skips the change to RUNNING. A task’s start commit is one transaction per task, and its NOTIFY made Postgres commit it one at a time with every other notifying commit. The web app sees RUNNING when it next re-reads the tasks. A listener ofhorsies_task_statusthat needs the start reads the row. -
A failed attempt, a timeout and their retry are one read and one statement (schema v56).
horsies_finish_failed_attempttakes the task advisory lock, then the row lock under the claim fence, and writes the attempt row with the retry or the terminal failure. The worker reads the retry facts of the attempt first, without a lock and fenced on the claim generation; the retry policy and its jittered delay are decided in Python, as before. Both statements run in autocommit. Counted at the wire with the defaultworker_pool_pre_ping=False:- a task error, a timeout or a crash, failed or retried: 2 round trips, before 7 (BEGIN, the locked read, one or two reads of the retry facts, the retry or the attempt row and the move, COMMIT, and the capacity wake of a failed plain task);
- a worker-level failure or a result that does not decode, which are never retried: 1 round trip, before 6. With the pre-ping on, each is 1 round trip more, before 2 more.
- The terminal failure of a plain task sends its capacity wake in the same statement, so it has no phase 2. A workflow task’s failure runs phase 2 as before, which sends its wake. A scheduled retry still wakes its queue when the retry is due.
will_retryof the attempt row is true only when the same statement wrote the retry. A retry whose next attempt would start at or aftergood_untilis written as the terminal failure; the log names both instants.good_untilis judged against the database clock under the row lock.- The retry count that the worker read is part of the fence: a write for another attempt of the same task writes nothing.
- A replay of a committed failure (the reply was lost) writes nothing and runs the phase 2 of a workflow task again; a plain task gets no second wake. A replay of a committed retry writes nothing: the retry ended the claim generation, so the replay is a lost claim.
-
Breaking: worker child exit codes 74 to 76. In 0.6.x, 74 meant only a forked child that does not reparent to the worker. In 0.7.0:
- 74: a child that stopped while threads of the executor of its task loop still ran. It leaves without waiting for them. New.
- 75: a child that failed its boot, after it reported the cause. New.
- 76: a forked child that does not reparent to the worker. It was 74. A worker that cannot write runner heartbeats and has not stopped 120 s after it started to stop exits with 77, also new.
-
Breaking: the claim function takes a budget per execution class.
horsies_claimtakesp_budgetsandp_routesin place ofp_processes. It returns arow_kindcolumn: one'CLAIMED'row per task it took, with the row’sexecution, and one'SHORT'row when it took fewer rows than it chose, with that count inshort_by. Schema v55 drops the earlier signature, so a 0.6.x worker fails every claim pass after a 0.7.0 process migrates. Stop every 0.6.x worker before the first 0.7.0 worker starts.- A claim pass reads the head of each (queue, class) stream without locks:
its pending rows and its rows with an expired lease, in one order. One
walk over all heads in rank order (queue priority, priority,
enqueued_at, id) takes a row when its queue, its budget and the pass each have room left, so a row that one limit drops takes no room that another class could use. The pass then takes the chosen rows from each stream withFOR UPDATE SKIP LOCKED. It locks only the rows it takes. Before, a pass over two queues of equal priority could lock up to its batch in each queue and return one batch. - A row that another claim holds at the take is skipped, and the stream
goes on to its next row. When a pass takes fewer rows than it chose,
because another transaction held or took them, the worker runs the next
pass at once, as no
NOTIFYannounces the rows that remain. At most 3 such passes run in a row; after that the worker waits for its next wake. - v55 builds
idx_horsies_tasks_claim_expired_classon(queue_name, execution, priority, enqueued_at, id)over CLAIMED rows concurrently, and dropsidx_horsies_tasks_claim_pendingandidx_horsies_tasks_claim_expired_orderedconcurrently after its version row commits, so an upgrade that fails leaves the earlier claim function the indexes it reads. Each claim stream is one range ofidx_horsies_tasks_claim_pending_classor of the new index. With 50,000 expired leases in one queue, the expired stream of a class with no rows reads 142 buffers on the new index and 7,848 on the unclassed one. - The claim functions
horsies_claim,horsies_claim_planandhorsies_claim_candidatesrun withjit = off. Their generic plan passedjit_above_cost, so a server with JIT compiled the statement on every pass: 26 to 47 ms per pass against 0.7 to 1.8 ms of work, at a 90,000-row backlog on PostgreSQL 17.
- A claim pass reads the head of each (queue, class) stream without locks:
its pending rows and its rows with an expired lease, in one order. One
walk over all heads in rank order (queue priority, priority,
-
Breaking: the worker and its children send a one-statement write in one round trip. The write runs in autocommit, as its own transaction, with no pre-ping, no BEGIN and no COMMIT. A write that needs several statements to commit together keeps its transaction.
- Worker process: the finalize of a plain task that succeeded
(
horsies_complete_task_fused: attempt, history row and capacity wake in one call) is 1 round trip, before 4 (pre-ping, BEGIN, the call, COMMIT). The claim statement, the orphan cancel and the wake of a retry that comes due are 1 round trip each, before 4 each. PostgresConfig.worker_pool_pre_ping(defaultFalse) sets the pre-ping of the worker process’s engine.pool_pre_ping(defaultTrue) applies to the other engines only: producers, the web UI, and the engine that task code in a child sends with. Setworker_pool_pre_ping=Trueto keep the pre-ping of 0.6.x.- With the pre-ping off, a pooled connection that died while idle fails
the first statement. The finalize returns a retryable connection error
and runs again on a new connection; a statement whose reply was lost is
classified on that run (
AlreadyApplied) and is not applied twice. The claim loop retries a pass that failed with a connection error. - Child pools: every connection is in autocommit. The RUNNING update of a plain task, with its first runner heartbeat, and the finalizing marker are 1 round trip each, before 4 each with the checkout check (3 without). The RUNNING update of a workflow task and the start of its node run in one transaction: BEGIN, update, node, COMMIT.
IsolatedPool.checkandSharedPool.checkdefault toFalse(they wereTrue). A dead pooled connection then fails the write that takes it: the RUNNING update ends asOWNERSHIP_UNCONFIRMEDand the claim lease returns the task; a failed finalizing marker is logged. Setcheck=Trueon thepoolof a kind to check each checkout.- The advisory lock of
horsies_claimends with the claim statement. It was held until the COMMIT one round trip later.
- Worker process: the finalize of a plain task that succeeded
(
-
The worker process writes the outcomes of plain tasks that succeeded in batches. Schema v57 adds
horsies_complete_tasks_fused: one statement callshorsies_complete_task_fusedonce per row, in task-id order, so each row keeps its claim-generation check, attempt row, history move and capacity wake. A batch holds up to 64 rows and is one round trip; a batch of one row runshorsies_complete_task_fuseditself.- There is no timer. Two flushers write whenever rows wait, each on a connection of the worker pool. A lone row is written at once; the rows that come while both flushers write form the next batch.
- When a full batch of 64 rows waits while both flushers write, one more flusher starts, up to the size of the worker pool. It writes full batches only.
- A finalizer ends, and frees its seat, only after the statement of its batch returned.
- A row that the server refuses (for example, no history partition for its retention class) refuses the whole statement. The batch is then written row by row, once each: the other rows commit, and the refused row takes a phase-1 error. A lost reply or a broken connection is a retryable error for each row of the batch; the retry finds a committed row already applied.
- At a stop, the batcher closes after the finalizers and writes the queued rows, for at most 10 seconds. A row that it does not write stays RUNNING and goes to stale RUNNING recovery.
-
The success of a workflow task is one statement (schema v58), and its phase 2 is shorter.
horsies_finish_workflow_successtakes the task advisory lock, then the row lock under the claim fence (worker and claimed_at), writes the COMPLETED attempt row, and moves the task to the history with the phase-2 record of its node, in autocommit. It sends no wake: phase 2 sends it after the dependents are enqueued. A refusal writes nothing; a replay after a lost reply is already applied, reads the stored result and runs phase 2. Counted at the wire with the defaultworker_pool_pre_ping=False(one more per phase with it on):- phase 1 of a success: 1 round trip, before 5 (BEGIN, the locking read,
the attempt row,
horsies_complete_locked_task, COMMIT); - phase 2, one transaction under the workflow row lock: a node that promotes its dependents 8, before 11; the last node of a workflow 7, before 9; a failed node whose dependent is skipped 12, before 13; a failed last node 8, before 10. The capacity wake is part of the node write, and the insert of each promoted task links its node in the same statement. The node write also returns whether a node of the workflow waits for the node: a node that no node waits for does not read its dependents. The dependencies of a workflow are written once, when it starts. One update moves a promoted dependent from PENDING to ENQUEUED, where a READY write and an ENQUEUED write did; the committed rows, their timestamps and the NOTIFYs are the same.
horsies_lock_running_task(schema v56) is dropped: its one caller was this success.- A row that is not a workflow task is refused (SQLSTATE 22023) with no write.
- phase 1 of a success: 1 round trip, before 5 (BEGIN, the locking read,
the attempt row,
-
A schema upgrade keeps each claim and recovery index whose form matches. An index whose catalog form matches its declaration is kept. One of another form, or an invalid one, is built concurrently under a temporary name; then one transaction renames the old index away and the new one to its name, and the old one is dropped concurrently. Running workers keep an index of that name throughout. Before, each upgrade dropped the recovery indexes and built them again, and the scans that read them ran without them meanwhile.
-
Breaking: a slot that cannot start a child for a cause that can clear no longer stops the worker. A child that fails its start now reports the cause to the worker before it exits.
- A cause that is the same on every attempt still counts one of three
attempts, and the third stops the worker: the app or a task module does
not import, the child connection pool cannot be set up, the event loop
for
async deftasks cannot be created for a cause other than the errno values below (for example an event loop policy that raises), the OS refuses the spawn with another errno (for exampleENOENTorEACCESfor the interpreter), a forked child does not reach the worker in 5 seconds, the child dies with no report, or it is not ready within 120 seconds. A failedon_child_process_starthook still stops the worker at once. - A cause that can clear takes only that slot down:
OSErrorwithEMFILE,ENFILE,ENOMEMorEAGAINin a start step of the child or of the worker (the socket pair of the child, the spawn, the event loop), or a thread that the OS refuses at the start (the control thread, the watchdog of a start hook, a thread of the connection pool). Before, each of these used an attempt, and the worker stopped after three, also while other children served tasks. A spawn refusal stopped the worker at once. - A slot that is down takes no task, and the claim passes only the slots that are not down as the worker’s process count. The worker starts a new child for that slot after 1 second, then 2, 4, up to 60 seconds, and logs one line per failed attempt with the slot, the cause and the next wait. The slot returns to service when a child starts.
- When every slot is down, a failed start for a cause that can clear counts an attempt, so a worker that serves no task stops after three in a row. A failed start while another slot is in service starts the count again. A worker with 1 process has no other slot, so each of its failed starts counts.
- At the first start, the worker starts degraded when at least one child is ready and each other slot is down. When no child starts, it exits non-zero.
- The error of a worker that stops names the cause and the error text of the last attempt.
- A cause that is the same on every attempt still counts one of three
attempts, and the third stops the worker: the app or a task module does
not import, the child connection pool cannot be set up, the event loop
for
-
Breaking: a result wait inside a task is refused at once. In a worker child,
TaskHandle.get()andget_async()on a sent task,app.get_result()andget_result_async(), andWorkflowHandle.get()andget_async()returnRetrievalCode.WAIT_INSIDE_TASK. The raw result readsapp.get_raw_result(),broker.get_raw_result_record()and their*_asyncforms returnErr(BrokerWaitInsideTask). The refusal is the first check of each call, before thetimeout_msandASYNC_CONTEXTchecks, and nothing waits. This covers the threads that a task starts and theon_child_process_starthooks. Before, such a wait kept its child: when enough tasks waited for other tasks, no child was free for those tasks, and nothing moved until timeouts ended the waits.send(), starting a workflow,TaskHandle.info()andworkflow_ctx.result_for()still work. Compose the steps as a workflow; the result-handling page lists the forms. -
An exhaustive
matchonRetrievalCodeneeds aWAIT_INSIDE_TASKcase, and one onBrokerOperationErrorneeds aBrokerWaitInsideTaskcase.BrokerErrorCodehas the new memberWAIT_INSIDE_TASK. TheRetrievalCodestring is reserved: anexception_mapperordefault_unhandled_error_codewith that value failshorsies checkwithHRS-212. -
Breaking: a child workflow has at most depth 32. A top-level workflow has depth 0, and a child workflow has the depth of its parent plus 1. A
SubWorkflowNodewhose child would have depth 33 does not start its child: the node goesFAILEDwith the newOperationalErrorCode.SUBWORKFLOW_DEPTH_LIMIT_REACHED, and theon_errorpolicy of its workflow applies. Before, nesting had no limit at run time, and a definition that included itself as a root node nested in one transaction until Python’s recursion limit. -
Breaking: a
WorkflowSpecholds at most 10,000 nodes. A spec with more nodes raisesHRS-038(WORKFLOW_TOO_MANY_NODES) when it is built. Inbuild_with(), that raise fails theSubWorkflowNodewithWORKFLOW_ENQUEUE_FAILED. Split more nodes intoSubWorkflowNodes: each child workflow has its own limit. -
horsies checkreports two nesting errors.HRS-036(WORKFLOW_SUBWORKFLOW_DEPTH_LIMIT): a nesting would start a child deeper than 32, and noSubWorkflowNodeon its path haswaits_for.HRS-037(WORKFLOW_SUBWORKFLOW_SELF_NESTING): a definition includes itself with the same kwargs, and noSubWorkflowNodefrom one level to the next haswaits_for, so the nesting never ends once it starts. Before, the check stopped at both with no finding. Through a node withwaits_for, the results of its dependencies decide the depth, so the check does not report the nesting, and the engine applies the limit at run time. -
An exhaustive
matchonOperationalErrorCodeneedsSUBWORKFLOW_DEPTH_LIMIT_REACHED,TASK_BLOCKED_LOOPandNEIGHBOUR_BLOCKED_LOOPcases. The code strings are now reserved: anexception_mapperordefault_unhandled_error_codewith one of these values failshorsies checkwithHRS-212. A process of an earlier version cannot decode an error stored with one of these codes. -
The worker process writes the runner heartbeats of its RUNNING tasks. One thread of the worker, not on its event loop, writes one statement per
runner_heartbeat_interval_msfor the tasks that its children hold. A row gets a beat only while it isRUNNING, owned by this worker at the claim generation of the dispatch, and without the finalizing mark, and only while its child process runs. So the beats of a task end when the worker sends its child a kill, when the child exits, when the child marks its result finalizing, and when the worker reads the result. The child writes only the first beat, in the transaction that sets the rowRUNNING, and runs no heartbeat thread. A child now uses one connection of its pool at a time, and the worker process uses one more connection, for the heartbeats. A timed-out task’s beats end with the kill at its deadline; the child no longer stops them one interval aftertimeout_ms. -
A worker that cannot write runner heartbeats stops, when its reaper fails stale RUNNING rows. With
auto_fail_stale_running=True(the default), when the heartbeat thread dies, or no heartbeat lands for half ofrunning_stale_threshold_ms, the worker logs the cause at critical level, kills every child that holds a task, stops, and exits with code 1. It exits with code 77 if it has not stopped 120 s later. A failed write is tried again within that half. The half counts from when the last heartbeat landed. After a heartbeat whose write took almost the whole half, the worker stops 0.5 s beforerunning_stale_threshold_mshas passed from the start of that write, because the reaper may count from there. Before, the reaper failed such tasks withWORKER_CRASHEDwhile they still ran, and a retry could run next to them. A killed task takes the child crash path, or stale-task recovery when the database cannot be reached. Withauto_fail_stale_running=False, or no recovery config, no reaper fails a RUNNING row, so the worker logs failed writes, at most once per 60 s, and keeps running. -
Breaking: worker settings move from
horsies workerflags into named worker profiles.AppConfig.worker_profilesmaps a name to aWorkerProfile, andhorsies worker <app> --profile <name>is required. There is no default profile: a missing or unknown name exits 1 before the app check and before any child starts, and the message lists the defined names. A profile holdsisolated=IsolatedChildren(count, recycle= IsolatedRecycle(after, above)),claim=ClaimLimits(per_queue_per_pass, claimed_per_worker),drain_timeout(atimedelta) andpreload(ChildPreload). A limit isTaskLimit(tasks=N)orNoTaskLimit(), andMemoryLimit(mb=N)orNoMemoryLimit(), in place of0andNone. The number of children is the sum of the per-mode counts.AppConfigvalidation checks every profile and names the profile and the field path (HRS-218,CONFIG_INVALID_WORKER_PROFILE): a name that--profilecannot carry, a profile with no children, and a claim limit above the capacity of the worker (children plusprefetch_buffer), which before had no effect. The bounds of the old flags still apply; new upper bounds are 1024 children, 2,147,483,647 tasks, 16,777,216 MB and a drain timeout of 24 hours. Migrating to 0.7.0 has the table from each flag to its field. -
Breaking: the enqueue fingerprint of a scheduled run covers the task’s options. A slot keeps its task id, but its
enqueue_shachanges for every schedule. This includes a task that declares no options, because its stored options text is not null. A slot that a scheduler before this release enqueued, and did not record because it stopped before it saved the schedule state, getsPAYLOAD_MISMATCHwhen this release fires it.- A slot whose task id already holds a different command counts as fired. The scheduler logs one warning, records the run with that task id, and moves to the next slot. In catch-up, the next due slots are enqueued in the same tick. Before, the scheduler logged the slot as a permanent enqueue failure, did not record a run, and stopped the catch-up of that tick at the slot. The same applies when a change to the schedule’s kwargs or to the task’s options meets a slot that is already enqueued.
- The task of such a slot keeps the row that the earlier scheduler wrote:
it runs with no timeout, no retries, and as
ISOLATED. Later slots carry the task’s options.
-
Breaking: with
auto_fail_stale_running=True,RecoveryConfig.running_stale_threshold_msis at least2 × (runner_heartbeat_interval_ms + 2,000 ms). The worker stops when no runner heartbeat lands for half of the threshold. Heartbeats land one interval apart, plus the time the next one takes to land, so that half must hold one interval plus 2 s. A smaller value raisesConfigurationErrorwithHRS-204(CONFIG_INVALID_RECOVERY); the message names both fields and the smallest valid value. Before, the bound was2 × runner_heartbeat_interval_ms. At that bound the half was one interval, and the worker stopped and killed its tasks while every heartbeat landed. Withauto_fail_stale_running=False, the bound stays2 × runner_heartbeat_interval_ms. -
Breaking:
RecoveryConfig.worker_state_snapshot_interval_msis at most 60,000 ms. A worker is live while its latest snapshot is at mostWORKER_STATE_LIVE_WINDOW_S(120) seconds old. The cap is half of that window, so a running worker stays live when a snapshot write is late by up to half the window. A larger value raisesConfigurationErrorwithHRS-204(CONFIG_INVALID_RECOVERY); the message names the value, the cap and the window. Before, the bound was 300,000 ms, and a value above it raised a pydanticValidationError. An interval above 120,000 ms made a live worker read as not live between its snapshots: the fleet check reported its streams as unserved, and the web UI marked it stale. -
horsies requires psycopg 3.3.7 or later. The floor of
psycopg, and ofpsycopg-binaryandpsycopg-c(thepg-binaryandpg-cextras), is 3.3.7, up from 3.3.2. When a running query is interrupted by a cancelled asyncio task,KeyboardInterruptorSystemExit, psycopg 3.3.6 and later wait at most 10 s: up to 5 s to send the cancel request and up to 5 s for the query to end. Then they close the connection. Before 3.3.6, psycopg waited for the query to end with no limit. 3.3.7 also sends the cancel request when the cancel connection is not ready at once; 3.3.6 skipped the request in that case. The 5 s limit on the cancel request needs libpq 17 or later; with an older libpq, psycopg sends the cancel with a blocking call that has no limit.psycopg-binarybundles its own libpq.
Removed
Section titled “Removed”ErrorCode.TASK_ASYNC_FUNCTION_NOT_SUPPORTED(HRS-106). No code raises it, because@app.task()accepts anasync deffunction. The number is retired and is not used again. Remove the member from eachmatchonErrorCode.- The
horsies workerflags--processes,--max-claim-batch,--max-claim-per-worker,--max-tasks-per-child,--max-memory-per-child-mb,--drain-timeoutand--child-preload. Each one exits 2 with a message that names the profile field that replaces it.--logleveland the app locator stay. PostgresConfig.worker_child_pool_min_size,worker_child_pool_max_sizeandworker_child_pool_check. Thepoolof each child kind in a worker profile replaces them. A config that still sets one fails when it is built withHRS-213(CONFIG_INVALID_BROKER_POOL), and the help names the replacement, for exampleIsolatedChildren(pool=IsolatedPool(min_size=N)). Every such field is reported at once.PostgresConfig.worker_pool_size,worker_max_overflowandworker_runtime_config().WorkerProfile.poolreplaces them for the worker process, and thetask_send_poolof each child kind for task code that sends in a child. A config that still sets one, also toNone, fails when it is built withHRS-213(CONFIG_INVALID_BROKER_POOL), and the help namesWorkerProfile(pool=ParentPool(max_size=PoolSize(connections=N))). These fields and the removed child pool fields are reported at once.
-
A child that starts a task of a paused workflow no longer deadlocks with a cancel of that workflow. At the start, the child ran its expiry check and the abandon of the task in one transaction: it held the task’s advisory lock and row lock while it asked for the workflow row. A cancel of the workflow and the expiry of paused workflows lock the workflow row first and then the task rows, so Postgres aborted one of the two. The expiry check now runs alone, and the abandon locks the workflow row before the task locks.
-
The claim’s in-flight count marks the dead entries of the status index (schema v61).
horsies_claimcounted the in-flight rows withstatus = 'RUNNING' OR (status = 'CLAIMED' AND <valid lease>), which Postgres runs as a bitmap scan. A bitmap scan does not mark dead index entries, so each claim read the heap for every task that finished since the last vacuum: two dead entries per task. The count now runs inhorsies_claim_in_flight, with one IN list onstatusand bitmap and sequential scans off. It is a plain scan ofix_horsies_tasks_status, which marks the dead entries that it passes. Same rows and counts. With 8 isolated children and 20,000 no-op tasks: 867-902 pages per count before, 21 after. -
A history probe uses the task id index of a new leaf. The duplicate guard of every terminal move (
horsies_task_history_duplicate_staged) and the four history readers kept the plan of each leaf probe. On a new leaf that statistics showed as empty, that plan was a sequential scan, kept until an ANALYZE of the leaf, so each probe read the whole leaf. The five generated functions run withenable_seqscan = off; every probe has an index. -
horsies[pg-binary]andhorsies[pg-c]install the C build of the installed psycopg version. The extras asked forpsycopg-binaryorpsycopg-c3.3.7 or later, with no tie to the version ofpsycopg. A resolver could pair psycopg 3.3.8 with psycopg-binary 3.3.9, and psycopg then fails at import. The extras ask forpsycopg[binary]andpsycopg[c], which pin the C build to the psycopg version. -
The live web UI sees a task finish (schema v59). A terminal move deletes the live row, so no status change sent
horsies_task_status, and in live mode a finished task stayed RUNNING on screen until another task changed. The single move and each batch move send the event with the task id, in the transaction that already sendstask_done. -
A finalize no longer deadlocks with stale recovery or a cancel of the same task. Every move function takes the task advisory lock before the row lock. The locked read of a workflow task’s success, the failure paths, the reaper’s stale recovery and the monitoring cancel took the row lock first and called a move function later, so two of them on the same task could deadlock (SQLSTATE 40P01), and Postgres aborted one. All of them now take the advisory lock first: the success of a workflow task calls
horsies_finish_workflow_success, the failure paths callhorsies_finish_failed_attempt, and stale recovery and the cancel take the advisory lock before their locked read. -
The phase 2 that advances a workflow node deletes the node’s outbox row. The terminal move of a workflow task records the progression it owes in
horsies_workflow_phase2_pending. Phase 2 advanced the node and left the row, so a later recovery pass consumed it as already applied. The statement that writes the node now deletes the row in the same transaction; a phase 2 that fails leaves it for the recovery pass. -
A capacity wake that the server refuses in the phase 2 of a workflow task is a phase-2 error. The wake ran in the transaction of the node write, and its error was dropped before the commit. A statement that fails on the server aborts the transaction, so the commit rolled the node write back without an error, and phase 2 returned as if it had committed. The error is now a phase-2 error with the usual retry classification.
-
A timed-out task’s row moves only after its child has exited. At the deadline the worker kills the child, waits up to 30 s for its exit, and only then writes
TASK_TIMEOUT, the retry and the attempt row, or requeues a task whose code had not started. The workflow step that follows the failure runs after that write. Before, the worker wrote the row first and killed the child last: a retry was kept apart from the attempt that it replaced only by its retry delay of at least 1 s, and anon_errororallow_failed_depssuccessor could start while the timed-out child still ran. If the child does not exit within 30 s, the worker writes nothing and logs at critical level, and stale-task recovery takes the task. A timed-out task no longer also goes through crash recovery, which replayed its workflow step and logged a finalize error. When theTASK_TIMEOUTwrite fails after the exit, the worker runs that write again up to 3 times, with growing waits; only then does crash recovery write the row, withTASK_TIMEOUTas the code that the retry policy reads. A result that the child sent after the deadline is never written. -
Crash recovery moves a lost child’s row only after it saw the child exit. When the worker kills a child and does not see it exit within 30 s (the socket to the child was lost, the child sent a bad frame, or the slot was cleared at a stop or after a defect), it writes nothing and logs the task id and pid at critical level. Stale-task recovery then takes the task. With stale RUNNING recovery off (no
RecoveryConfig, orauto_fail_stale_running=False), nothing else would move the row, so the worker waits for the exit and writes the row once it sees it, for a timed out task and for a lost child alike; a stop of the worker ends that wait. Before, the worker wroteWORKER_CRASHEDor scheduled a retry at once, so the retry could run next to a child that was still alive. -
A child that the worker killed takes no next task. A kill can land after the child’s result arrived and before the worker read it. The worker then gave the next queued task to the dying child, and that task was lost and requeued. Now the worker starts a new child for it.
-
The cycle check of a
WorkflowSpectakes time linear in its nodes and dependencies. It scanned every node for each node that it removed. On an Apple M2 with Python 3.13, a spec of 10,000 nodes took about 7.5 s to build, and now takes 0.23 s. The findings do not change. -
A task that holds the GIL is no longer failed as stale while it runs. C code that held the GIL in a child stopped the child’s heartbeat thread, so after
running_stale_threshold_msthe reaper failed the task withWORKER_CRASHED, and a retry could run next to it. CPU-bound tasks need no extra margin in the stale thresholds. -
A scheduled run carries the options of its task. The scheduler enqueued each slot with no task options, so the run had no timeout, no retries (
max_retries0), and ran asISOLATED, whatevertimeout_ms,retry_policy,executionandcancel_grace_msthe task declared. A scheduled run now stores the options text that a plain.send()of the task stores, and the worker applies these options. The scheduler takes that text once at start, so every slot of a schedule sends the same text.
0.6.1 — 2026-11-08
Section titled “0.6.1 — 2026-11-08”0.6.1 identifies a retention class by its period and makes the windowed monitoring reads take a bounded window type. The schema does not change.
Changed
Section titled “Changed”- Breaking: a retention class is identified by its period. The class
key is
standard_<days>d, so a key always means one duration, and two processes cannot disagree about it. Queues mapped to one period share one class: one parent, one daily partition for each retained day, and their indexes, where each mapped queue had its own set. The default class keeps its key,standard_30d.queue_retentionkeepsdict[str, timedelta | None]. A period is whole days from 1 to 1825; a fractional day, zero, or more than 1825 days is refused at load. Any configured queue name can be mapped: the queue name is no longer part of the class key, so the 18-character key limit and the identifier rule do not apply to it.retention_classesandRetentionClassConfigare removed, and a config that setsretention_classesis refused at load. List the finite periods that sends choose explicitly inRetentionConfig.retention_periods.with_options(retention=...)replaceswith_options(retention_class_key=...): a whole-daytimedeltathat this process registers, orNonefor forever. Any other value, such as a class key string, returnsVALIDATION_FAILEDat the send call.RerunEnqueuePolicytakesretention: timedelta | Nonein place ofretention_class_key.PostgresBroker.enqueueandenqueue_asynctakeretention_class_key: RetentionClassKey | None; a key comes fromhorsies.core.history.periods.period_class_key, so a raw string is a type error.
- A broker refuses to initialize when the key of one of its periods is
registered with another duration. Only a class declared by name before
0.6.1 can hold such a key, for example
standard_14ddeclared with 7 days. The error names the key and both durations, and it is not retryable. - Breaking: the windowed monitoring reads take a
MonitoringWindow.list_tasks,task_stats,task_facetsandtask_breakdowntakewindow: MonitoringWindow, notHistoryWindow. Onlyresolve_monitoring_window(since=..., until=...)builds one, and it returnsMonitoringWindow | WindowRefused. So a window longer thanMONITORING_WINDOW_MAX(30 days) cannot reach a read, which a directly builtHistoryWindowcould. A directMonitoringWindowwith other bounds raisesAssertionError. horsies.monitoringexportsMonitoringWindow,WindowRefused,resolve_monitoring_window,MONITORING_WINDOW_DEFAULTandMONITORING_WINDOW_MAX.
- The
horsies.monitoringmodule docstring example passedoffset=andlimit=tolist_tasks, which takespage: TaskPage, and importedHistoryWindowfrom an internal module. The docs check now type-checks the python blocks of module docstrings too.
Upgrade notes
Section titled “Upgrade notes”- Replace
retention_classes=(RetentionClassConfig(key=..., duration=d),)withretention_periods=(d,), andwith_options(retention_class_key='<key>')withwith_options(retention=d).with_options(retention_class_key=None)becomeswith_options(retention=None). - Make each period whole days. Sub-day periods such as
timedelta(hours=36)are refused. - Classes from earlier versions stay registered:
q_<queue>_<duration>and named declarations. Their rows are deleted on their own durations, and new sends never stamp them. Remove nothing from the database. A class that no send stamps keeps about (days + 4) empty daily partitions. - A keyed send whose first attempt ran on 0.6.0, resent after the upgrade on a mapped queue or with an explicit 0.6.0 class key, is refused as a payload mismatch for the rest of its reservation window: the stamped class is part of the stored request fingerprint.
- A rerun can no longer target a class from an earlier version (
q_*or a declared name):RerunEnqueuePolicytakes a period. - The schema does not change.
- Upgrade the workers before the processes that only send. A 0.6.1 send on
a mapped queue stamps a
standard_<days>dclass, and a 0.6.0 worker does not register that class. Until a 0.6.1 worker registers it, such a row finds no history partition when the task finishes.
0.6.0 — 2026-11-03
Section titled “0.6.0 — 2026-11-03”0.6.0 moves task execution to child processes that the worker owns, closes the error types of the public API, and makes the JSON boundary strict. Read the 0.6.0 migration page before you upgrade.
- Stop rule: stop every process, upgrade all of them, then start. There is no rolling upgrade from 0.5.x to 0.6.0.
- Schema v52: one new empty table,
horsies_workflow_recovery_failures. - Worker-owned children: each slot has one child process and one socket.
ProcessPoolExecutor,multiprocessingand pickle are gone from the task path, and a timeout or a crash in one child touches no other task. - Closed error unions:
TaskSendError,BrokerOperationError,WorkflowStartError,HandleOperationError,MonitoringQueryErrorandTaskActionErrorare unions of frozen variants. Match on the variant. - Workflow recovery limit: a workflow whose recovery raises a persistent
error in
workflow_recovery_failure_limitreaper passes fails withWORKFLOW_RECOVERY_LIMIT_REACHED. - Strict type contract: a wrong value is refused at every boundary, not
converted.
.send(), schedules, workflow definitions and task results prove that a reader gets a value back before it is stored.@app.taskrefuses each shape that cannot cross the wire, andbuild_withfollows the same rules. A badTaskError.datano longer costs a task its error code. - Strict JSON boundary: every JSON parse goes through
codec/json_io.py.loads_jsonrefuses nesting deeper than 300 levels, a number outside the finite binary64 range and a lone UTF-16 surrogate.
Upgrade notes
Section titled “Upgrade notes”Stop every process before the upgrade
Section titled “Stop every process before the upgrade”- Stop every process, upgrade all of them, then start. There is no rolling
upgrade from 0.5.x to 0.6.0. Each version stores errors that the other
cannot decode: a 0.5.5 process gets
Unknown built-in task codefor a result that 0.6.0 stores withWORKFLOW_OUTPUT_SKIPPEDorWORKFLOW_RECOVERY_LIMIT_REACHED, and 0.6.0 cannot decode an error that a 0.5.5 process stores without a code. A 0.5.5 broker checks the schema version once, at its first use, so a 0.5.5 process that is already running does not stop on a v52 database.
Strict type contract
Section titled “Strict type contract”- Values are strict at every boundary. A conversion is refused:
"7",7.0orTruefor anint,"yes"for abool, an epoch number or a str for adatetime, a str for anEnum,UUIDorDecimal, a set for a list. A stored value is read through its JSON text in strict mode, so a datetime still reads from its string. A model withConfigDict(strict=True)now round-trips. - A value is proved before it is stored.
.send(), schedules, workflow definitions and task results decode what they encoded. A mutated instance,model_constructor a dataclass with a wrong field no longer fails in the worker withWORKER_SERIALIZATION_ERROR:.send()returnsTaskSendValidationFailedwith nothing stored. A task result gets one more decode in the child, in proportion to its size. - Data is not lost with no error. An unknown key is refused at any depth at send, and a subclass with fields that the declared type lacks is refused. An unknown key on read stays ignored, so old rows stay readable after a model change. The Deploying Code Changes page has a recipe for each model change.
- A workflow start stores what the spec proved. A list or a dict in a
JsonValuekwarg of a node failed at start withnon-JSON-native value: tuple. A start no longer encodes. - A node binds to named, typed parameters. Each key of
kwargsandargs_frombinds to one named, typed parameter of the target.*argsand**paramstake nothing from a node, in any form:args_frominto**paramsskipped the wiring check, and the child got a value with no declared type. A key that does not bind is refused when the spec is built. For an open set of values, declareparams: dict[str, JsonValue]. .send()refuses a call that cannot run. A missing required argument, and a task that requires a parameter only a workflow can fill.@app.taskrefuses each shape that cannot cross the wire.Field(exclude=True),exclude_if,extra='allow',PlainValidator,WrapValidator,SkipValidation, a schema hook,Json[T],Literalwith a plainEnum, a key that the read does not accept, a nestedTaskResult, a generator function, and a dataclass withrevalidate_instances. Each spelling gets the same answer. A dataclass field with a leading underscore is checked.- Defaults hold their declared type. A parameter or a field default is checked at definition, with no user code. An omitted parameter is validated before the task body runs, so the function gets its declared type. A list or a dict default is a fresh copy for each run.
Annotatedholds on a return type. A constraint or a discriminator was dropped. A stored value outside a constraint now reads asRESULT_DESERIALIZATION_ERROR.- A receiver of
args_fromgets the type that it declares, in a worker and inbuild_with. A constraint or a validator on the receiver now runs. - Workflow wiring.
Annotatedis ignored at every depth when types are compared.boolwired tointis refused.WorkflowDefinition[Any]cannot feedargs_fromor be an output. build_withfollows the task rules for eachSubWorkflowNodetarget, because its kwargs cross the wire. Hints that do not resolve are refused and no longer fall back toJsonValue.- A wrong return is
RETURN_TYPE_MISMATCH.return 5andreturn Noneno longer take the exception path, soexception_mapperanddefault_unhandled_error_codedo not apply. This replaces the earlier meaning ofTASK_EXCEPTIONfor a returnedNone: that code now means an exception outside the task body. A policy withauto_retry_for=['TASK_EXCEPTION']no longer retries a forgottenreturn. JsonInputis what a JSON position accepts.JsonValuestays what a reader gets: an exactlistordict. Alist[str]fitsJsonInput, and a tuple does not. At run timeJsonInputisJsonValue.JsonListandJsonDictare the protocols behind it. All three are exported fromhorsies.- An error keeps its code.
TaskError.datatakesJsonInputand givesJsonValue. A value that cannot cross becomes{'data_refused': <cause>}, and a lone surrogate in the text is escaped. The code stays, soauto_retry_forand routing still match. ASubWorkflowErrorkeeps its class. ATaskErrorsubclass with extra fields comes back as a plainTaskError. MAX_TYPED_JSON_DEPTHis 200, the deepest value under a declared type.MAX_JSON_DEPTHstays 300.
Worker process model
Section titled “Worker process model”- Workers run tasks in children that they own, one process and one socket
per slot.
ProcessPoolExecutor,multiprocessing, and pickle are gone from the task path. On Linux a preload parent forks each child. Other platforms start each child withexec:--child-preloadselects what the preload parent imports:library(the default),app, ornone.WorkerConfig.child_preloadis the same setting. It takes aChildPreloadfromhorsies.core.worker.config.- With
app, the worker refuses to start if the import leaves a Python thread or a socket. Off Linux,applogs a warning and usesexec. - A Linux host that cannot run the preload parent logs a warning and uses
exec: a kernel below 5.3, or a blockedprctl. A Python build with noos.pidfd_openkeeps preload, because the worker calls the syscall. - Horsies’ libraries load before your app. A setting that a library reads at
import, such as
PSYCOPG_IMPL, must be in the process environment, not in a.envfile that your app loads. A monkeypatch that must run before those imports, such as gevent, arrives too late; use--child-preload=none. - A recycled child keeps the library code that the preload parent imported,
and with
--child-preload=appit also keeps your app code. New code from an in-place upgrade needs a worker restart. max_tasks_per_child=0no longer forks from a parent that imported your app. Each child imports the app; use--child-preload=appto share that import.- A forked child shows the preload parent’s command line in
ps, so a monitor that countsmultiprocessing-forkprocesses needs a change. children_memory_mbcounts shared pages once per child, so it reads above real use. Read the container or cgroup total.multiprocessing.parent_process()returnsNonein a child, andmultiprocessing.current_process().nameishorsies-child-<slot>.- An app locator of
__main__:appis refused at worker start. A child does not run your script as__main__. - A child no longer imports
__main__again as__mp_main__.
timeout_msstarts when a child takes the task. Before, the clock started at dispatch, so time spent waiting for a free child counted. (#412)- A row waiting for a child returns to
PENDINGat worker stop. Before, it waited for its claim lease to end. max_tasks_per_childaccepts1. The floor was 2.max_memory_per_child_mbno longer depends on the CPython version. The CPython 3.13 gate is gone withhorsies.core.worker.mem_pool.ExecutorRestartFailedErroris renamedChildStartFailedError. There is no alias.WarmupIncompleteErrorandMemoryRecycleUnsupportedErrorare removed; the worker no longer raises them.
Definitions, config and tooling
Section titled “Definitions, config and tooling”-
Each startup error names a fix. Twenty refusals had no
help_text, and now each has one: an emptyWorkflowSpec, a blanknode_id, the fourjoinandmin_successrefusals, theWorkflowDefinitionclass checks, a repeateddefinition_key, a cached spec frombuild_with(), a kwarg that does not read back or that loses a field,node.key()with nonode_id, a failed broker start, and a refused signature. A test walks the source, so a new error site with nohelp_textor no code fails. -
Three startup errors now have a code.
node.key()with nonode_idraisesHRS-003(WORKFLOW_INVALID_NODE_ID).build_with()that returns a spec it did not build in this call raisesHRS-029(WORKFLOW_CHECK_BUILDER_EXCEPTION). TheircodewasNone. -
A refused signature reports its parts.
TaskDefinitionErrorfor a refused@app.tasksignature holds the banned type inmessage, the reason innotes, and the fix with the docs link inhelp_text. A refusedbuild_withreports the same parts.str()of aSignatureValidationErrordoes not change, and its newdetailattribute (SignatureRefusal) holds the parts. Its constructor now takes aSignatureRefusal, not a string. -
Breaking:
CustomQueueConfigrefuses a queue name that is empty, has whitespace at the start or the end, or is longer than 100 characters (the length of the databasequeue_namecolumn). The config raisesConfigurationError(HRS-200) when it is built. Only configs that were already unusable change: a task could not target such a queue, or the database refused the name. -
Breaking:
@app.task()refuses anasync deftask function withHRS-106(ErrorCode.TASK_ASYNC_FUNCTION_NOT_SUPPORTED, a new member). Define task functions withdef. Before, such a task registered and failed on every run. Registration refuses anasync deffunction, also one that yields. The check runs before the return-type checks, so anasync defwithout a return annotation getsHRS-106, notHRS-100.horsies checkreports it. -
Breaking: every config model refuses unknown keys.
AppConfig, its nested configs,ScheduleConfigand the schedule patterns raiseValidationErrorwith the typeextra_forbiddenfor a key that is not a field. Before, the key was dropped with no error. Remove or correct the key. The models that setextra='forbid'areAppConfig,PostgresConfig,RecoveryConfig,PayloadPolicy,CustomQueueConfig,RetentionClassConfig,RetentionConfig,WorkerResilienceConfig,ScheduleConfig, and the schedule pattern models (IntervalSchedule,HourlySchedule,DailySchedule,WeeklySchedule,MonthlySchedule,CronScheduleand its field and day models).TaskSchedulealready refused unknown keys. -
Breaking: a
WorkflowDefinitionsets its options withconfig = WorkflowDefinitionConfig(...), not a nestedclass Meta.WorkflowDefinitionConfigis a new export, andErrorCode.WORKFLOW_INVALID_DEFINITION_CONFIG(HRS-033) is a new member. A class that definesMeta, or setsconfigto a value of another type, raisesWorkflowValidationErrorwithHRS-033when the class is created. A nested class body cannot name the nodes of the enclosing class, so the documentedclass Meta: output = saveraisedNameError, and working code setMeta.outputafter the class. Before:class Pipeline(WorkflowDefinition[SaveResult]):name = 'pipeline'definition_key = 'myapp.pipeline.v1'save = TaskNode(fn=save_result)class Meta:output = Noneon_error = OnError.FAILPipeline.Meta.output = Pipeline.saveAfter:
class Pipeline(WorkflowDefinition[SaveResult]):name = 'pipeline'definition_key = 'myapp.pipeline.v1'save = TaskNode(fn=save_result)config = WorkflowDefinitionConfig(output=save, on_error=OnError.FAIL)WorkflowDefinitionConfig(output=None, on_error=OnError.FAIL, success_policy=None)is frozen and generic in the output type.WorkflowDefinition.configdefaults toWorkflowDefinitionConfig(); a subclass without its ownconfiguses the parent’s. The check runs for every workflow class when it is defined, whether or notbuild()runs, and refuses any attribute namedMeta, also one from a base class. TheHRS-025messages nameconfig.output. The run-timeWorkflowMetathat a task receives asworkflow_metadoes not change. -
Breaking:
@app.task('')raisesTaskDefinitionErrorwithTASK_INVALID_OPTIONS(HRS-102). Before, the task registered: a send returnedENQUEUE_FAILEDwithretryable=False, and a workflow start with the task as a root returnedVALIDATION_FAILED. -
Breaking: the environment variable
TASKLIB_SUPPRESS_SENDSis renamed toHORSIES_SUPPRESS_SENDS. The old name has no effect. SetHORSIES_SUPPRESS_SENDS=1where the old variable was set. Without it, a send or schedule is no longer a no-op, and it connects to the broker database. The send error messages and theTaskSendErrorCodedocumentation use the new name. -
Breaking: the agent skill files moved into one skill folder. The five files
SKILL.md,tasks.md,workflows.md,configs.mdandmonitoring.mdare now inhorsies/.agents/skills/horsies/, not inhorsies/.agents/skills/. OnlySKILL.mdhas frontmatter (name: horsies); the four reference files have none. Copy or link the new folder, and remove the old paths from any agent tool setup. -
Breaking:
horsies get-docsdownloads the docs of the release tagv{installed version}, not themainbranch. A development version (for example0.5.6.dev0or0.5.5+local) has no tag, and the command exits with code 1. Pass--ref mainto download themainbranch docs. -
Breaking:
timeout_mshas an upper bound.TaskOptions.timeout_mstakes 1000 toMAX_TASK_TIMEOUT_MS(4,294,967,295 ms, about 49.7 days), the limit of the Rust port’su32field.@app.taskrefuses a larger value withHRS-102(TASK_INVALID_OPTIONS). Before, no upper bound applied, and above about 1.8e308 ms the worker raisedOverflowErrorin the parent’s wait and in the child’s heartbeat thread. A queued row whose stored timeout is not an int from 1000 to that bound, or whose stored options do not parse, fails withWORKER_SERIALIZATION_ERRORwhen a worker claims it. The child sets the rowRUNNINGand fails it before it resolves the task, so no user code runs. A value outside the bounds retries ifWORKER_SERIALIZATION_ERRORis inauto_retry_for, and each retry fails the same way. Options that do not parse do not retry, because the retry policy is in them. Before, such a row ran with no timeout.
Send, enqueue and broker errors
Section titled “Send, enqueue and broker errors”- Breaking:
TaskSendErroris a union of nine frozen variants, one perTaskSendErrorCodemember. Code that readserr.task_id,err.payload,err.retryableorerr.exceptionon a send error must match on the variant first; pyright reports each site.TaskSendError(code=..., ...)in tests becomes a variant constructor.isinstance(err, TaskSendError)is not valid on a type alias; match on the variants.- The variants:
TaskSendSuppressed(message),TaskSendAsyncContext(task_id, payload, message),TaskSendValidationFailed(message, exception),TaskSendEnqueueFailed(task_id, payload, message, retryable, exception),TaskSendPayloadMismatch(task_id, payload, message),TaskSendPayloadTooLarge(message, size_bytes, limit_bytes),TaskSendIdempotencyKeyConflict(task_id, payload, message, owner_task_id),TaskSendEventLoopLimit(message)andTaskSendLoopRunnerFailed(task_id, payload, message, exception).err.codeis a class variable on each variant and keeps itsTaskSendErrorCodevalue. OnlyTaskSendEnqueueFailedhasretryable.VALIDATION_FAILEDandPAYLOAD_TOO_LARGEerrors carry no task id;PAYLOAD_TOO_LARGEcarries the size and the limit as fields. The repr shows the class name and the other fields;payload,messageandexceptionare not in it. exceptionisBaseException | NoCauseonTaskSendValidationFailedandTaskSendEnqueueFailed.NoCause()is the value when a check found the failure without a Python exception; before, it wasNone. Match on it beforeraise ... from,exc_info=or an error tracker call.TaskSendLoopRunnerFailed.exceptionisBaseException: always theLoopRunnerError.- The retry methods take a
TaskSendRetryable(TaskSendEnqueueFailed | TaskSendAsyncContext | TaskSendIdempotencyKeyConflict | TaskSendLoopRunnerFailed). Another variant is a type error, and at run time it fails an assertion that names the variant. Before, the methods returnedVALIDATION_FAILEDfor another code or for an error without a payload.
- The variants:
- Breaking:
BrokerOperationErroris a union of frozen variants. The nine operation codes fromSCHEMA_INIT_FAILEDtoLISTENER_SUBSCRIBE_FAILEDshareBrokerOperationFailed, which keeps the code in itscodefield; every other code has a variant of its own.BrokerOperationFailedCode, exported fromhorsies, is the type of thatcodefield. Every variant hascode,messageandretryable. Code that readserr.exceptionmust match on the variant first, or callbroker_error_cause(err); pyright reports each site.exceptionisBaseExceptiononBrokerOperationFailedandBrokerLoopRunnerFailed, andBaseException | NoCauseon the other variants that have it.BrokerOperationError(code=..., ...)in tests becomes a variant constructor.isinstance(err, BrokerOperationError)is not valid on a type alias; match on the variants. PostgresBroker.ensure_schema_initialized()returnsResult[None, BrokerOperationFailed | BrokerEventLoopLimit], its only failures. The run-time behavior does not change. In amatchon its error, an arm for another broker variant is dead; remove it. Strict pyright reports such an arm (reportUnnecessaryComparison) when it stands alone or in an OR pattern without parentheses, not inside a parenthesized OR pattern.PostgresBroker.enqueue_async()returnsResult[str, BrokerEnqueueError], andenqueue()returnsResult[str, BrokerEnqueueError | BrokerSyncCallError], their only failures.BrokerEnqueueErrorisBrokerEnqueueFailed | BrokerPayloadMismatch | BrokerIdempotencyKeyConflict | BrokerInvalidTaskOptions | BrokerInvalidInput | BrokerEventLoopLimit.BrokerSyncCallErrorisBrokerAsyncContext | BrokerLoopRunnerFailed. Both are new names, exported fromhorsiesandhorsies.core.brokers. The run-time behavior does not change. Code that assigns the result toBrokerResult[str]still type-checks. In amatchon the error, an arm forBrokerOperationFailed,BrokerNotAttached,BrokerNoTypeAvailable,BrokerInvalidJsonPayloadorBrokerListenerDisabledis dead; remove it. Strict pyright reports such an arm (reportUnnecessaryComparison) when it stands alone or in an OR pattern without parentheses, not inside a parenthesized OR pattern. A send or schedule whose broker returns one of these five variants (only a mock can) fails an assertion that names the variant. Before, it returnedENQUEUE_FAILED.- New broker codes
INVALID_INPUTandLISTENER_DISABLED. An exhaustivematchonBrokerErrorCodeneeds a case for each. These failures move to them:INVALID_INPUT: a non-positivetimeout_secondsormin_responsesofping_workers, or atarget_worker_idthat is not a string (before:WORKER_PING_FAILED); a non-positivelimitofget_worker_state_history(before:MONITORING_QUERY_FAILED); bothenqueued_atandenqueue_delay_seconds, a futuresent_atwithout either, or malformed args or kwargs JSON passed toenqueue()(before:ENQUEUE_FAILED).LISTENER_DISABLED:ping_workerson a broker built withassume_initialized=True(before:WORKER_PING_FAILED).
- Malformed
task_optionsJSON passed toenqueue()returnsINVALID_TASK_OPTIONS. Before, it returnedENQUEUE_FAILED.ENQUEUE_FAILEDnow means that schema initialization failed, or that the enqueue write failed or could not be confirmed. - Breaking:
enqueue_asyncrefuses bad input before the database, with its own code, and a send or schedule that the broker refuses withINVALID_INPUTorINVALID_TASK_OPTIONSreturnsTaskSendValidationFailed. A refused input used to fall into theexcept Exceptionof the write, so a direct call gotENQUEUE_FAILED, whose contract says the write may have happened, and a send got an error that the retry methods accepted. These now giveINVALID_INPUTand write nothing, and a send getsVALIDATION_FAILED: an empty task or queue name, a priority outside 1..100, a naivegood_until, a negativeenqueue_delay_seconds, an emptyretention_class_key, an empty idempotency key or one over 255 bytes, a naivesent_at, and a naiveenqueued_at. A naivesent_atraised a rawTypeErrorbefore, and a naiveenqueued_atwas stored under the session time zone with no error.args_jsonmust be null or an empty array, because the worker refuses every other row, andkwargs_jsonmust be an object or null. An empty string is refused for both, and a falsy value no longer turns into an empty container. The broker refused the input before the write: nothing was written, and the same payload is refused again. The retry methods do not acceptTaskSendValidationFailed; correct the input and send again. Before, the send returnedENQUEUE_FAILEDwithretryable=False, which the retry methods accept. - Breaking:
send_async(),schedule_async(), their retry methods andTaskHandle.get_async()let an exception from the broker call propagate, as the sync forms do. The broker still returns its database and connection failures as anErr, so these methods still returnTaskSendEnqueueFailedor aBROKER_ERRORresult for them. Before, they returned any other exception asENQUEUE_FAILEDor as aBROKER_ERRORresult. - Breaking: a failed broker construction propagates from every send and
schedule method.
app.get_broker()raisesHorsiesErrorwithBROKER_INIT_FAILED(HRS-211), a configuration error.send(),send_async(),schedule(),schedule_async()and their retry methods let it propagate. Before, they returned it asENQUEUE_FAILEDwithretryable=False. Correct the broker configuration; a retry does not clear this error. - Breaking: a code defect in the enqueue write crashes loud. The write now
catches database, network and history-contract errors only, so any other
exception leaves
enqueue_async, and so leavessend(). Schema init keeps its catch: an incomplete cutover is stillENQUEUE_FAILED.
TaskError contract and retired codes
Section titled “TaskError contract and retired codes”- Breaking:
ContractCode.PYDANTIC_HYDRATION_ERRORis removed. The library has not returned it since 0.1.2. Code that names it raisesAttributeError; match onRESULT_DESERIALIZATION_ERROR,BrokerErrorCode.INVALID_JSON_PAYLOADorRETURN_TYPE_MISMATCH. A stored error code{"__builtin_task_code__": "PYDANTIC_HYDRATION_ERROR"}does not decode:TaskHandle.get()returnsRESULT_DESERIALIZATION_ERRORandapp.get_result()returnsINVALID_JSON_PAYLOAD, each withUnknown built-in task codein the message. A result stored before 0.1.2 already fails to decode, because it has no__h_task_result__marker. - The string
PYDANTIC_HYDRATION_ERRORis no longer reserved.TaskError(error_code='PYDANTIC_HYDRATION_ERROR')makes a user code. Anexception_mapperordefault_unhandled_error_codevaluePYDANTIC_HYDRATION_ERRORno longer failshorsies checkwithHRS-212. The monitoring API and the web UI put a stored codePYDANTIC_HYDRATION_ERRORin theDOMAINcategory, notCONTRACT. - Breaking: the alias
ErrorCode.WORKFLOW_EXCESS_POSITIONAL_ARGSis removed. UseErrorCode.WORKFLOW_POSITIONAL_ARGS_NOT_SUPPORTED; the valueHRS-026does not change. Code that names the alias raisesAttributeError. - Breaking:
TaskErrorrefuses a key that it does not declare. The rule applies at construction and at decode, andSubWorkflowErrorinherits it. A stored error with an unknown key is stored data that does not parse: the refusal names the key, and each reader folds it as for any other stored error that does not decode. Before, the key was dropped, so{"nope": 1}read as an error with no code and no message. A task that returns aTaskErrorsubclass with its own fields fails withWORKER_SERIALIZATION_ERROR. Before, those fields were dropped. - Breaking:
TaskError.error_codeis required and must not be null, on the wire and in the type.message,dataandexceptionstay optional, anderror_codestaysBuiltInTaskCode | str. Before,TaskError(message='m')built an error with no code, andTaskError()built an empty one. Now each raisesValidationError(aValueError). In a task body, that exception ends the task withUNHANDLED_EXCEPTION. Pyright reports these calls. - Breaking: a stored error without a code does not decode. This includes
rows that an earlier version wrote with a null
error_code. No migration rewrites them.TaskHandle.get(),WorkflowHandle.get()andresult_for()returnRESULT_DESERIALIZATION_ERRORwith the cause.app.get_result()andapp.get_task_info(include_result=True)returnErr(BrokerInvalidJsonPayload).app.get_raw_result()still returns the stored value.
Sync calls and the event loop
Section titled “Sync calls and the event loop”-
Breaking: a sync call inside a running event loop returns
ASYNC_CONTEXT. Every sync facade fails closed on a thread that runs an event loop. It returnsASYNC_CONTEXTin its own error type, runs nothing, and does not start the broker’s sync runner. The message names the*_asyncform of the function that was called. Code that makes a sync call from a coroutine or a Jupyter cell must call the*_asyncform. Before, the call blocked that loop until the broker answered. An exhaustivematchonBrokerErrorCode,WorkflowStartErrorCode,HandleErrorCodeorRetrievalCodeneeds anASYNC_CONTEXTcase. The error type of each facade:BrokerErrorCodeforPostgresBroker.enqueue(),.get_task_info(),.get_raw_result_record()and.close(), forapp.get_result(),app.get_raw_result(),app.get_task_info(),app.ping_database(),app.ping_workers(),app.list_worker_states(),app.get_worker_state()andapp.get_worker_state_history(), and forTaskHandle.info().RetrievalCode, not cached, forTaskHandle.get(),WorkflowHandle.get()andWorkflowHandle.result_for(). ATaskHandlethat already holds its result still returns it.HandleErrorCodefor the syncWorkflowHandlemethodsstatus(),results(),tasks(),cancel(),pause()andresume(), and forpause_workflow_sync()/resume_workflow_sync().WorkflowStartErrorCodeforspec.start()andspec.retry_start().await spec.retry_start_async(err)completes that start with the error’s workflow id.
send(),schedule()and their sync retries already returnedTaskSendErrorCode.ASYNC_CONTEXT. A brokerASYNC_CONTEXTthat reachessend()keeps that code instead of becomingENQUEUE_FAILED. -
ASYNC_CONTEXTis now a reserved built-in code (RetrievalCode.ASYNC_CONTEXT). Anexception_mapperordefault_unhandled_error_codevalueASYNC_CONTEXTfailshorsies checkwithHRS-212, andTaskError(error_code='ASYNC_CONTEXT')is refused. -
An event loop limit at schema initialization returns
EVENT_LOOP_LIMIT. Before, a workflow start returned it asENQUEUE_FAILED, whichretry_start()accepted. -
Breaking: a sync method returns an
Erronly for a failure of the broker’s sync runner; every other exception propagates. This covers the sync methods ofPostgresBrokerandHorsies, the syncWorkflowHandlemethods,start_workflow()andspec.start(),pause_workflow_sync(),resume_workflow_sync(), and the syncTaskHandle.get(),.send()and.schedule(). The called coroutine still returns its expected failures as anErr. Before, the sync method returned any exception as its operation’s error: anErrwithTASK_INFO_QUERY_FAILED,ENQUEUE_FAILED,CLOSE_FAILED,DB_PING_FAILED,WORKER_PING_FAILED,MONITORING_QUERY_FAILEDorINTERNAL_FAILED, or aBROKER_ERRORresult. -
Breaking:
LOOP_RUNNER_FAILEDis a new member ofBrokerErrorCode,WorkflowStartErrorCodeandTaskSendErrorCode, andWorkflowStartErrorCode.INTERNAL_FAILEDis removed. A sync method returns it when the broker’s sync runner cannot run the call: the runner is stopped, its thread did not start, or scheduling failed. Nothing ran.retryableisFalse, andexceptionis theLoopRunnerError.close()andclose_async()stop the runner, and a stopped runner does not start again; call the*_asyncform or use a new broker. Before, the sync broker and app methods returned this failure as their operation’s code (for exampleCLOSE_FAILED), andstart_workflow()returnedINTERNAL_FAILED. An exhaustivematchonBrokerErrorCode,WorkflowStartErrorCodeorTaskSendErrorCodeneeds aLOOP_RUNNER_FAILEDcase.- A runner failure in
start_workflow()returnsLOOP_RUNNER_FAILED; any other exception propagates. Code that namesINTERNAL_FAILEDfails pyright. - A sync send or schedule whose broker runner cannot run the enqueue returns
TaskSendLoopRunnerFailed(task_id, payload, message, exception), notTaskSendEnqueueFailed. It has noretryablefield and keeps the task id and the payload. Nothing ran and nothing was written. The retry methods accept it: a stopped runner does not start again, so a sync retry returns the same error, andretry_send_async(err)orretry_schedule_async(err)completes the send. Before, the send returnedENQUEUE_FAILEDwithretryable=False. TaskHandle.get()maps it toBROKER_ERROR, not cached.HandleErrorCode.LOOP_RUNNER_FAILEDis unchanged.
- A runner failure in
Codec and JSON boundary
Section titled “Codec and JSON boundary”-
Breaking: the codec raises
ValueValidationError, not pydanticValidationError.encode_value,decode_valueand the task-result and kwargs functions inhorsies.core.codecraiseValueValidationErrorwhen a value fails its declared type, or when pydantic cannot serialize it. It is a subclass ofSerializationErrorand ofValueError, soexcept ValueErrorstill catches it;except ValidationErroraround these calls must catchValueValidationError.TaskSendValidationFailed.exceptionis aValueValidationErrorfor a kwarg that fails its declared type. -
Breaking:
RETURN_TYPE_MISMATCHdoes not store the returned value.datano longer hasactual_value, andexceptionisNone.dataaddsactual_typeandvalidation_error_count. Eachvalidation_errorsentry holds onlyloc,msgandtype, at most 10 entries. A returned value that fails its declared type and is not JSON-encodable now storesRETURN_TYPE_MISMATCH. Before, the mismatch error carried the value, could not be encoded, and the worker storedWORKER_SERIALIZATION_ERROR. -
Breaking:
encode_valuerefuses a lone surrogate in a string or in a dict key, at any depth. It raisesStrictJsonError, so it agrees withdumps_json. The text names the kwarg and holds no input value. A dict key is the larger change. Before,dump_python(mode='json')wrote U+FFFD in place of the surrogate of a key with a declared type: the send and the task succeeded with a changed key, and two keys could become one. For a key without a declared type, a rawUnicodeEncodeErrorleftencode_value. What each caller gets now:- A workflow node kwarg fails spec construction with
WORKFLOW_KWARGS_NOT_SERIALIZABLE(HRS-031). A child node inbuild_with()fails theSubWorkflowNodewithWORKFLOW_ENQUEUE_FAILED. Before, for a string,spec.start()returnedVALIDATION_FAILED, and a child start failed theSubWorkflowNodewithWORKER_SERIALIZATION_ERROR. .send()returnsTaskSendValidationFailed, with theStrictJsonErrorasexception.prepare_schedulesrefuses the schedule at boot withCONFIG_INVALID_SCHEDULE.- A task that returns such a value ends with
WORKER_SERIALIZATION_ERROR.
- A workflow node kwarg fails spec construction with
-
Breaking: a lone surrogate in the error text of a validator raises
StrictJsonErrorfromencode_valueanddecode_value. A validator that writes the refused value in its text, such asraise ValueError(f'bad name {value}'), is the usual cause. Pydantic cannot build such a text. The new text names the kwarg and holds no input value. Before, a rawUnicodeEncodeErrorleft both functions. What each caller gets now:.send()returnsTaskSendValidationFailed, with theStrictJsonErrorasexception. Before, it raised.prepare_schedulesrefuses the schedule withCONFIG_INVALID_SCHEDULE. Before, it raised.- A workflow node kwarg fails spec construction with
WORKFLOW_KWARGS_NOT_SERIALIZABLE(HRS-031). Before,app.workflow()raised. - A task whose return value hits this ends with
RETURN_TYPE_MISMATCH, in a direct call and in a worker. The task wrapper refuses the value, as it refuses any other value that does not validate, and the value is not stored. Before, the task ended withUNHANDLED_EXCEPTION, or in a worker withTASK_EXCEPTIONfor the context of aPydanticCustomError. - A decode of the kwargs of a task ends with
WORKER_SERIALIZATION_ERROR, notTASK_EXCEPTION. - A decode of a stored result gives
RESULT_DESERIALIZATION_ERROR. Before, the raw error left the read.
-
Breaking:
encode_valueraisesValueValidationErrorfor a dict key without a JSON form. Such a key is afrozenset, or an Enum member whose value is a list, a dict or a set. The text names the kwarg and the type. Before, a rawTypeErrorleftencode_value. Registration refuses these types in a task signature (HRS-105), so.send()and a schedule do not reach this. ASubWorkflowNodekwarg fails spec construction withWORKFLOW_KWARGS_NOT_SERIALIZABLE(HRS-031). Before,app.workflow()raised. A task whoseTaskError.dataholds such a key ends withWORKER_SERIALIZATION_ERROR, notTASK_EXCEPTION. Any otherTypeErrorfrom the dump is a defect and propagates, for example from a computed field. -
Breaking:
encode_valueraisesValueValidationErrorfor a value that holds itself, at a position that is not aJsonValue. Such a value is a list or a dict that contains itself, or an instance of a self-referencing model that links to itself. The text names the kwarg. Before, a rawValueErrorleftencode_value. What each caller gets now:.send()returnsTaskSendValidationFailed, with theValueValidationErrorasexception. Before, it raised.prepare_schedulesrefuses the schedule withCONFIG_INVALID_SCHEDULE. Before, it raised.- A workflow node kwarg fails spec construction with
WORKFLOW_KWARGS_NOT_SERIALIZABLE(HRS-031). Before,app.workflow()raised. - A task that returns such a value, or whose
TaskError.dataholds one, ends withWORKER_SERIALIZATION_ERROR, notTASK_EXCEPTION.
Any other
ValueErrorfrom the dump propagates. At aJsonValueposition the JSON boundary refuses the value; the entry on nesting depth states that outcome. -
Breaking: the JSON boundary refuses nesting deeper than 300 levels, and
encode_valueraisesValueValidationErrorfor a value that is nested too deep.MAX_JSON_DEPTHinhorsies.core.codec.json_valueis the bound. It counts nested lists and dicts.- Pydantic refuses a dump above 255 levels of nesting, and it says
Circular reference detected (depth exceeded)also for a finite value.encode_valueraisesValueValidationErrorfor such a value, and the text names the kwarg. Before, a rawValueErrorleftencode_value. Each caller gets the same outcome as for a value that holds itself:TaskSendValidationFailedfrom.send(),CONFIG_INVALID_SCHEDULEfromprepare_schedules,WORKFLOW_KWARGS_NOT_SERIALIZABLE(HRS-031) fromapp.workflow(), andWORKER_SERIALIZATION_ERRORfor a task, notTASK_EXCEPTION. dumps_jsonreturnsErr, andencode_valueraisesStrictJsonErrorat aJsonValueposition, for a deeper value and for a value that holds itself. Before,RecursionErrorleftdumps_json, which returns aSerdeResult, and.send(),prepare_schedulesandapp.workflow(). These now giveTaskSendValidationFailed,CONFIG_INVALID_SCHEDULEandWORKFLOW_KWARGS_NOT_SERIALIZABLE(HRS-031).loads_jsonandloads_json_bytesreturnErrfor a deeper text. Before, a text of 1,000 or more levels raisedRecursionErrorout of the parse or out of the next walk. A stored task or workflow result is nowRESULT_DESERIALIZATION_ERROR, kwargs in a worker areWORKER_SERIALIZATION_ERROR, andenqueue_asyncreturnsBrokerInvalidInput.- Each value that
encode_valueaccepts stays inside the bound, because pydantic refuses a dump above 255 levels. A text of 301 or more levels that adumps_jsoncaller stored before this release no longer parses.
- Pydantic refuses a dump above 255 levels of nesting, and it says
-
Breaking:
encode_valueraisesValueValidationErrorforbytesthat are not UTF-8. This holds at a declaredbytesposition and atAny, also for a dict key. The text is fixed and holds no byte value. Before, a rawUnicodeDecodeErrorleftencode_value. Abuild_with()kwarg fails spec construction withWORKFLOW_KWARGS_NOT_SERIALIZABLE(HRS-031). ATaskError.dataends the task withWORKER_SERIALIZATION_ERROR, notTASK_EXCEPTION. Registration refusesbytesin a task signature, so.send()and a schedule do not reach this. AUnicodeDecodeErrorfrom a decode in a computed field propagates.
Workflow start
Section titled “Workflow start”- Breaking:
WorkflowStartErroris a union of frozen variants, one perWorkflowStartErrorCodemember. Every variant hascode,message,retryable,workflow_nameandworkflow_id. Code that readserr.exceptionmust match on the variant first; pyright reports each site.exceptionisBaseExceptiononWorkflowStartEnqueueFailedandWorkflowStartLoopRunnerFailed, andBaseException | NoCauseonWorkflowStartValidationFailed; the other variants have no cause.WorkflowStartError(code=..., ...)in tests becomes a variant constructor.isinstance(err, WorkflowStartError)is not valid on a type alias; match on the variants. - New code
WorkflowStartErrorCode.WORKFLOW_ID_CONFLICT. A start whose workflow id exists for another workflow name returnsWorkflowStartIdConflict, with the other name inexisting_workflow_name. Nothing was written; use another id. Before, it returnedVALIDATION_FAILED. An exhaustivematchonWorkflowStartErrorCodeneeds a case for it. - Breaking:
retry_start()andretry_start_async()takeWorkflowStartRetryable. It isWorkflowStartEnqueueFailed | WorkflowStartAsyncContext | WorkflowStartLoopRunnerFailed; each carries the id of its start, and the retry starts again with that id. Another variant is a type error, and at run time it fails an assertion that names the variant. Before, the retry methods took every code and returnedVALIDATION_FAILEDfor a code other thanENQUEUE_FAILED. An error of another workflow still returnsVALIDATION_FAILED. - Breaking: a defect in the workflow start transaction propagates. The
start transaction catches only
sqlalchemy.exc.SQLAlchemyErrorandpsycopg.Error, and returns them asWorkflowStartEnqueueFailed. Any other exception propagates. Before, the start returnedSerializationErrorandValueErrorasVALIDATION_FAILED, and every other exception asENQUEUE_FAILED. - Breaking: a workflow task node with a
priorityoutside 1–100 or a naivegood_untilis refused when the spec is built.app.workflow(),WorkflowDefinition.build()andWorkflowSpec(...)raiseWorkflowValidationErrorwithWORKFLOW_INVALID_PRIORITY(HRS-034) orWORKFLOW_NAIVE_GOOD_UNTIL(HRS-035). The check covers every task node, root or not, andhorsies checkreports it. A child spec with such a node fails its build, and the engine marks the subworkflow nodeWORKFLOW_ENQUEUE_FAILED. Before, a root node made the start returnVALIDATION_FAILED, and a node that is not a root started and failed later, when the engine enqueued it. - Breaking: a defect in the workflow start checks propagates. The checks
before the start transaction return only a
WorkflowValidationErrorasWorkflowStartValidationFailed. Any other exception propagates, aValueErrorincluded. A node that has positional args when it reaches the start transaction (only possible throughobject.__setattr__after construction) fails anassertthat names the node. A task node withpriority=None, in a spec thatapp.workflow()did not build, getsWorkflowStartValidationFailedfrom the checks, withWORKFLOW_UNRESOLVED_PRIORITY. Before, the start returned aValueError,MultipleValidationErrorsorSerializationErrorfrom the checks, and aValueErrorfrom the transaction, asVALIDATION_FAILED. - An exhaustive
matchonErrorCodeneeds cases forWORKFLOW_INVALID_PRIORITY,WORKFLOW_NAIVE_GOOD_UNTIL,WORKFLOW_INVALID_DEFINITION_CONFIG,TASK_ASYNC_FUNCTION_NOT_SUPPORTEDandCHECK_LIVE_ASYNC_CONTEXT. - Breaking: a
SerializationErrorfrom the workflow start, a child workflow start or the workflow engine propagates.- The workflow start transaction encodes only values that the process built
and validated before the start: the options that the task decorator
serialized, and the node kwargs that spec construction encoded and
decoded. A failure there is a defect, for example
task_options_jsonthat was changed after registration. Before, the start returned it asVALIDATION_FAILED. The transaction rolls back as before, and nothing is written. - The workflow start transaction raises
SerializationErrorfor task options that are JSON but not an object when it merges thegood_untilof a node. The options are the task decorator’s serialization of a validatedTaskOptions, so such a value is a defect, for exampletask_options_jsonthat was changed after registration. The message names the node and the type of the value. The transaction rolls back, and nothing is written. Before, the start replaced the options with{}and stored onlygood_until. The child workflow start already raises for the same value. - When the engine starts the child workflow of a
SubWorkflowNode, it parses the task options of each child root and of each child task withgood_until. It reads them from the task function, as the workflow start transaction does. Options that do not parse to a JSON object are a defect, for exampletask_options_jsonthat was changed after registration. The child start raisesSerializationErrorwith the node name before it writes a row. The caller rolls back its transaction:start_workflow()andresume()raise the error, the worker’s workflow finalize phase returns a finalize error, and the reaper logs the recovery candidate as failed and counts the raise towardworkflow_recovery_failure_limit. Before, a child root with such options endedFAILEDinside the child workflow, and a child task withgood_untilfailed theSubWorkflowNodewithWORKER_SERIALIZATION_ERROR. The promotion of a node whose stored options do not parse still fails only that node. - A child workflow start raises
SerializationErrorwhen the kwargs of a child node do not encode. Spec construction inbuild_with()encodes and decodes every child kwarg, so a refusal at the child start is a defect. The message names the child node. Before, the child start failed theSubWorkflowNodewithWORKER_SERIALIZATION_ERROR. Stored kwargs of the parent node that do not decode still fail only that node. - A value that the workflow engine built and that does not serialize raises
SerializationError. This is true for the kwargs of a promoted node, for each value that a child workflow start writes, for the result of a completed node, for the parent node result and the summary of a finished child workflow, and for the workflow error. The engine builds these values in the process, some of them from values that it parsed from stored rows, so a failure is a defect. The message names the value. The caller rolls back its transaction: the worker’s workflow finalize phase returns a finalize error, the reaper logs the recovery candidate as failed and counts the raise towardworkflow_recovery_failure_limit, andstart_workflow()andresume()raise the error. Before, the engine logged the failure and stored the textnullor NULL in the result or error column, or failed the node withWORKER_SERIALIZATION_ERROR. The outputless workflow result keeps its fallback.
- The workflow start transaction encodes only values that the process built
and validated before the start: the options that the task decorator
serialized, and the node kwargs that spec construction encoded and
decoded. A failure there is a defect, for example
Workflow handle and result reads
Section titled “Workflow handle and result reads”- Breaking:
HandleOperationErroris a union of frozen variants, one perHandleErrorCodemember.HandleResult[T]and the method signatures do not change. Every variant hascode,message,retryableandworkflow_id. Code that readserr.exceptionmust match on the variant first; pyright reports each site.exceptionisBaseExceptiononHandleDbOperationFailed,HandleStoredDataInvalid,HandleResumeRecoveryFailedandHandleLoopRunnerFailed; the other variants have no cause.HandleEventLoopLimitcarriesloop_limit.HandleOperationError(code=..., ...)in tests becomes a variant constructor.isinstance(err, HandleOperationError)is not valid on a type alias; match on the variants. - Breaking:
HandleErrorCode.INTERNAL_FAILEDis removed, and a defect inpause_workflow()orresume_workflow()propagates. They catch onlysqlalchemy.exc.SQLAlchemyError. Before, they returned every other exception, also a failedassert, asINTERNAL_FAILED. Code that namesINTERNAL_FAILEDfails pyright. The web pause and resume routes answer such a defect with HTTP 500, not 503. - New code
HandleErrorCode.STORED_DATA_INVALID.tasks()returnsHandleStoredDataInvalidwith the parse error for a subworkflow summary that does not parse; before, it returnedDB_OPERATION_FAILEDwithretryable=Falseand no cause.status()andcancel()return it for a stored workflow status that is not aWorkflowStatus; before, theValueErrorpropagated. Nothing changed, and the same call returns the same error.get()folds it intoBROKER_ERROR. The web routes answer it with HTTP 500. An exhaustivematchonHandleErrorCodeneeds a case for it. - New code
HandleErrorCode.RESUME_RECOVERY_FAILED.resume()returnsHandleResumeRecoveryFailedwhen the resume committed and the recovery pass for the workflow tree, which runs after the commit in a second transaction, fails with a database error. The workflow isRUNNING; the worker reaper runs the same pass over all workflows. Before, it returnedDB_OPERATION_FAILED, the code of a failure before the commit. The web resume route answers it as resumed with thepost_resume_recovery_failedwarning and does not read the status again; aDB_OPERATION_FAILEDfrom resume is always HTTP 503. An exhaustivematchonHandleErrorCodeneeds a case for it. - Breaking:
WorkflowHandle.results()andtasks()returnHandleWorkflowNotFoundfor an id that has no workflow. The same is true forresults_async()andtasks_async(). Each method reads the workflow row and its nodes in one statement. Before, an unknown id gaveOk({})andOk([]), the same values as an existing workflow without results or nodes. An existing workflow before its first stored result still givesOk({}). - Breaking:
WorkflowHandle.get()returnsRetrievalCode.WORKFLOW_NOT_FOUNDwhen the workflow row is removed after its status was read. Retention can delete a terminal workflow between the status read and the result read or the error read. Before,get()returnedTaskResult(ok=None), a success, for a completed workflow, andOutcomeCode.WORKFLOW_FAILEDfor a failed, cancelled or expired workflow. A terminal workflow without a stored error gets theOutcomeCodeof the status thatget()read (WORKFLOW_FAILED,WORKFLOW_CANCELLEDorWORKFLOW_EXPIRED), andget()no longer makes a free-textWORKFLOW_<status>code. - Breaking:
WorkflowHandle.get()returnsRESULT_DESERIALIZATION_ERRORfor a stored workflow error that is JSON but not an object. This is true for a failed, cancelled or expired workflow. The message names the Python type of the decoded value, for examplegot list. Before,get()read such a value as no stored error and returned theOutcomeCodeof the status (WORKFLOW_FAILED,WORKFLOW_CANCELLEDorWORKFLOW_EXPIRED). - Breaking: a defect in the stored-error decode of
WorkflowHandle.get()propagates.get()returnsRESULT_DESERIALIZATION_ERRORonly for the two refusals of the decode,StrictJsonErrorandValueValidationError. Before, it returnedRESULT_DESERIALIZATION_ERRORfor every exception from the decode. - Breaking:
WorkflowHandle.get()returnsRESULT_DESERIALIZATION_ERRORfor aCOMPLETEDworkflow whose stored result is NULL or empty, and for an empty stored workflow error. The library stores a result in the statement that setsCOMPLETED, and it does not store an empty error. So these values are stored data that does not parse. The message names the case, anddataholds the workflow id. Before, the result case returnedTaskResult(ok=None), a success, and an empty error returned theOutcomeCodeof the status. A NULL error still gives theOutcomeCodeof the status: a cancel does not write the error. - Breaking:
WorkflowHandle.get()returnsOutcomeCode.WORKFLOW_OUTPUT_SKIPPEDfor aCOMPLETEDworkflow whose output node isSKIPPED. A success case that does not require the output node lets this state occur. The workflow stores an error envelope with the new code, anddatanames the workflow and the output node. ASubWorkflowNodeof such a child holds that error, soresult_for(), a dependent withallow_failed_deps=Trueand theget()of a parent see the same code. Before, the workflow stored the textnull,get()returnedRESULT_DESERIALIZATION_ERROR, and a downstream node receivedRESULT_DESERIALIZATION_ERRORwithStored TaskResult JSON corrupt. Code that reads this state throughRESULT_DESERIALIZATION_ERRORmust match on the new code. A workflow that already stored the textnullkeeps it, andget()returnsRESULT_DESERIALIZATION_ERRORfor it. AFAILEDoutput node does not change: the workflow stores that node’s error. An exhaustivematchonOutcomeCodeneeds aWORKFLOW_OUTPUT_SKIPPEDcase. The code string is now reserved: anexception_mapperordefault_unhandled_error_codewith that value failshorsies checkwithHRS-212. - Breaking: the
WorkflowHandle.get()map of an outputless workflow holdsOutcomeCode.WORKFLOW_OUTPUT_SKIPPEDfor aSKIPPEDterminal node. A success case that does not require the node lets this state occur.datanames the workflow and the node. Before, the entry heldRESULT_DESERIALIZATION_ERROR, although nothing is corrupt. An entry of aCOMPLETEDorFAILEDnode that does not parse keepsRESULT_DESERIALIZATION_ERROR, and so does a stored entry of a workflow that already completed. - Breaking: result waits refuse a
timeout_msthat no wait can use.TaskHandle.get()/get_async()andWorkflowHandle.get()/get_async()return the newRetrievalCode.INVALID_INPUT, not cached.PostgresBroker.get_raw_result_record()/_async()andHorsies.get_result()/get_raw_result()and their_asyncforms returnBrokerInvalidInput(INVALID_INPUT). Each refuses before any read: a negative value, a bool, a value that is not an int, or a value aboveMAX_TASK_TIMEOUT_MS.0still checks once with no wait. Before, a negative value worked like0, and a value above the float range raisedOverflowErrorfromWorkflowHandle.get_async()and came back asBROKER_ERRORfromTaskHandle.get(). Add anINVALID_INPUTcase to each exhaustivematchonRetrievalCode.
Workflow recovery limit
Section titled “Workflow recovery limit”- Breaking: a workflow whose recovery raises on every pass now fails.
After
workflow_recovery_failure_limitreaper passes (default 25) with a persistent raise, the workflow fails closed withWORKFLOW_RECOVERY_LIMIT_REACHED. A persistent raise is a defect, stored data that the engine cannot use, or a SQL error outsideOperationalError. Before, the reaper logged the raise on each pass, and the workflow stayedRUNNING.- At the limit, the node that raised goes
FAILED, otherPENDINGandREADYnodes goSKIPPED, a stored first error stays, and live nodes finish. - The new setting
RecoveryConfig.workflow_recovery_failure_limithas the default 25 and the range 3–1000. - A workflow in that state when you upgrade reaches the limit and fails: at the default check interval, that is about 720 seconds after the first pass. Set a higher limit to get more time for a fix.
- A connection, lock or timeout error does not count, so a database outage cannot fail a workflow.
- The count is stored in the new table
horsies_workflow_recovery_failures, so a worker restart keeps it, and all workers share it. The count has no reset, so a later unrelated raise after a repair reaches the limit sooner. - The worker health surface gets
failures_countedandlimit_reachedper case,over_failure_limitfor each workflow recovery pass, andraised_countedfor each phase-2 pass.
- At the limit, the node that raised goes
- Breaking: a phase-2 row whose progression raises a persistent error now
counts toward
phase2_quarantine_after_attempts. At that bound the row moves to quarantine, and the workflow fails withWORKFLOW_RECOVERY_LIMIT_REACHED. Before, the raise did not count, and the row was retried on every pass. - An exhaustive
matchonOperationalErrorCodeneeds aWORKFLOW_RECOVERY_LIMIT_REACHEDcase. The code string is now reserved: anexception_mapperordefault_unhandled_error_codewith that value failshorsies checkwithHRS-212. - Schema v52 adds the table
horsies_workflow_recovery_failures. The table is new and empty, and there is no data step. The first schema check of a 0.6.0 broker applies v52. Ahorsies workerorhorsies schedulerstart runs that check. So doScheduler.start(), a workflow start, an enqueue, a result read and a task info read, so a process that only sends tasks can apply v52. A broker built withrun_schema_migrations=Falseorassume_initialized=Truedoes not apply it. A 0.5.5horsies workerorhorsies schedulerrefuses to start on a v52 database, withdatabase schema v52 is newer than this build supports (v51).
Subworkflow node outcome
Section titled “Subworkflow node outcome”- Breaking: a
SubWorkflowNodeisFAILEDwhen its child isCOMPLETEDwith an err output. A node isCOMPLETEDonly when its result is ok, as for a task node. A success case that does not require the child’s output node lets that node endSKIPPEDorFAILED. The parent node holds that err, soon_errorruns, a dependent withallow_failed_deps=FalseisSKIPPED,join="any"andjoin="quorum"do not count the node, aSuccessCasethat requires the node is not met, and a parent without a success policy isFAILED.SubWorkflowSummary.statusstaysCOMPLETED. Before, the node wasCOMPLETED, and a dependent with the default options ran and received the err. To let a parent complete, keep the node out of its success cases. To run a dependent, setallow_failed_deps=True. A child with no output node is not affected. - Breaking: a
SubWorkflowNodeisFAILEDwithRESULT_DESERIALIZATION_ERRORwhen the stored result of itsCOMPLETEDchild is not a well-formed result envelope, or is NULL or empty. The node holds an error envelope that names the child and the cause, so each reader and theget()of the parent report that code. The stored text stays on the child workflow row. Before, the node wasCOMPLETEDand held that text.
Schedules
Section titled “Schedules”- Breaking: a schedule whose kwargs exceed
payload.reject_bytesfails at scheduler start.Scheduler.start()raisesConfigurationError(HRS-205,CONFIG_INVALID_SCHEDULE), andhorsies checkreports the schedule. The notes give the size and the limit. Disabled schedules are not checked. Before, the scheduler started and skipped every slot of that schedule. - Breaking: schedule kwargs are encoded once, at scheduler start. Every
slot sends that text until the next start. A kwargs model field with a
default_factory, or a validator that reads the clock, runs once per start and not once per fire, so the task receives the same value on every fire. If such a validator refuses the value after the start, the scheduler still enqueues the task, and the task fails at the worker withWORKER_SERIALIZATION_ERROR. Before, the scheduler logged an error and skipped the slot.
Monitoring, task actions and rerun
Section titled “Monitoring, task actions and rerun”- Breaking:
RerunSourceAbsent.predates_retained_flooris nowRerunSourceAbsent.reason, aTaskAbsenceReason.TrueisPREDATES_RETAINED_HISTORY,FalseisWITHIN_RETAINED_HISTORYandNoneisNOT_CLASSIFIED.TaskAbsenceReasonis the enum thatTaskDetailNotFound.reasonuses; import it fromhorsiesorhorsies.monitoring. Code that readspredates_retained_floormust match onreason. - Breaking:
MonitoringQueryErroris a union of three frozen variants, one perMonitoringQueryErrorCodemember. Every variant hascode,messageandretryable.exceptionisBaseExceptiononMonitoringQueryDbOperationFailedandMonitoringQueryHistoryContractViolation.MonitoringQueryEventLoopLimithas noexceptionand carriesloop_limit. Code that readserr.exceptionmust match on the variant first; pyright reports each site.MonitoringQueryError(code=..., ...)in tests becomes a variant constructor.isinstance(err, MonitoringQueryError)is not valid on a type alias; match on the variants. - New codes for stored data that does not parse:
MonitoringQueryErrorCode.HISTORY_CONTRACT_VIOLATIONandTaskActionErrorCode.STORED_DATA_INVALID.- A monitoring query that reads history data that does not decode returns
MonitoringQueryHistoryContractViolation, with theHistoryContractErrorinexception.list_tasksreturned it before asDB_OPERATION_FAILEDwithretryable=False.get_task_detailandget_workflow_noderaisedHistoryContractErrorbefore; now they return the variant. An exhaustivematchonMonitoringQueryErrorCodeneeds a case for it. The web read routes answer 500 for it, and 503 for the other two codes, as before. cancel_taskreturnsTaskActionStoredDataInvalidwhen stored data does not parse:exceptionis theHistoryContractErrorof the history read or theOutcomeDecodeErrorof the cancel outcome. Nothing changed. Before, both exceptions propagated. An exhaustivematchonTaskActionErrorCodeneeds a case for it.POST /api/tasks/{task_id}/cancelanswers 500 for it.- A history row whose stored status is not a
TaskStatusmember is stored data that does not parse. The history detail read parses the status one time and raisesHistoryContractErrorfor such a value.get_task_detailandget_workflow_nodereturnHISTORY_CONTRACT_VIOLATION; before, they returned the text as the status.cancel_taskreturnsSTORED_DATA_INVALID; before, it returnedTASK_NOT_CANCELLABLEwithcurrent_status=None.rerun_taskreturnsRerunSourceCorrupt; before, it treated the row as a source that is notCOMPLETED.
- A monitoring query that reads history data that does not decode returns
- Breaking:
TaskActionErroris a union of six frozen variants, one perTaskActionErrorCodemember. Every variant hascode,message,retryableandtask_id.current_statusis only onTaskActionNotCancellableandTaskActionWorkflowTask, and it is aTaskStatus. Before, every error hadcurrent_status: TaskStatus | None.TASK_IS_WORKFLOW_TASKnow carries the status.TaskActionDbOperationFailed.exceptionholds the database error, andTaskActionEventLoopLimit.loop_limitholds the limit.TaskActionError(code=..., ...)in tests becomes a variant constructor.isinstance(err, TaskActionError)is not valid on a type alias; match on the variants. TaskActionErrorCodeis astrenum.TaskActionErrorCode.TASK_NOT_FOUND == 'TASK_NOT_FOUND'is now true.- Breaking:
get_task_detailreturnsTaskDetailNotFoundfor a task that does not exist, notOk(None). Itsreasonis aTaskAbsenceReason:PREDATES_RETAINED_HISTORY(retention can have removed the task),WITHIN_RETAINED_HISTORY(retained history covers the id and has no row) orNOT_CLASSIFIED. Both are exported fromhorsies.monitoring. Code that checksdetail is Nonemust match onTaskDetailNotFound. The web route still answers 404; forPREDATES_RETAINED_HISTORYthe detail text says that the id is older than the retained task history.get_task_detailandget_workflow_nodenow read in one REPEATABLE READ snapshot. Before, a task row that was committed between the live read and the history read gaveOk(None). - New rerun outcome
RerunSourceCorrupt.rerun_taskreturns it, with theHistoryContractErrorasexception, when the source history row does not decode: for example a stored status that is not aTaskStatusmember. Nothing is written. Before,rerun_taskraised the error into the caller’s transaction. It is exported fromhorsies. An exhaustivematchonRerunOutcomeneeds a case for it. - Breaking:
list_tasks,task_breakdownandlist_workflow_runstake a page value.list_taskstakespage: TaskPagein place ofoffsetandlimit,task_breakdowntakeslimit: BreakdownLimit, andlist_workflow_runstakeslimit: WorkflowRunsLimit. Build each withparse, which returnsErr(PageRefused)for a value outside its bounds:offset0 or more,limit1 to 200 andoffset + limitat most 500 forlist_tasks,limit1 to 500 fortask_breakdown, andlimit1 to 200 forlist_workflow_runs.list_tasksno longer raisesValueError. Before, a negativelimitgave a different result in each read, and a directlist_taskscall with alimitfrom 201 to 500 ran; it is now refused. Web answers do not change.
Changed
Section titled “Changed”-
/api/tasks/facetsscans the window once. A poll ran four group-bys and four category counts over the history window. It is now one statement. On 1.4M history rows, one poll over the default 24 hours falls from 46,750 buffers to 5,187, and from 90–108 ms to 16–17 ms. The answer does not change. -
Open dashboards share
/api/tasks/facetsand/api/tasks/breakdown. Both answer through a 10-second cache per scope, as the status cards do. One load serves every concurrent request, and each count is at most 10 seconds old. -
A worker claims once per wake. The main loop ran two claim passes for each wake. With 8 children and 20 tasks/s,
horsies_claimcalls for 400 tasks fall from 1,524–1,602 to 799–807, and the start lag does not change. A stop no longer runs one more claim. With--max-claim-batch, a pass that claims runs the next pass with no wait, because noNOTIFYannounces work that a limited pass leaves. -
The first runner heartbeat is part of the RUNNING statement. Each task start costs one round trip fewer. A start that does not apply writes no beat.
-
Worker children on Linux start in milliseconds and share memory. A preload parent imports code once, and each child is a fork of it. Each pair below compares
--child-preload=noneof this release with the defaultlibrary, or withappwhere the text names it. With 8 children and task work against Postgres, on a small app: next child 618 ms to 27 ms, summed PSS 461 MiB to 169 MiB. With--child-preload=app, on an app that imports pandas: next child 1,174 ms to 26 ms, summed PSS 715 MiB to 236 MiB. With the defaultmax_tasks_per_child=100and 10,000 no-op tasks, throughput goes from 437 to 803 tasks/s with a hard cap, and from 484 to 1,032 tasks/s with prefetch 8. The worker stays the real parent of each child, and a dead preload parent stops no running task. -
WorkerConfig.drain_timeout_sandhorsies worker --drain-timeoutset the shutdown drain. The default stays 300 seconds. At the deadline the worker kills the children that still run a task, and each task takes the crash path, so its row does not wait for the reaper. Set it below the stop window of your supervisor. -
The top-level
errorsof theworkflow_recoveryhealth counts every recovery case. Before, it left outcase_0,case_1,case_1_5andcase_1_6. -
The parent advancement after a global workflow completion runs in its own SAVEPOINT. A raise there no longer rolls back the completion of the child, so the child ends, and recovery counts the raise on the parent workflow.
-
SuccessCase.requiredandSuccessPolicy.optionalaccept any node. The element type isAnyNode(TaskNode[Any] | SubWorkflowNode[Any]). Before, it wasTaskNode[Any], but spec validation already accepted aSubWorkflowNodethere, so the call needed a cast. The change widens a public type. A caller that passesTaskNodevalues is not affected. -
Action responses no longer carry
next_attempt_number. The field was alwaysnullafter the manual retry was removed. The web UI action responses keepoutcome,was_statusandwarning. -
New command
horsies skills install --target DIR [--force]. It copies the skill folder from the installed package toDIR/horsies/.--targetis required, because each agent tool reads a different skills directory. The command leaves an identical copy as it is, refuses to replace a copy that differs unless--forceis given, and exits with code 1 on any failure.get-docshas a new option--ref main. -
The release workflow checks that the built wheel contains the skill folder.
-
Three operations guides are new: Atomic Enqueue and Topology, Deploying Code Changes and Testing. The Testing guide documents
HORSIES_SUPPRESS_SENDS. Two guides are new: Async Frameworks and Start a Workflow Once per Entity. -
TaskSendErrorCode.IDEMPOTENCY_KEY_CONFLICTis a new member. A send whose idempotency key is reserved by another task with a different command returnsTaskSendIdempotencyKeyConflictinstead ofENQUEUE_FAILED. Itsowner_task_idnames the task that holds the key. The retry methods accept the error; a retry after the reservation ends is accepted. An exhaustivematchonTaskSendErrorCodeneeds anIDEMPOTENCY_KEY_CONFLICTcase. -
The
retryableflag of a failed conflict verification follows the cause. When an enqueue’s task id conflicts and the query that compares the stored payload fails,ENQUEUE_FAILEDcarries the driver classification of that error, as at every other driver site. Before,retryablewas alwaysTrue. -
A retry that changes the idempotency key of a sent task id returns
PAYLOAD_MISMATCH. Before, it returnedENQUEUE_FAILED, which the retry methods accept, and a retry of it could not succeed while the task row was live. -
Sync
close()inside a running event loop closes nothing. It returnsASYNC_CONTEXT, and the sync runner keeps its state. Before, it ran on the sync runner, returnedCLOSE_FAILEDafter 5 s when that loop had made broker calls, and then stopped the runner. -
app.check(live=True)inside a running event loop reportsHRS-217(CHECK_LIVE_ASYNC_CONTEXT) and does not connect. Before, it reportedBROKER_INVALID_URL(HRS-203), which named the wrong cause. -
PostgresBroker.run_sync()fails an assertion when it is called on a thread that runs an event loop. Nothing runs, and the sync runner does not start. Before, it blocked that loop until the call completed. -
The async send, schedule and result methods call the broker without an
except Exception. This matches their sync forms. The send and schedule methods also callapp.get_broker()without a catch. The only other call inside atryis thehorsies schedulercommand, a process boundary: it logs the error, closes the broker and exits with status 1. -
Statement errors do not show bound values. The broker’s engines are created with
hide_parameters=True. A failed statement shows[SQL parameters hidden due to hide_parameters=True]in place of its values inerr.message, instr(err.exception), in tracebacks, and inecho=Truelogs. Before, the error text held up to 300 characters of each value, which included task arguments and results. The values stay in the exception’sparamsattribute. -
The worker’s finalize error log does not print the task result. The three log lines for a finalize stage that fails print the result’s length as
result_json_charsin place ofresult_json_str. The retry still reads the full result from the error data. -
Validation failure text does not contain the refused value. Before, the text of a pydantic
ValidationError, with each input value, went into storedTaskErrormessages and data, the attempt error text, workflow node errors, log lines and returned errors. Now each failure is<loc>: <msg> [<type>], at most 10 per error, then(+N more). A message part that comes from the input is written as<input>; text that a validator writes is kept as written. For a task kwarg,<loc>starts with the kwarg name. A flattenedTaskError.exceptionuses the same text inmessage,reprand the traceback, for eachValidationErrorin the printed chain or in an exception group. A key from the input can still be part of<loc>: for a dict or mapping type, and for a key thatextra='forbid'refuses. -
The Syce Overview page states that Syce shows live tasks only. Syce code and SQL do not change.
Removed
Section titled “Removed”ErrorCode.WORKFLOW_SUBWORKFLOW_BUILD_WITH_BINDING(HRS-023). No code raised it. The number is retired and is not used again. The skill rule thatbuild_with()must accept*_argsand**_kwargsis removed; no such check exists.- Unused history codecs.
core/history/archive/rerun_input.py,encode_result_envelopeandencode_attempt_snapshothad no caller in the library. SQL functions write the stored bytes, anddecode_input_envelopereads a stored rerun input. - Unused archive helpers.
select_result_payload, its result types,decode_history_row_version, and the registry accessorsis_retainedandretained_versionshad no caller in the library.RETAINED_ARCHIVE_VERSIONSstays: the transcode decoder set reads it. - The docs page Operational Indexes. It indexed terminal rows of
horsies_tasks, and the live table no longer holds them.
- A timeout or a crash in one child no longer touches other tasks on the
worker. Before, one lost child broke the shared pool, and every started
task on that worker failed with
WORKER_CRASHED. Now only the task of that child is recovered, and only that slot starts a new child. (#375) - A child stops when its worker dies. Before, the child ran its task to the end, sent heartbeats, and wrote to the row with no parent to read the result. (#410)
- SIGTERM to every process no longer fails running tasks. Heroku and systemd
KillMode=control-groupsignal the whole tree. Children now ignore SIGTERM and SIGINT, and the worker drains them. (#411) - A timeout kill cannot reach the next task. The kill is bound to the task, so it does nothing once the result is read. (#413)
- A timeout kill reaches the subprocesses of the task. The kill goes to the
session of the child. A subprocess that calls
setsidleaves that session. (#414) - A forked helper cannot hide a dead child. The worker takes a child’s death from the process exit, not from the socket. (#415)
- The docs say what
SuccessPolicy.optionaldoes. The field records which nodes may fail, and it has no runtime effect: the engine stores it and reads only the required nodes of each case. A node that no case requires does not change the result, with or withoutoptional. Before, the docs said thatoptionalexcludes a node from failure accounting. - Phase-2 recovery progresses a failed node whose stored error does not
decode as a
TaskError, and it records the cause. The recovery recordsRESULT_DESERIALIZATION_ERRORwith the cause in the message. A stored result envelope that does not validate and stored text that is not JSON get the same error. The log names the refusal. Before, theValueValidationErrorof the decode rolled back the recovery of that row on each pass, and the node did not progress. Text that is not JSON recorded the error that the terminal status names. - A workflow keeps the cause when the stored result of its first failed task
does not parse. This applies to text that is not JSON, a result envelope
that does not validate, and an error that does not decode. The workflow
error is
RESULT_DESERIALIZATION_ERRORwith the cause in the message. Before, the cause was lost, and the workflow took a NULL error orWORKFLOW_SUCCESS_CASE_NOT_MET. - A lone surrogate in the text of a node failure no longer loses the
text. The text of an exception from
build_with()can hold one, for example a file name. The node now fails with the text, with the surrogate written as a\udXXXescape. Before, the node stored the fixed textsecondary serialization failure. A node whose completion result does not encode keeps its failure text in the same way. Before, that node stored the textnull. - The web UI no longer shows a Retry action that the server cannot run.
The server removed the manual task retry route, so the Retry button on a
task detail panel could only fail. The button, its confirm text, the
TASK_NOT_RETRYABLEandTASK_EXPIRY_PASSEDconflict codes in the UI types, and the--enable-actionshelp text for retry are removed. To run a terminal task again, use the rerun API. - A task cancel accepts every form of a task id that Python reads as a
UUID. A form such as
urn:uuid:<id>returnedDB_OPERATION_FAILED. The cancel now uses the canonical id text, and the result carries it. - A cancel of a task in history reads only the status and the workflow flag. A history row whose attempt snapshot does not decode no longer fails the cancel.
- The web UI banner in the
CUTOVER_REQUIREDschema state no longer reports a version mismatch. The stored and expected versions are equal in this state. The banner now names the one version, says that the offline task-history cutover is not complete, and tells the operator to run the cutover stages through tighten and validation. TheMISMATCHbanner is unchanged.GET /api/metahas a new fieldschema_state(aSchemaStatevalue); the UI reads it and no longer derives the state from the versions. - The agent skills in the wheel and the docs state the codes and signatures
that the code uses. A raised exception in a task gives
UNHANDLED_EXCEPTION;TASK_EXCEPTIONmeans the task returnedNone, or the worker caught an exception outside the task wrapper. Anauto_retry_forthat lists onlyTASK_EXCEPTIONdoes not retry a raised exception, so the examples now listUNHANDLED_EXCEPTION. The skills also listschedule_asyncandretry_schedule_async, state theRetryPolicyconstructor rules, type the broker URLs asSecretStr, putpaused_workflow_auto_cancel_afteronAppConfig.retention, and giveHandleResultas the return type ofWorkflowHandle.status()and.tasks(). - Each docs list of an enum’s members holds every member. The docs
site, the skills files and
llms.txtnow listOutcomeCode.TASK_EXPIRED,TASK_TIMEOUTandWORKFLOW_EXPIRED,ContractCode.NO_TYPE_AVAILABLE, all 21BrokerErrorCodemembers, andSchemaState.CUTOVER_REQUIREDwhere they were missing. The errors page adds HRS-105 and HRS-213 to HRS-215, and tables forWorkflowStartErrorCodeandHandleErrorCode.llms.txtadds HRS-016, HRS-017, HRS-031, HRS-032, HRS-104, HRS-105 and HRS-213 to HRS-216. HRS-104 means a pre-decorated task function (it has a__wrapped__chain), not a function decorated by another app. A unit test reads each list and fails when a member is missing or extra, or when a member list has no marker comment. TaskHandle.get_async()at the event loop limit returnsRetrievalCode.EVENT_LOOP_LIMIT, not cached. In 0.5.5 it returnedOperationalErrorCode.BROKER_ERROR, withbroker_codeEVENT_LOOP_LIMITindata.WorkflowHandle.get_async()already returnedRetrievalCode.EVENT_LOOP_LIMIT. The syncTaskHandle.get()runs on the sync runner loop, which never reaches the limit.close_async()does not block its event loop while it stops the sync runner. The stop joins the runner thread for up to 2 s; it now runs on a worker thread throughasyncio.to_thread().- The sync bridge debug log no longer holds call arguments.
LoopRunner.callloggedCalling <function> with args: ... and kwargs: ...atDEBUG. For a syncsend()orschedule()the arguments hold the task’s serialized kwargs, so task input went into the debug log. The line now names the function only, and it no longer formats every argument on each call whenDEBUGis off. - Every JSON parse in the library goes through
codec/json_io.py, andloads_jsonrefuses a number outside the finite binary64 range and a lone UTF-16 surrogate.- A literal such as
1e9999parsed toinf, becauseparse_constantsees only the tokensNaN,Infinityand-Infinity. An enqueue from raw JSON text then stored the tokenInfinityin the rerun-input envelope, which is not JSON, and a rerun copied it into a new row that the worker refused. Each reader now refuses the literal:enqueue_asyncreturnsINVALID_INPUTorINVALID_TASK_OPTIONSand writes nothing, the worker returnsWORKER_SERIALIZATION_ERROR, and cutover preparation counts the row as a decode failure. A rerun of a stored envelope with a non-finite number returnsRerunInputCorruptand writes nothing. A literal that underflows, such as1e-9999, still parses to0.0. - An escape such as
\ud800decoded to a string that UTF-8 cannot encode, so the failure came later: one such row aborted a cutover preparation batch, and a rerun of a stored envelope raisedUnicodeEncodeError. Each reader now refuses the value:enqueue_asyncreturnsINVALID_INPUTand writes nothing, the worker returnsWORKER_SERIALIZATION_ERROR, cutover preparation counts the row as a decode failure, and a rerun returnsRerunInputCorrupt.dumps_jsonalready refused such a string. A history read refuses it too, asinvalid_json. A surrogate pair such as\ud83d\ude00is one valid character and stays accepted. - A history read refuses a stored result or attempt snapshot that is not
strict JSON:
NaN,Infinity, a number outside the finite binary64 range, an integer over 4300 digits, an empty payload, or bytes that are not UTF-8. A result read returnsINVALID_JSON_PAYLOAD, as a read of a live task does, andget_task_detailreturnsHISTORY_CONTRACT_VIOLATION. Before, the archive reader acceptedNaNandInfinity, and an oversized integer raised a rawValueError.CorruptArchiveValue.detailfor these failures is nowinvalid_json, not an exception class name.plan_rows_from_explainraisesHistoryContractErrorfor the same text.
- A literal such as
decode_input_envelopereturnsInputEnvelopeCorruptfor an integer over the parse limit of 4300 digits. Before, it raisedValueError.prepare_legacy_batchno longer aborts on a row that breaks an enqueue rule. An empty task or queue name, or a priority outside 1..100, raised a rawValueErrorout of the batch, so one row stopped every row around it and the error did not name it. The row now stays unprepared, andPreparationBatch.refused_rowsnames it with the rule. The walk ends in the newPreparationBlocked, which counts the rows that remain, not inPreparationComplete. Repair the rows and run preparation again; the tighten refuses while any remain.
0.5.5 — 2026-10-28
Section titled “0.5.5 — 2026-10-28”Upgrade notes
Section titled “Upgrade notes”- No schema change. The schema stays at v51.
- Code that reads
PostgresBroker.session_factory,.async_engineor.listenermust change. Callawait broker.select_resources()on the event loop that uses the resources, and match itsOkorErr(EventLoopLimit). - An exhaustive
matchonBrokerErrorCode,TaskSendErrorCode,WorkflowStartErrorCode,HandleErrorCode,MonitoringQueryErrorCode,TaskActionErrorCodeorRetrievalCodeneeds anEVENT_LOOP_LIMITcase.
Changed
Section titled “Changed”- Sync workflow calls stop when the broker’s sync runner stops. After
sync
close(), orclose_async()after any sync call,spec.start()returnsINTERNAL_FAILED, the syncWorkflowHandlewrap methods andpause_workflow_sync/resume_workflow_syncreturnLOOP_RUNNER_FAILED, andWorkflowHandle.get()/result_for()return aBROKER_ERRORresult. Sync task calls already behaved this way. Before, sync workflow calls continued on a process-wide event loop. - A broker keeps one resource set per open event loop. A set is a
runtime engine with its pool, a session engine when
session_database_urldiffers, and a listener. Sync calls use the set of the broker’s own sync runner loop. A set is built on the first call from its loop, the set of a loop that closed is released at the next broker call, andclose_async()/close()close every set. A process that makes broker calls from two loops can open twice the connections of one loop; Broker Config → Event Loops gives the ceiling. EVENT_LOOP_LIMITis a new member ofBrokerErrorCode,TaskSendErrorCode,WorkflowStartErrorCode,HandleErrorCode,MonitoringQueryErrorCode,TaskActionErrorCodeandRetrievalCode. A broker serves at most 5 open event loops besides its sync runner loop. A call from one more open loop returns this code and runs nothing; repeating it on that loop returns the same code until one of those loops closes.send()keeps the code instead of folding it intoENQUEUE_FAILED,get()does not cache it, and the web task actions return HTTP 503.PostgresBroker.session_factory,.async_engineand.listenerare removed.await broker.select_resources()returns the running loop’sBrokerResources(engine, session factory, session engine, partition maintenance database, andListenerEnabledorListenerDisabled), orErr(EventLoopLimit).- A
PostgresListenercall from a loop other than its owner loop fails an assertion instead of raisingRuntimeError.listen(),listen_payload()andlisten_many()returnErr(LISTENER_SUBSCRIBE_FAILED)when a concurrent reset tears down the dispatcher connection during the call; before, they raisedRuntimeError.
- Sync workflow calls and sync task calls share one sync runner.
spec.start(), the syncWorkflowHandlemethods andpause_workflow_sync/resume_workflow_syncnow run on the broker’s own runner, the same one thatsend(),TaskHandle.get()and the app’s sync pings use. Before, they ran on a second, process-wide event loop that shared the broker’s connection pool and listener. A sync call that waited for a pool connection returned by a call on the other loop waited untilpool_timeout(30 s by default); after that, every pool wait from the other loop failed at once with “bound to a different event loop” (ENQUEUE_FAILEDfromsend(),INTERNAL_FAILEDfrom a handle). The listener bound to the first loop that used it: after a sync workflowget(),app.ping_workers()returnedWORKER_PING_FAILEDfor the rest of the process, and result waits from the other loop fell back to polling. - Sync and async calls in one process no longer share a pool or a
listener. Before, an async call that held the only free pool connection
made a sync call wait until
pool_timeout(30 s by default); after that, pool waits from the other loop failed at once with “bound to a different event loop”. The listener bound to the first loop that used it: result waits from the other loop fell back to polling, andping_workers()from the other loop returnedWORKER_PING_FAILED. - A broker used from successive event loops, such as repeated
asyncio.run()calls or function-scoped pytest loops, gets a new pool and listener on each loop. Before, after one pool wait on an earlier loop, every pool wait from a later loop failed with “bound to a different event loop”. PostgresListener.close()from another loop after the owner loop closed closes both listener connections. Before, it logged a warning and left them open.
0.5.4 — 2026-10-20
Section titled “0.5.4 — 2026-10-20”Upgrade notes
Section titled “Upgrade notes”- The schema advances from v36 to v51 when the first 0.5.4 process starts. Mixed 0.5.3 and 0.5.4 fleets are not tested: stop every horsies process (workers, schedulers, web), then start 0.5.4.
- Schema v37 refuses unknown workflow node statuses. Before it adds a
CHECK for the seven
WorkflowTaskStatusvalues, activation reads every node. A node with another status (for exampleCANCELLED, which the phase-two function accepted before) stops the upgrade withWorkflowNodeStatusMigrationErrorand changes nothing. Correct those rows, then start again. - Index removals (
DROP INDEX CONCURRENTLY) and index builds (CREATE INDEX CONCURRENTLY) do not block claims, enqueues or finalizes.
Changed
Section titled “Changed”- A FAILED workflow stores no result.
MARK_WORKFLOW_FAILED_SQLwritesresult = NULL, and the final result is built only for a COMPLETED workflow. For a FAILED sub-workflow,SubWorkflowSummary.outputisNone. The Rust runtime storesNULLin the same case. - Enqueue validates the whole
retry_policy.enqueue_async, workflow node promotion and the workflow fast-path root validate the policy asRetryPolicyin strict mode (no type coercion, no unknown fields, the model’s strategy rules). A malformed policy is refused withINVALID_TASK_OPTIONSbefore any write; the message names each failing field. Refused now, accepted before: a partial policy such as{"max_retries": 3}, which stored 3 retries but never retried becauseauto_retry_forwas missing;max_retriesoutside 1..20 inside a policy (no policy still means 0 retries); and a policy that is not an object. Options built fromRetryPolicy(every.send(), workflow and scheduler path) always pass. PostgresConfig.tcp_user_timeout_ms, default 30 000. Sets libpqtcp_user_timeouton the broker pools, the child-process pools, theLISTENconnections and the monitoring listener. A write to a connection that died without a reset now fails after 30 s instead of the kernel retransmission limit (about 924 s on Linux withtcp_retries2=15); the pool then opens a new connection. A long query is not affected. With keepalives on, an idle dead connection is closed at about 40 s instead of 60 s.0uses the operating system default. Applies on Linux only.- Broker connections run with
jit = offwhere the session keeps the setting: the runtime pool (not in PgBouncer transaction mode), the session pool, and the single-connection engine. On a fresh database the partition coverage probe costs 123,447, above the defaultjit_above_cost(100,000). - The monitoring status cards are cached for 10 seconds per scope.
/api/tasks/statscounts can be up to 10 seconds old. - Worker health reports
completion_candidate_yield: completion candidates per workflow scanned by the global recovery audit. foreverhistory uses one open-ended leaf. Partition coverage stops creating one dailyforeverleaf per day. Closed, empty dailyforeverleaves are detached and dropped by the prune pass, at most 32 per pass.
- Partition maintenance under a session time zone other than UTC.
Catalog anchors are decoded to UTC. Before, under
Europe/Berlin, the leaves whose bound is in the repeated hour of 2026-10-25 failed withMETADATA_MISMATCHon every pass; the heartbeat andforeversweeps failed on the fold day; and relocation west of UTC named leaves for the wrong day. Leaf bounds and names now refuse a value that is not in UTC. - A pause that races a resume no longer strands a node. The abandon
functions lock the workflow row and apply only while the workflow is still
PAUSED. Before, a resume that committed between the status read and the abandon left the nodeREADYwith no task while the workflow wasRUNNING, until workflow recovery ran (30 s by default). - The child pre-start stop writes the node only when its transition
applied. For a PAUSED workflow, the node is reset to
READYonly when the abandon applied; for a CANCELLED workflow, the node is markedSKIPPEDonly when the cancel applied, and after it. Before, a lost claim or a row that ended another way still had its node rewritten. - Worker start no longer fails when its children start at different
times. Each warmup call waits at a barrier with one party per child, so
one round of calls reaches every child. Before, a child that finished its
initializer more than about 0.6 s after the others failed the start with
worker child warmup started 1/2 process(es), and the start path does not retry that error. A child later than 20 s now fails with aWarmupIncompleteErrorthat names the barrier. - Schema lock waits do not hold a snapshot. A broker that waits for the
schema lock polls
pg_try_advisory_lockevery 25 ms, so it cannot block the lock holder’sCREATE INDEX CONCURRENTLY. - The finalizing marker is fenced to the claim generation, so a stale attempt cannot stamp a re-claimed row.
- A retried task keeps its node’s first
started_at. The node handoff writes the node only when it is not alreadyRUNNING. - The scheduler runs with a one-connection runtime pool. Its lock transactions use a private single-connection engine.
- The claim loop wakes when a finalizer ends, so a requeue to
PENDINGor an expiry before start does not wait for the next poll. - Workflow node status is constrained to the seven
WorkflowTaskStatusvalues, and the phase-two function no longer acceptsCANCELLED. - Pending-expiry outcomes come back in deadline order.
- A transcode target must be a version this build can decode.
- History writers that take outside values check the history column
rules first. Cutover relocation refuses a legacy row that breaks a rule
with
RelocationRuleViolation; transcode refuses a target version below 1, a codec outside 1..64 bytes, and a forward rerun-input frame that would not fit. - Rendered staged readers carry the manifest stamp they were rendered from. Publication health and the coverage pass compare it, so a reader rendered by an older release is republished.
Performance
Section titled “Performance”- Claim. Each eligibility arm and the merge are limited to the smallest of the claim budget, the queue’s remaining cap and the batch limit, and the claimed rows come back in order. The candidate selection runs as a generic-plan helper. The paused or cancelled workflow lookup reads each workflow by primary key.
- Recovery. A partial index on nonterminal workflow nodes serves the recovery probes: a workflow with 1,001 terminal nodes no longer scans its terminal entries. The global audit probes for any node only when no nonterminal node exists (a page of 200 workflows examines 200 node rows instead of 400).
- Phase two plans its history lookup once per connection.
- Result reads.
get_resultandTaskHandle.getread a task result through one staged reader statement (horsies_task_result_staged): a terminal live task takes 1 statement instead of 2, a task in history 1 instead of 3. - Workflow finalize reads its inputs once and the parent link with the
status. The pause paths and finalize leave the
workflow_doneNOTIFY to the status trigger. - History moves encode each attempt snapshot once and hash a result once. A batch move hashes its result once per batch instead of once per row. Cutover relocation encodes each snapshot once.
- The duplicate-identity guard probes only the leaves that can hold the id (by UUIDv7 birth time); ids from a producer clock ahead of the terminal time are recorded in an exception table.
- Reader publication. The manifest is written in one statement; a coverage pass publishes once instead of once per created leaf (2 new classes × 61 leaves: 241 publications and 53.7 s became 1 publication and 6.1 s); a detach republishes only when the readers changed. A partial index serves the leaf catalog’s active records.
- Removed objects with no reader: 2 task retention indexes, 3 redundant workflow-node indexes, the task update notify trigger on cut-over databases, the 24 single-column CHECKs of the history table (the writers hold those rules; the 8 cross-column CHECKs stay), and the task retention statistics.
0.5.3 — 2026-08-26
Section titled “0.5.3 — 2026-08-26”-
Global recovery scans now have bounded work. Durable
(created_at, id)cursors give each scan cycle a fixed upper bound. A global pass audits at most 200RUNNINGworkflows. An orphan-task pass audits at most 500 live workflow tasks. Page claims prevent duplicate global scans, while workflow-tree recovery stays uncapped inside the requested tree. Schema v36 adds exact partial recovery indexes and serializes their concurrent repair.RecoveryConfig.orphan_task_audit_interval_mssets the orphan-task audit interval and defaults to 60 seconds. Worker health reports scan work, duration, refusals, and errors. -
Partition maintenance now requires the session-capable database path. It uses the same direct engine as schema DDL and
LISTEN/NOTIFY. Leaf advisory locks are nonblocking. Relation lock waits return a typed busy result after a bounded wait. Cancellation closes a connection when lock ownership is not certain. Each changed leaf commits before the next leaf or retry starts. A healthy complete coverage check uses one set-based catalog query. Ten declared retention classes with the default horizons stay within three SQL statements on this path. -
Partition coverage now verifies complete physical conformance. The physical partition bound must match both the catalog and the requested interval. Task lookup and enqueue-order indexes must have the required validity, columns, collation, operator class, and sort order. Index repair locks and rechecks the exact index relation before removal. Loader publication uses a final transaction and compares all manifest metadata. Partition health uses the same bound and index conformance rules.
Changed
Section titled “Changed”- Fresh unkeyed enqueues now use one constant SQL statement and one
AUTOCOMMITround trip. The keyed reservation transaction and conflict checks are unchanged. Task options are parsed once per enqueue. Withsend_async, 64-way concurrency, a pool size of 10, and local PostgreSQL 18, throughput increased from 880 to 2,105 sends per second and CPU time fell from 1.08 ms to 0.43 ms per send.
0.5.2 — 2026-08-12
Section titled “0.5.2 — 2026-08-12”- Per-queue retention:
AppConfig.retention.queue_retention. Maps a queue name to how long a task sent on it keeps its history record, or toNoneto keep it forever. A queue with no entry is unchanged and takes the immutable 30-day class. The duration is part of the class the mapping derives (emailsat seven days isq_emails_7d), so remapping mints a new class rather than redefining one and records enqueued under the previous mapping age out under the promise they were given.q_is reserved: aretention_classeskey may not use the prefix. The class is resolved at the send and snapshotted on the row, so a retry replays the class its original send chose. Precedence is the explicitretention_class_key(including an explicitstandard_30d), then the queue’s mapping, thenstandard_30d— omitting the argument now means “the queue’s mapping if there is one”, which is the behaviour change for anyone who maps a queue. The direct successor to 0.4.x’sRecoveryConfig.queue_terminal_record_retention_hours.
-
A monitoring filter change no longer evicts the panels it scopes. Every aggregate is keyed by the filters it is scoped to, so changing one started a query with nothing cached under the new key and the panel rendered nothing until it landed — taking its row off the page and putting it back, shifting the page below it twice per filter change.
useTaskStats,useFacetsanduseWorkflowRunsnow keep the previous scope’s data across a key change, asuseTasksanduseBreakdownalready did. The status strip also holds its row before any counts exist, sincetask_statsalways returns all seven lifecycle statuses including zeros. -
Schema v35 is an offline upgrade for existing 0.5.0 and 0.5.1 databases. Stop producers, workers, schedulers, and monitoring/web before the first v35 migration starts, and keep them stopped through validation; there is no rolling-upgrade path. The
foreverconversion uses non-concurrent partition DDL and rebuilds the legacy leaf’s task-ID and enqueue-order indexes in the migration transaction. Its window therefore depends on blocking transactions and the complete legacyforeverpopulation, even though only current-day rows move. A 0.5.1 worker does not create the dailyforevercoverage required by v35 and must not remain connected after conversion. Do not rerun the 0.5.0 cutover: after migration, runvalidate_cutoverin a committing transaction and restart only after it writestask_history_v1_validated_v1. Fresh databases require no offline conversion. -
Schema v35 separates migration completion from offline-cutover completion. The integer schema watermark no longer authorizes a fleet by itself. Post-cutover validation writes the versioned
task_history_v1_validated_v1attestation only after every structural check passes; tighten no longer writes it, and the earliertask_history_v1marker is not accepted. Worker startup and monitoring actions require both the exact supported schema version and the validated attestation. A database at a newer schema version is refused. -
Bounded history windows can prune the
foreverclass. The class is a RANGE-subpartitioned parent. Existing pre-v35 rows before the current UTC day remain in one bounded legacy leaf; only current-day rows move during conversion, and subsequent rows use daily leaves. Offline relocation routes older classless rows into that existing legacy leaf instead of attempting an overlapping daily partition. This removes the per-request scan of the complete forever population without another terminalization-path index. -
UUIDv7 birth time is a lookup hint, not an absence proof. Staged history readers probe birth-compatible leaves first, then every skipped leaf before returning absence. A caller-supplied task UUID whose embedded clock differs from the enqueue clock remains readable.
-
A task-ID conflict cannot leave a live idempotency reservation without an owning task or an unbound successful key. After exact-ID payload verification, a keyed retry atomically binds the existing live task to the claimed key. A later fresh-ID retry replays that task, and a task already bound to a different key is refused.
-
WorkflowStatus.EXPIREDis terminal across result waits, notifications, and subworkflow propagation. Paused-workflow expiry stores a structuredWORKFLOW_EXPIREDerror. Handles also decode the plain-text expiry value written by 0.5.0 and 0.5.1. -
Monitoring pagination and aggregates no longer return plausible but false results. Requests with
offset + limit > 500are refused explicitly instead of querying only 500 rows and returning an empty later page. Nullable descending history sorts useNULLS LAST, matching the live-side merge. Facet counts are summed across live and history before global ranking and capping, selected error families are filtered before their code cap, and error-category totals use uncapped category counts. -
History task detail verifies result envelopes before decoding them. The task-info path now enforces the stored result digest and envelope metadata, matching the raw-result path.
-
Partition detach and finalize always carry a statement timeout. The heartbeat sweep’s detach and the interrupted-detach finalize both took an unbounded wait:
statement_timeout_msdefaulted toNone, so a call site that omitted it waited on a conflicting lock with no bound while holding the cluster-wide maintenance gate. A blockedDETACH PARTITION ... FINALIZEis the worse of the two — PostgreSQL refuses every further detach on a parent while one is pending finalization, so the stall propagates to that parent’s whole retention sweep. The finalize command gained the field, the executor applies and restores it, and the default is gone from both commands:Nonestill means unbounded, but only where a call site writes it. -
A leaf the catalog names and the database no longer has stops failing every finalize. The staged history readers are generated from the leaf catalog, and PostgreSQL does not validate PL/pgSQL bodies at
CREATE, so a reader naming a relation dropped out of band built successfully and failed at execution — inside the provenance probe on the terminalization path, which made one missing leaf fail every finalize in every queue and every retention class. The probe list is now filtered by relation existence when the manifest is built, so republication is self-correcting, and the maintenance pass republishes when the published readers name a relation that no longer resolves rather than only when a leaf was created. The vanished leaf is named on the coverage health surface and logged at warning level. The catalog row is left untouched: a leaf that disappeared behind the manager’s back is an operator’s decision, and stamping it dropped would erase the evidence. The history that leaf held is unreadable. A leaf inside the coverage horizon cannot be recreated while its catalog row survives, so that class gains no new partitions until an operator resolves the conflict; other classes are unaffected. -
A destroyed leaf’s tasks are no longer reported as too old to keep.
predates_retained_floor, which reaches the rerun surface, is derived from the attached leaf catalog rather than from the published probe list. The two differ by exactly the leaves whose relation was dropped out of band, and those leaves did retain history until someone removed it, so answering from the probe list would present an accidental drop as ordinary ageing. -
Retention configuration errors carry their own error code. Every refusal from
AppConfig.retention— an unusable queue name or class key, a non-positive or fractional duration, a reserved or duplicated class key, a key too long for the relation names it is spliced into, and aqueue_retentionmapping naming a queue the deployment does not have — now reportsCONFIG_INVALID_RETENTION(HRS-216). These answered withCONFIG_INVALID_RECOVERY(HRS-204), left over from when the fields lived onRecoveryConfig, and the queue cross-check answered withCONFIG_INVALID_QUEUE_MODE(HRS-200). Setting a moved field onRecoveryConfigstill reportsCONFIG_INVALID_RECOVERY: that refusal belongs to the object that no longer has the field. -
The e2e suite honours a database URL override. Its modules built the local fallback URL as the
defaultargument ofos.environ.get, so the fallback was evaluated before the override was chosen and a missingDB_PASSWORDraisedKeyErrorat import even when the override was set and was the value that would have been used. Resolution is lazy and shared, and the failure names both ways out. -
A worker ready-check that raises is reported, not swallowed. The e2e helper retried a raising probe and discarded the exception, so a probe that could never succeed was indistinguishable from a slow worker and surfaced as a startup timeout. The last failure is now carried into the timeout message.
-
Per-queue retention now applies on every enqueue path.
.schedule(),.schedule_async(), aTaskSchedulefiring on its cron, and a workflow node’s backing task resolve the queue mapping exactly as.send()does, with the class chosen at enqueue. Previously only the immediate sends resolved it, so the same task landed a different retention class depending on how it was enqueued, and nothing reported the divergence. A cron fire resolves from the scheduler process’s own configuration; a workflow node from the queue its node runs on. -
An explicit retention choice on a delayed send is honored.
with_options(retention_class_key=None).schedule(...)asked for the record to be kept forever and the record was deleted at 30 days; a named class was discarded the same way. The delayed methods never carried the option.idempotency_keyrode the same gap and is carried with it. -
Worker startup registers queue-derived classes. Startup registered declared classes only, so a task sent on a mapped queue before any maintenance pass had no partition to move into at terminalization. The window is not one tick: the periodic pass sits behind the cluster-wide maintenance gate, which can deny.
-
A failing retention class no longer stops heartbeat coverage or the classes after it. Containment previously covered raised exceptions in history-class leaf coverage and nothing else. A returned refusal still ended the pass, and every queue-derived key sorts before
standard_30d, so the feature made the adverse ordering systematic rather than incidental. Heartbeat coverage ran after the failure report and so was skipped entirely — and heartbeat leaves gate worker startup, so one poisoned class row could leave a restarted fleet unable to come back. Each class now runs inside a savepoint, so a database-level error rolls back to that savepoint instead of poisoning the caller’s transaction and taking the pass down with it. -
A queue named in
queue_retentionmust be a queue the deployment has. A misspelled queue was accepted, minted a class, and grew leaves and indexes on every pass forever while nothing routed into it — and the queue the adopter meant kept the 30-day default. Checked in both queue modes, refused at construction naming the configured queues. -
RetentionChoiceis exported fromhorsies. It appears in thewith_optionssignature, so it was a type a reader could see and not import. -
A retention class key longer than 31 characters could stop partition coverage across a deployment. Configuration accepted any key up to PostgreSQL’s 63-byte identifier limit, but the key is interpolated into the relations the class owns and the longest of those adds 45 bytes, so the class registered and every later maintenance pass then raised on its leaf name. The pass reads registered classes from the database rather than from configuration, so removing the declaration did not recover it. Keys are now bounded at 18 characters — the point past which a derived name exceeds the limit — and refused at configuration with the arithmetic. Two shorter bands also break and are inside the bound: from 19 the index name is silently truncated, and at 30 and 31 both of a leaf’s index names truncate to identical bytes and the second
CREATE INDEXfails on a duplicate. -
A failing retention class no longer denies coverage to the others. Classes are served in key order and a failure inside one escaped the loop, so every class sorting after it stopped receiving partitions. Each class is now bounded on its own: the failure is recorded against its key, the remaining classes are served, and the pass reports a refusal naming every class that failed. This also restores the health signal — the refusal is a value the caller matches on, where a raised failure skipped that match and left coverage health reporting whatever the previous pass had put there.
Changed
Section titled “Changed”-
Retention configuration moved to
AppConfig.retention. Nine fields leaveRecoveryConfigfor a newRetentionConfig:retention_classes,terminal_record_retention_hours,worker_state_retention_hours,retention_sweep_interval_s,retention_delete_batch_size,history_leaf_horizon_days,heartbeat_leaf_horizon_hours,partition_maintenance_interval_s, andpaused_workflow_auto_cancel_after.The two answer different questions. Recovery is about work that went wrong — stale claims, dead runners, the thresholds for noticing. Retention is about work that went right and is now history: how long its record survives, how partitions are kept ahead of writes, and how they are reclaimed.
paused_workflow_auto_cancel_aftermoves with them because expiring a workflow that has sat paused past a declared age is a policy about how long a record stays actionable, not a response to a crash.Setting a moved field on
RecoveryConfigfails at construction and names its new home; every misplaced field in a config is reported at once. There is no alias and no deprecation shim, per the pre-1.0 posture.RetentionConfigandRetentionClassConfigexport fromhorsiesdirectly. No schema change, and the worker-state monitoring payload keeps its existing keys.
0.5.1 — 2026-08-10
Section titled “0.5.1 — 2026-08-10”-
Declared retention classes. A deployment may declare additional finite retention classes in
RecoveryConfig.retention_classes, each a key and a duration. The maintenance owner registers every declared class at startup and on each maintenance pass, exactly as it registers the classes the library ships, so partition DDL stays out of adopter hands and registration keeps a single owner. Tasks are then sent into a declared class withwith_options(retention_class_key=...).Declarations are validated where they are written: a key must be a usable identifier, must not be one the library owns (
standard_30d,forever,heartbeats), must not repeat, and its duration must be positive. Every problem in a config is reported together rather than one per run. Re-declaring an existing key with a different duration is refused by the registration machinery and named at startup — classes are immutable.durationis a minimum, not an exact age: history leaves span one day and a leaf is dropped only once its whole day is past the duration, so a row survives betweendurationandduration + 1 day. Sub-day durations are accepted and never under-retain, but cannot expire faster than daily partition granularity allows.
Changed
Section titled “Changed”- Send-time retention validation widens from the two shipped classes to the set this deployment declares. The check stays process-static and costs no database round trip, which makes the contract per-process: a process whose config omits a class refuses it even if another deployment registered that class in the same database. The rerun path’s registry lookup is unchanged.
0.5.0 — 2026-08-10
Section titled “0.5.0 — 2026-08-10”Task-history live/history split. Schema v34. This release requires an offline cutover for existing deployments: stop all units, upgrade the package, run migrations, run the cutover program, start units. Fresh installs are born at the cutover’s end state; the history subsystem is dormant until invoked and adds no cost to deployments before cutover.
Changed
Section titled “Changed”- Workflow retention removes unconsumed phase-2 evidence with the workflow. A terminal task hands its workflow node’s progression to an outbox, and deleting the workflow deletes any evidence still waiting there — the same disposition the consumer would have reached for a workflow that is already terminal. Retention is not held up by a stalled consumer.
- Enqueue-time facts are required columns on a fresh install. The retention class, command fingerprint, rerun-input flag and prepared disposition are added nullable so that an upgraded deployment’s running fleet cannot violate them before the cutover backfills; the cutover’s tighten stage makes them required. A database created by this release has no such fleet and never runs that stage, so it applies the same tightening at creation: the columns are required and their declared checks present from the first row. Upgraded deployments are unchanged — the columns tighten at the cutover, where they always did. The cutover now refuses to tighten while any live row carries no retention class, naming the count.
- Terminal task records move to a partitioned history archive at
terminalization. The live table holds only live rows (
PENDING,CLAIMED,RUNNING); completed, failed, cancelled, and expired tasks live inhorsies_task_history, partitioned by retention class and day. Task and workflow identity columns areuuid; task ids are UUIDv7. Retention works by dropping whole partitions instead of deleting rows. The default retention class keeps terminal records 30 days; passingretention_class_key=Noneat enqueue keeps a record forever. - Crashed-worker workflow recovery flows through the terminalization outbox. A worker that dies between recording a task’s terminal state and advancing its workflow node used to be found by scanning the live table for terminal tasks; terminalization now records the owed progression as it moves the task, and the reaper consumes those records. Same outcome, different mechanism, and the same grace window and per-pass bound as before.
- Manual retry of a terminal task is removed. A terminal record is immutable; re-execution is a new request through the rerun API, with a new task id and recorded lineage. The dashboard’s retry action is replaced by rerun.
- Tasks cancelled by a workflow-level cancellation carry no error
summary of their own. The terminal record’s
error_codeandfailed_reasondescribe the terminalization itself; requeue residue from earlier attempts stays in the attempt history instead of being frozen beside a status it does not describe. - Attempt rows purge with their task’s history record. At terminalization the attempts are archived into the history record’s snapshot and the live attempt rows are deleted; the snapshot is the attempt history’s only home from then on, and it ages with the record’s retention class.
- The dashboard’s task totals are live rows plus a bounded terminal
window. Monitoring reads default to the last 24 hours of terminal
history (
since/untilaccepted, 30-day maximum); a request over the maximum is refused with the bound named, never silently clamped. - Monitoring history reads ride a per-leaf enqueue-order index.
Every task-history leaf carries a btree on
enqueued_at(schema v34); the dashboard list’s default sort merge-appends leaves in index order and stops at its LIMIT instead of sorting every matched row per call. The migration builds the index on each existing leaf at upgrade. The unfiltered total’s history side becomes the planner estimate the surface already documents; totals under any active filter stay exact. - Pausing a workflow relocates its claimed tasks’ terminal
records. Pause abandons a claimed internal task row, and the
abandoned row terminalizes to history as
CANCELLEDwith a message naming the pause; resume enqueues fresh rows. A node still pending moves nothing, and the paused workflow itself is unchanged — only its abandoned claimed tasks gain history records.
WorkflowStatus.EXPIREDandpaused_workflow_auto_cancel_after. A workflow paused longer than the configured age is expired by policy:CANCELLEDcontinues to mean someone decided,EXPIREDmeans time ran out. The policy is opt-in (defaultNone); the workflow’serrorcolumn records the policy name and configured age. An expired child propagates to its parent exactly as a cancelled one. Exhaustive matches overWorkflowStatusgain a member.- Rerun and idempotency API: re-execute a terminal task by reference with recorded lineage; scoped idempotency keys with a reservation window.
- Workers own partition coverage and pruning: heartbeat and history
partitions are created ahead of writes at worker startup and by a
periodic maintenance pass, and the same pass detaches and drops
leaves whose retention class duration has elapsed — a refusal (a
partition pinned by recovery evidence, a reader outliving the 5 s
detach timeout) skips that leaf, surfaces with its reason in the
partition_pruninghealth payload, and retries next pass. The worker role needsCREATEon the partition parents; a deployment that withholds it must run an external coverage cron.run_schema_migrationscontinues to govern versioned migrations only. retention_class_keyat enqueue.with_optionsand both send paths accept it: omitted lands the immutable 30-day default class, explicitNoneis the only route to a forever record, and an unknown class name is refused at the send statement with nothing written.- Unresolvable crashed-worker recovery evidence quarantines after a
bounded attempt count. A phase-2 pending row whose disposition
keeps refusing to resolve is retried for
phase2_quarantine_after_attemptsrecovery passes (default 25, bounds 3–1,000), then moved to a quarantine table with its evidence preserved; discovery stops retrying it. Quarantined counts, rows over the attempt bound, and the quarantine function’s refusals join the worker health surface beside the phase-2 pass summary. rerun_taskis importable fromhorsiesdirectly, with its command, policy, exhaustive outcome union, andNotEligibleReason.
Removed
Section titled “Removed”queue_terminal_record_retention_hoursandheartbeat_retention_hours: terminal task records age by retention class and heartbeats drop with their partitions. Setting either now fails validation naming the successor.terminal_record_retention_hoursnarrows to workflow records only.
Known incompatibilities
Section titled “Known incompatibilities”syceis not compatible with this release; compatibility returns in a post-0.5.0 release.
0.4.7 — 2026-08-05
Section titled “0.4.7 — 2026-08-05”Terminal failure-summary cleanup. Schema v26.
Every worker calls the shared terminalization functions, so the corrected summaries take effect fleet-wide as soon as the first upgraded process migrates the schema to v26. They do not roll out worker by worker.
- Terminal writers own the complete final-attempt summary. Automatic retry
clears
error_codewhen it requeues a task but neverfailed_reason, so a task that failed, was retried, and then reached a different terminal outcome carried the earlier attempt’s reason beside that outcome: a completed task could show a stale failure reason, and an expired task kept one besideTASK_EXPIRED. Locked failure now assigns its reason unconditionally — a NULL reason clears the column instead of preserving the old value — and the completion and expiry operations clearfailed_reasonat the transition. Per-attempt reasons remain inhorsies_task_attempts.
Documentation
Section titled “Documentation”- Corrected the retention description for workflow-backing task rows: they use the global window and are protected while their workflow is non-terminal; once the workflow is terminal, each row ages from its own terminal timestamp. The previous wording claimed a workflow and its task rows are retained as a unit.
0.4.6 — 2026-08-05
Section titled “0.4.6 — 2026-08-05”Task terminalization consolidation. Schema v25.
Rolling-upgrade prerequisite: complete the 0.4.5 rollout on every worker in a deployment before installing 0.4.6. This release adds a constraint that requires terminal tasks to carry a terminal timestamp. The first upgraded process enforces it on new writes as soon as it migrates the shared schema; workers older than 0.4.5 do not set that timestamp, so their terminal transitions would fail. This ordering is per deployment, not a calendar delay. A full-stop deployment may upgrade directly from an earlier version because no older worker remains active when the migration begins.
horsies_tasks.terminalization_kindrecords which operation ended each task. Replays are recognized only when the committed operation is equivalent to the requested transition; a terminal row produced by another operation is reported as a conflict rather than inferred from its status alone.
Changed
Section titled “Changed”- Terminal task transitions are owned by fifteen database functions behind one typed boundary. Callers receive explicit applied, replay, lost-claim, source-conflict, or absent outcomes instead of inferring them from row counts. Refusal diagnostics carry the locked values used to evaluate claim, staleness, deadline, and workflow-link guards.
- The schema program advances from v17 to v25 in one startup migration,
including a one-time backfill of
terminal_atfor existing terminal rows before the terminal-status/timestamp constraint is validated. Installations may upgrade directly from v17 through every intermediate schema step. - The measured completion, failure, workflow-success, and expiry paths remained
within their declared statement-count, transaction, latency, and WAL budgets;
the evidence and conditions are committed under
tests/perf/results. The orphan, workflow-scoped, child-stop, and administrative-cancellation paths ship under correctness-only verification; their measurements remain available through the manual performance workflow.
- Sub-second task-staleness thresholds are honored at millisecond precision instead of being truncated to whole seconds.
0.4.5 — 2026-08-03
Section titled “0.4.5 — 2026-08-03”Task lifecycle release. Schema v17 (additive).
0.4.6 adds a CHECK constraint tying terminal_at to terminal status.
Complete your 0.4.5 rollout on every worker before deploying 0.4.6. The
constraint rejects terminal writes that omit terminal_at, which is what a
pre-0.4.5 worker produces; a deployment still running mixed versions would
fail those workers’ finalize statements. The requirement is per deployment,
not a release gate — 0.4.6 ships when ready, and each deployment applies it
once its own 0.4.5 rollout is complete. A full-stop upgrade (all workers
stopped) may move to 0.4.6 from any version directly; the rollout requirement
applies to rolling restarts, where pre-0.4.5 workers would keep writing
terminal rows during the migration.
Changed
Section titled “Changed”- Workflow pause and cancellation no longer terminalize a task whose claim
has been replaced. A task whose lease lapsed, was requeued by the reaper,
and was re-claimed carries a new claim generation; the abandon or cancel
now applies only to the generation it was issued against. Batch statements
fence per task, so one batch spanning several claim transactions rejects
only the entries whose generation moved. The child pre-start branches fence
to their own claim. A requeued
PENDINGrow under a cancelled workflow carries no claim to fence and is still cancelled.
horsies_tasks.terminal_at TIMESTAMPTZ(schema v17): the canonical instant a task became terminal, assigned from database time in the same transaction as the terminal transition and cleared by manual in-place retry. Every terminal transition from this release forward records it. Eight of the terminal paths previously recorded no end timestamp at all, leaving terminal age to fall through toupdated_at. Nothing reads the column yet;completed_atandfailed_atare unchanged.
0.4.4 — 2026-08-02
Section titled “0.4.4 — 2026-08-02”Workflow retention performance release. No schema change. At 40k workflows (35k eligible, 1-2 nodes each) / 460k tasks, a full backlog drain drops from 12.0s across 177 statements to 0.5s across 107 (per 500-node batch: ~49k buffer reads → ~4.2k, 9ms).
Changed
Section titled “Changed”- Workflow retention batches by workflow, not by node row, in a single
statement. Candidates come from the workflows retention index; each is
guarded once per statement against a materialized set of workflows with
non-terminal backing tasks, computed from the non-terminal side (served
by the tasks status index) instead of a per-candidate
NOT terminalheap walk. Node rows purge set-wise in the same statement; a workflow with more nodes than the whole batch budget drains alone instead of starving. Deletion semantics unchanged. - Finalize logs scope their claim to the finalize-retry loop; the broken-pool message states that the task was already recovered per its retry policy and the pool replaced. The old wording read as task loss when no task is lost.
- The agent skill’s task-info example steered readers to the broker-level
get_task_info_async, whosedecoded_resultis always empty; it now showsapp.get_task_info_async(typed decode) as the primary path.
0.4.3 — 2026-08-02
Section titled “0.4.3 — 2026-08-02”Monitoring read-path performance release. Schema v16
(idx_horsies_tasks_enqueued_at, idx_horsies_tasks_task_name). At 1M
rows / 1.3GB heap, the task list drops from ~167k buffer reads (full seq
scan + top-N sort per page view) to 53, and the unfiltered task-name facet
from ~167k to ~1.2k (index-only scan).
- Schema v16: indexes for the monitoring read path —
enqueued_at(task list default sort) andtask_name(facet index-only scan). Neither column appears in any worker-path predicate or leading sort key; a plan test pins the claim statement toidx_horsies_tasks_claim_pending.
Changed
Section titled “Changed”- The monitoring task list loads only the columns the list view renders;
args,kwargs,result, andtask_optionsare no longer fetched, so a list page stops shipping task payloads (previously up to several MB of egress per 50-row page) and stops detoasting large results. - The list’s
totalis a planner estimate (pg_class.reltuples) on the unfiltered view; any active filter keeps the exact count. A never-sampled table falls back to the exact count. - Sort ordering wraps only nullable sort keys in
NULLS LAST; on the NOT NULLenqueued_atthe wrapper blocked index-order scans while changing no results. - Web UI: the task aggregates (stats, facets, breakdown) no longer refetch on every task event. They refresh on their existing cadences (10s/12s/30s) in both live and fallback modes, on stream reconnect, and on explicit user action. SSE keeps driving the task list, task detail, and workflow run/node views.
- The docs changelog frontmatter failed YAML parsing (a
:sequence in the 0.4.2 summary); the docs site had not deployed since the v0.4.2 merge.
0.4.2 — 2026-08-01
Section titled “0.4.2 — 2026-08-01”Retention and payload-path performance release. Schema v15
(idx_horsies_tasks_queue_retention). Retention sweep defaults changed:
5-minute cadence and 500-row batches replace hourly 5,000-row batches.
- Per-queue terminal retention windows:
RecoveryConfig.queue_terminal_record_retention_hoursmaps queue names to override windows (hours) for plain (non-workflow) tasks. Queues not listed use the global window; overrides apply even when the global window isNone; workflow-backing task rows always age under the global window and are protected while their workflow is non-terminal. After workflow termination, each task row ages from its own terminal timestamp. Override keys must name declared queues — an unknown key fails config construction (and thereforehorsies check) with HRS-200. Override deletes are served by the new queue-leading composite partial index (schema v15); the global delete keeps the v11 index. - Payload-size guardrail:
AppConfig.payload(PayloadPolicy, exported fromhorsies).warn_bytes(default 1 MiB,Nonedisables) logs a warning rate-limited to once per task name and payload kind per process;reject_bytes(defaultNone— off) fails an enqueue closed before any row is written —.send()returnsErr(TaskSendError(PAYLOAD_TOO_LARGE)), a scheduled fire fails its slot. Results are never rejected. The check is one integer comparison against the already-serialized string. Coverage by boundary is documented on the app-config page. RecoveryConfig.retention_sweep_interval_s(default 300, 30s–24h) andRecoveryConfig.retention_delete_batch_size(default 500, 50–10,000) replace the previously hardcoded hourly cadence and 5,000-row batch size.
Changed
Section titled “Changed”- Retention deletes purge
horsies_task_attemptsset-wise in the same statement as the parent task batch instead of relying on the per-row FKON DELETE CASCADE(which remains as a correctness net). The row-level cascade cost more in aggregate than the parent delete itself (at 37,500 deletes/day: 20.6 s/day cascade versus 17.6 s/day parent batch). - Retention batch statements are bounded to tens of milliseconds by the new defaults; the previous hourly 5,000-row batches ran 0.5–1.6 s each while holding row locks.
- The result-wait terminal fetch selects only
task_name,status, andresultinstead of the full task row, so args/kwargs no longer ship with result polls — for kwargs-heavy tasks this removes nearly all poll egress. SameRawResultRecordcontract. - The scheduler’s missing-row existence check runs on a ~60 s wall-clock cadence (every tick while a row is missing or a re-init failed) instead of every tick. At 108 schedules on a 1 s tick the per-tick check read 9.1M rows/day from a 108-row table for an answer that changes only on rare events; worst-case detection of an externally deleted state row moves from one tick to ~60 s.
0.4.1 — 2026-07-27
Section titled “0.4.1 — 2026-07-27”Worker lifecycle fix: a timeout kill landing while the replacement process pool warmed no longer stops the worker. No schema change (v14).
- Executor replacement published the new pool before its children finished
warming, so dispatch ran real tasks on the warming pool; a per-task
timeout SIGKILL landing on one of them — or warm rounds finding too few
distinct children because real tasks occupied them — failed the warmup
itself, and the worker exited fail-closed
(
worker child warmup started X/N process(es)). A backlog of ~12 queued timeout-prone tasks crash-looped the worker at any pool size. The pool is now published only after warmup completes: dispatch waits on the restart lock instead of feeding a warming pool, and no timeout kill can target one. Present since 0.1.8. - A dispatch-path “executor missing” request that lost the race with a concurrent restart could destroy the healthy pool that restart had just built. Ensure and replace are now separate operations: the dispatch path creates a pool only when none is published and never tears one down; force-replace requires the failed pool and is identity-guarded.
- Warmup interruptions are classified by type:
BrokenProcessPooland the newWarmupIncompleteErrorretry the create+warm cycle (3 attempts, stop-aware delay) before failing closed; child-hook failures still stop the worker without retrying; OS-level and memory-baseline failures stay immediately fatal. - Cancellation during warmup (the 30s start-attempt timeout) tears the unpublished pool down instead of leaking its children; a stop requested during warmup discards the pool unpublished.
0.4.0 — 2026-07-27
Section titled “0.4.0 — 2026-07-27”The web monitoring dashboard: a typed monitoring API in core, registry-free
task cancel/retry primitives, a mountable FastAPI app plus a horsies web
CLI behind the new web extra, and the Acme Clothing showcase application.
No schema change (v14).
horsies.monitoring(core, no extra): typed read-query API over tasks, workflows, workers, and schedules, witherror_categoryfiltering expanded server-side from the built-in code registry. An absent row isOk(None);Erris reserved for database failures.cancel_task/retry_task: registry-free, single-transaction compare-and-set task actions. A committed cancel cannot be overwritten by claim, finalize, auto-retry, or reaper paths. Retry reuses the task row, preserves attempt history, never modifiesmax_retriesorgood_until, and emits the queue NOTIFY the insert trigger does not fire for updates. Cancelling a RUNNING task requiresinclude_running=True: the row flips durably but the process keeps executing. Workflow-bound rows are refused.horsies[web]extra:create_monitoring_app(app, *, auth_policy, config)and thehorsies webCLI (app-path form, or registry-less--database-urlform — the latter supports all reads, task actions, and workflow pause/cancel, but not resuming runs whose next nodes carryargs_from). The dashboard SPA ships inside the wheel; no Node at runtime. The library never owns identity: mounted mode requires an explicitMonitoringAuthPolicy; the CLI is fail-closed (loopback default, trusted-header auth required off-loopback, actions off unless--enable-actions).- No-DDL guarantee: brokers constructed by the web layer skip schema migrations; schema state is probed and reported as MATCH / MISMATCH / ABSENT / UNKNOWN. Mismatch serves reads and refuses actions; absence is reported, never initialized; an unreachable database is UNKNOWN, never conflated with absence.
- Live updates over SSE from the existing monitoring NOTIFY channels (one listener per web process, coalesced invalidation events); client polling is a fallback only.
showcase/: Acme Clothing, a runnable demonstration application — 35 tasks, 12 workflow definitions, 31 schedules, 8 scenarios — with deterministic failure draws. See the Quick Start showcase page.- CI:
webuiandshowcasejobs; the release workflow blocks unless the wheel carries the built dashboard assets.
args_fromenvelopes now decode for optional targets: theTaskResult[T, TaskError] | Noneannotation — the documented requirement forjoin='any'/'quorum'targets — previously failedWORKER_SERIALIZATION_ERRORwhenever the source resolved in time. Unions are unwrapped before the envelope passthrough test.
0.3.1 — 2026-07-24
Section titled “0.3.1 — 2026-07-24”The workflow retention DELETEs plan onto their v13 index. Schema v14 — two expression-statistics objects plus an ANALYZE, applied automatically by the broker’s advisory-locked schema init on next startup. Documented PostgreSQL floor raised from 12+ to 14+.
The planner never uses statistics gathered on a partial index for
whole-table selectivity, so the v11/v13 retention indexes provided no
estimate for their own COALESCE expressions and the retention cutoff was
costed at the default 1/3 selectivity. Index-vs-walk then depends on table
size alone: at 1M retained tasks the heap walk is expensive enough that the
index wins regardless; at 36k retained workflows the planner kept a
full-table walk — estimate 12,245 vs 13 actual, 4–5 s per statement, two
statements per hourly reaper pass, independent of eligible-row count — and
the same misestimate degraded the NOT EXISTS guard into a seq scan of the
1M-row tasks table whenever the outer side did use the index. v14 creates
CREATE STATISTICS ... ON (<retention COALESCE>) objects for
horsies_tasks and horsies_workflows — whole-table expression statistics
the planner does use — and runs ANALYZE in the same migration transaction;
extended statistics are empty until the table is analyzed after their
creation, and ANALYZE is sampled, takes SHARE UPDATE EXCLUSIVE, and never
blocks reads or writes. Measured at 1M tasks / 36k workflows / 144k
workflow_tasks with the shipped statements: workflows delete 98 ms →
7.7 ms, workflow_tasks delete 413 ms → 34 ms. No statement changes.
CREATE STATISTICS ON (expression) requires PostgreSQL 14, so the
documented floor moves from 12+ to 14+ (12 and 13 are past end-of-life).
0.3.0 — 2026-07-23
Section titled “0.3.0 — 2026-07-23”Sync send fails closed on a running event loop (breaking for callers that silently blocked their loop), service-loop death is process-fatal, and three child-process expiry/recycle defects are fixed. Schema v13 (one partial index, applied automatically by the broker’s advisory-locked schema init on next startup).
Breaking: sync .send(), .schedule(), .retry_send(), and
.retry_schedule() called inside a running event loop return
Err(TaskSendError(ASYNC_CONTEXT)) before any broker work. The sync
executors are blocking database round trips; inline on a loop they stall
every coroutine on it for the duration of the enqueue. The error carries the
prepared task_id and payload — complete the dispatch with
retry_send_async(err) / retry_schedule_async(err), or call the matching
*_async entry point with the original arguments. Off-loop callers are
unaffected. schedule_async() and retry_schedule_async() are new, on task
functions and with_options(...) builders, with the same delay validation
as their sync counterparts.
Service-loop death is process-fatal. The four worker-lifetime loops (claimer
heartbeat, worker-state snapshot, ping responder, reaper) contain their own
per-iteration errors, so ending with no shutdown requested is a defect — and
a worker that keeps claiming with a dead heartbeat loop stops renewing claim
leases, disappears from monitoring, and contributes no reaper passes while
looking healthy. Previously one log line; now ServiceLoopDiedError is
captured, the worker stops, and run_forever re-raises for a non-zero exit
and supervisor restart.
TASK_EXPIRED workflow finalize gap: a claimed workflow task whose
good_until passed before child start was marked EXPIRED by the child, but
the parent’s finalize skipped phase 2 — the workflow node stayed ENQUEUED
against a terminal task row until reaper recovery case 1.7 repaired it
10–40 s later at default thresholds. Phase 1 now loads the child-persisted
terminal result and proceeds to phase 2, so the workflow resolves per
on_error without waiting for the reaper.
Recycle spawn-failure containment: a failed replacement spawn during child
recycle (fork ENOMEM/EMFILE) killed the executor manager thread with no
containment — submit() kept accepting work and pending futures hung
PENDING forever. The pool is now marked broken, pending futures fail with
BrokenProcessPool, and the worker’s existing recovery requeues and
restarts the executor. Recycling children also exit via os._exit(0) after
flushing output: the result is on the wire before exit, and skipping
teardown of the very heap that triggered the recycle stops head-of-line
blocking sibling results (~35 ms per million heap objects).
Schema v13 adds idx_horsies_workflows_retention: both workflow retention
deletes filter horsies_workflows on terminal status +
COALESCE(completed_at, updated_at, created_at) < cutoff; unindexed, every
hourly retention pass full-walked the workflows table twice, serial under
FOR UPDATE, regardless of eligibility (36k retained workflows ≈ 4–5 s per
statement with zero eligible rows). See the new
operational indexes page for opt-in DDL
covering adopter-side history queries.
0.2.9 — 2026-07-16
Section titled “0.2.9 — 2026-07-16”Two correctness fixes on the workflow-cancel and task-finalize paths, plus a
retention index. Schema v12 (horsies_claim return type + one index, applied
automatically by the broker’s advisory-locked schema init on next startup;
rolling deploys are safe — pre-v12 workers select named columns and never bind
the new fence parameter).
Cancel/completion deadlock: the cancel transaction locked horsies_tasks →
horsies_workflow_tasks → horsies_workflows, while task completion locks
horsies_workflows → horsies_workflow_tasks. A task completing while its
workflow is cancelled deadlocked; Postgres aborted one side with SQLSTATE
40P01 — a spurious DB_OPERATION_FAILED on cancel() or an aborted
completion transaction. Cancel now takes the workflow row lock first, for the
parent and for each descendant in the cascade, matching the invariant every
other lifecycle path already followed: lock horsies_workflows before
horsies_workflow_tasks.
Stale finalize could clobber a live attempt. Finalize and recovery statements
fenced on (status, worker_id) alone cannot reject a stale actor when the
same worker re-claims its own reaper-requeued task: a runner heartbeat starved
past running_stale_threshold_ms gets requeued, the retry NOTIFY wakes the
same worker, it re-claims — and the stalled first attempt, finishing later,
marked the task COMPLETED with its own result while the second attempt was
still executing (attempt history attributed to the wrong attempt). Every
statement acting on a row the worker believes it owns — the fused ok-path
finalize, the err-path context lock, the timeout-handler and future-failure
row locks, the standalone unclaim, orphan termination, and the child’s
CLAIMED→RUNNING ownership confirm — is now fenced to the claim generation:
claimed_at, returned by the claim statements and cleared by every requeue.
The confirm fence also closes soft-cap same-worker double dispatch after
lease expiry.
Schema v12 also adds idx_horsies_heartbeats_sent_at: heartbeat retention
deletes filter sent_at < cutoff, but the composite
(task_id, role, sent_at DESC) index cannot serve a leading-column sent_at
range, so every hourly retention pass scanned the heartbeats heap — the
highest-insert-rate table in the schema. The v11 retention indexes covered
tasks and worker_states and omitted heartbeats.
Docs: the soft-cap concurrency page states the real overshoot bound (with
prefetch_buffer > 0, RUNNING can exceed a queue’s cap for the duration of
the excess tasks, up to N workers × cap), replacing the short-lived-bursts
description.
0.2.8 — 2026-07-09
Section titled “0.2.8 — 2026-07-09”Removes the two remaining full-table scans on the monitoring/retention path and bounds retention deletes. Schema v11 (two indexes, applied automatically by the broker’s advisory-locked schema init on next startup).
list_worker_states previously ran DISTINCT ON (worker_id) over the whole
horsies_worker_states timeseries — Postgres has no loose index scan, so the
read cost scaled with total retained snapshots, not worker count. At 118k
retained rows across 6 workers the call took 10.3 s. It now runs a recursive
skip-scan (one (worker_id, snapshot_at DESC) index probe per worker): 9.2 ms
on the same table, dead/idle workers still listed. On the write side of the
same table, the snapshot cadence is now configurable
(RecoveryConfig.worker_state_snapshot_interval_ms, 1 s–5 min) and its default
moves from the previously hardcoded 5 s to 30 s — ~20k rows per worker per week
at the 7-day default retention instead of ~120k. Set it to 5_000 to keep the
old chart resolution.
The hourly retention pass previously seq-scanned the full horsies_tasks and
horsies_worker_states heaps on every run — even with zero eligible rows — and
ran all five DELETEs unbounded in a single transaction, so enabling retention
on a long-running database meant one DELETE of the entire backlog (WAL burst,
task_attempts cascades, full rollback on any failure). Schema v11 adds
idx_horsies_tasks_retention (a partial expression index over terminal
statuses; a row enters it once, at its finalize transition — claim and
lease-renewal updates never maintain it) and
idx_horsies_worker_states_snapshot_at. Deletes now run in 5,000-row batches,
one transaction per batch, under a 60 s per-pass budget; a larger backlog
drains across consecutive hourly passes, and concurrent passes drain disjoint
batches via FOR UPDATE SKIP LOCKED.
Two fixes. Per-queue max_concurrency is enforced when queue_priorities is
not configured: a cap-only queue previously either ran uncapped (empty priority
map — concurrent claimers over-claimed past the cap) or was never serviced
(partial priority map omitting the capped queue). And workflow finalization
phase-2 replay accepts EXPIRED rows: an expired-before-start task writes
terminal EXPIRED plus a TASK_EXPIRED err result, but the replay reload only
accepted COMPLETED/FAILED and discarded it as “terminal task result
unavailable”, wedging in-process replay for the expired case.
0.2.7 — 2026-06-24
Section titled “0.2.7 — 2026-06-24”Collapses the worker claim critical section into a single server-side statement.
Each claim pass previously held the cap-serialization advisory lock across many
client round trips — one pg_advisory_xact_lock per capped queue, the cap-counts
query, per-queue claims, then COMMIT — so at pooled-connection RTT a
client-side stall (GC/CPU starvation, a hung pooled connection) while holding the
lock froze every claimer cluster-wide. The new horsies_claim(...) SQL function
(schema v10, applied automatically by the broker’s advisory-locked schema init on
next startup) acquires the locks, computes the cap/budget accounting, and runs
the windowed claim in one statement, so the lock is held only across that
statement plus the commit, never across a client round trip. The claim pass now
issues one statement under the lock instead of 7–9; at ~33 ms RTT with 16
concurrent claimers the worst single advisory-lock wait drops from ~5 s to ~0.5 s
and throughput scales with worker count instead of flat-lining at the lock’s
serialization ceiling. Cap enforcement is unchanged: the function never
over-claims (under SKIP LOCKED contention it may claim fewer rows per pass,
deferring them to the next pass). One behavior change: when two queues share the
same priority, the tie-break was previously the configured queue order; the claim
now pools the equal-priority band and orders by task priority, then enqueue time
(FIFO), so equal-importance tasks across such queues are claimed FIFO while an
explicit task/workflow-node priority still preempts within the band. Distinct
queue priorities are unaffected.
0.2.6 — 2026-06-23
Section titled “0.2.6 — 2026-06-23”Subworkflow tasks now persist at their queue’s configured priority. A
parameterized sub-workflow whose build_with() returns a direct
WorkflowSpec(...) bypassed app.workflow() binding, so its TaskNodes reached
the engine with priority=None; the engine then defaulted them to a literal
100. On a CUSTOM queue (e.g. scraping at priority 30) the child task landed
on the right queue but at the wrong intra-queue claim order
(ORDER BY priority ASC, enqueued_at ASC). Queue and priority resolution is now
centralized in a single bind boundary (resolve_node_queue_and_priority) shared
by app.workflow(), the engine subworkflow child branch, and check(),
replacing three divergent else 100 fallbacks. An explicit node.priority is
preserved; None inherits the queue priority. DEFAULT-mode queues are unchanged
(priority 100). An invalid child queue is now contained with
WORKFLOW_ENQUEUE_FAILED rather than guessing a default. No schema change
(still v9). Also rolls in dev and website dependency bumps via Dependabot.
0.2.5 — 2026-06-17
Section titled “0.2.5 — 2026-06-17”The workflow reaper no longer races healthy finalizers. Task finalization is two
phases (Phase 1 commits the task terminal; Phase 2 advances the workflow DAG),
and the reaper’s Case 1.7 recovery fired the instant a task went terminal — so
under load (amplified by frequent child recycling) it “recovered” tasks whose
Phase 2 was merely in flight, adding up to one reaper interval of latency and
noisy crashed worker logs. Recovery now honours a grace window
(RecoveryConfig.crashed_worker_recovery_grace_ms, new, default 10s, independent
of the heartbeat-coupled thresholds): a task terminal within the window is left
for its in-flight finalizer; only genuinely-stuck tasks are recovered (a genuine
crash recovers after the grace plus one reaper sweep). Correctness was never at
risk (recovery replays the stored result idempotently; the task body never
re-runs) — this is a latency and log-noise fix.
Idle pooled broker connections reaped server-side (e.g. PlanetScale’s PgBouncer
pooler, which drops idle connections within ~1–2h) surfaced as a mid-query
OperationalError on the next claim or heartbeat — pool_pre_ping and
pool_recycle are checkout-time guards and cannot catch a connection that dies
in-flight. New TCP keepalive fields on PostgresConfig (tcp_keepalives,
default on, with tcp_keepalives_idle/interval/count) keep idle sockets
warm at the socket layer. libpq enables keepalives by default but leaves the
idle interval at the OS default (often 7200s); Horsies sets it to 30s, applied
to the broker engine pool and each child-process pool. No configuration is
required for remote/pooled deployments.
A direct WorkerConfig(pgbouncer_transaction_mode=True) built without
child_connect_kwargs left child pools with prepared statements enabled against
a transaction-pooled PgBouncer. WorkerConfig.__post_init__ now ensures
prepare_threshold=None in child_connect_kwargs when the flag is set (without
overriding an explicit value), so the flag alone is sufficient; the CLI path was
unaffected.
0.2.4 — 2026-06-17
Section titled “0.2.4 — 2026-06-17”Per-child memory recycling complements count-based --max-tasks-per-child: a
task count is a poor proxy for a bytes budget, so the correct recycle point
depends on a child’s RSS. New --max-memory-per-child-mb N (off by default,
CPython-only, forces spawn) recycles a child once its own resident memory
reaches N MB — the child samples RSS after each task and exits cleanly via the
stdlib exit_pid marker, replacing only that child. A startup baseline guard
fails the worker if the threshold is at or below the warmed child baseline.
Also fixes a latent CPython gh-115634 hang in the existing count-recycle path:
count recycling now overrides _adjust_process_count so a recycled child is
always replaced, falling back to the stock pool if the internals are absent.
Worker logging no longer writes raw ANSI color escapes to non-TTY sinks (log
drains, container logs, journald, files), which broke grep and log parsers.
Color is gated by stream isatty(), with NO_COLOR / FORCE_COLOR overrides;
non-TTY output is plain text with the same layout. Task execution log lines also
stop wrapping the id/name in one-element lists; both the start and completion
lines now read task_name (task_id).
0.2.3 — 2026-06-16
Section titled “0.2.3 — 2026-06-16”Worker child processes can now be recycled to bound memory. Long-lived
executor children accumulate memory the OS never reclaims (allocator
high-water from heap fragmentation, C-extension caches, leaks), which crashes
memory-quota platforms (containers, PaaS dynos). --max-tasks-per-child (default 100)
recycles each child after N tasks; new children_memory_mb telemetry exposes
the per-child footprint the parent-only memory_usage_mb metric hid. Schema
bumped to v9 (additive). Behavior change: recycling is on by default and
forces the spawn start method (incompatible with fork); set
--max-tasks-per-child=0 to keep fork/no recycling.
--max-tasks-per-child Nworker flag (WorkerConfig.max_tasks_per_child,N >= 2, default100): recycle each worker child process after N tasks (per-child and staggered) to bound memory for workloads that retain it (allocator high-water, C-extension caches, leaks).0disables recycling. Behavior change: because the stdlib budget is incompatible withfork, any non-zero value (including the default) forces thespawnstart method — on Linux this replacesfork, so children re-import the app instead of fork-cloning the parent (higher baseline RSS, slower child startup). Set--max-tasks-per-child=0to keepfork/no recycling.children_memory_mbcolumn onhorsies_worker_statesand field onWorkerStateSnapshot: summed RSS of the executor child processes. The existingmemory_usage_mbis the parent process only; per-child memory growth (the memory-quota driver) was previously invisible. Schema bumped to v9 (additive, idempotentADD COLUMN).
0.2.2 — 2026-06-15
Section titled “0.2.2 — 2026-06-15”Producer-side strictness lands on both axes: task parameters must be
keyword-only, and encode_value validates a value against its declared type
before serializing — so a positional or mistyped task call fails at the type
checker or at app.check instead of returning an ignorable Err that silently
drops the send. Schedules become kwargs-only and app.check() validates them
as a preflight. Orphaned workflow tasks self-heal instead of churning the
requeue loop. No schema change (still v8).
- Orphaned workflow tasks (a
CLAIMEDworkflow task whoseworkflow_tasklinkage is missing or terminal) are no longer requeued and re-dispatched forever. They are cancelled — at finalization when detected, and by a reaper self-heal step — which frees in-flight budget and lets retention sweep them. - Retention no longer orphans a live task row: a terminal, expired workflow is
retained until every backing task is terminal, instead of deleting the
workflow/
workflow_taskrows while a backing task is still non-terminal. encode_valuevalidates a value against its declared type before serializing.dump_pythonalone only serialized — a mistyped value (adict/int/listin astrslot) passed through with a warning. The producer now fails closed, symmetric withdecode_valueon the consumer; mistyped task kwargs, results, andargs_frombindings are rejected at send / atapp.check. (#146)
RecoveryConfig.auto_terminate_orphaned_workflow_tasks(defaultTrue): cancel orphaned workflow tasks at finalization and in the reaper. WhenFalsethey are leftCLAIMEDfor inspection (never requeued or retention-deleted).app.check()validates configured schedules (timezone, task registration, queue, and the kwargs wire contract) as a preflight phase, shared with scheduler boot. Skipped for the worker role, which never enqueues schedules. Previously a malformed schedule passedcheckand only failed at scheduler startup.
Changed
Section titled “Changed”- Task parameters must be keyword-only.
check_task_signaturenow rejectsPOSITIONAL_OR_KEYWORDparams, so a task is declareddef f(*, x: T).ParamSpecthen carries the keyword-only-ness, turning a positional call (f.send(42)) into a call-site type error instead of a runtimeErr(VALIDATION_FAILED)that an unchecked caller silently drops. The producer-side runtime guard stays as defense-in-depth. Breaking: add a bare*,before the first parameter of every task definition. Execution is unaffected (the worker already passes every argument by keyword).
Removed
Section titled “Removed”TaskSchedule.argsremoved; schedules are kwargs-only (strict-serde has no positional wire form).TaskSchedulenow setsextra='forbid', soargsor any unknown field raises a validation error at construction instead of being silently dropped. Breaking: pass every scheduled argument as akwargsentry.
0.2.1 — 2026-06-14
Section titled “0.2.1 — 2026-06-14”A failed outputless child workflow used through a SubWorkflowNode leaked the
internal terminal-results envelope into the parent node, raising a strict-serde
reserved-key error that wedged the parent and, via the recovery loop, starved
recovery for every stuck workflow. The propagation path and recovery isolation
are both fixed. No schema change (still v8).
- Outputless child workflows no longer leak the internal terminal-results
envelope into parent
SubWorkflowNoderesults orSubWorkflowSummary.output. Completed outputless children propagate asTaskResult[None, TaskError]withok=None. - Workflow recovery now isolates candidate failures so one poison workflow row cannot abort the whole recovery pass.
Changed
Section titled “Changed”WorkflowDefinition[T]with noMeta.outputis now rejected for concreteT. UseWorkflowDefinition[None]for outputless orchestration workflows, or setMeta.outputto a node producing the declared type.
0.2.0 — 2026-06-12
Section titled “0.2.0 — 2026-06-12”Worker hot-path statement budget halved (27.2 -> 12.8 statements per task measured at 33ms RTT): optional per-checkout health checks, child pre-exec collapsed to one transaction, and plain-task ok-path finalization fused into a single statement. Remote soft-cap throughput 2.35x; local 1.15-1.36x. Fixes a reaper-breaker misclassification that could strand CLAIMED tasks after connection-slot exhaustion. No schema change (still v8).
Performance
Section titled “Performance”- Child pre-exec runs one transaction instead of three: the redundant pre-flight expire/workflow check is deleted (the RUNNING transition’s guards and miss-path diagnosis already enforce it atomically) and the first runner heartbeat rides the RUNNING transaction. (#134)
- Plain-task ok-path finalization is one statement
(
FINALIZE_TASK_COMPLETED_SQL): lock, attempt upsert, COMPLETED CAS, and capacity notify in a single transaction; phase 2 is skipped for this path. Err results, workflow tasks, and decode failures keep the multi-statement flow. (#134) - The child result payload is decoded before any SQL;
WORKFLOW_STOPPEDresults finalize without opening a session. (#134)
worker_child_pool_check(PostgresConfig, defaulttrue): disable the per-checkout health check on child pools; pairs withpool_pre_ping=falsefor high-RTT deployments (~20% of the per-task statement budget at 30ms+ RTT). (#134)- Remote PostgreSQL deployment guide: connection-budget formula, pooled multi-worker setup, prefetch at high RTT, health-check trade-offs. (#136)
sqlalchemy.exc.TimeoutError(engine pool checkout timeout) now classifies as retryable. Previously three consecutive reaper passes during connection-slot exhaustion latched the stale-CLAIMED requeue breaker off for the process lifetime, leaving orphaned CLAIMED tasks without their designed backstop after the pressure cleared. The mark-failed breaker had the same exposure. (#135)
Changed
Section titled “Changed”- The first runner heartbeat commits atomically with the RUNNING transition: a task row is never observable as RUNNING without heartbeat coverage. The heartbeat thread no longer sends an immediate beat. (#134)
- Capacity notify for plain ok-path tasks fires on the finalize commit instead of a separate phase-2 transaction. (#134)
- Reaper breaker state machine pinned: counter resets on transient failures and successes, 3-failure latch, latched operations skipped, breaker independence, pool-timeout-counts-as-transient end to end. (#137)
0.1.10 — 2026-06-12
Section titled “0.1.10 — 2026-06-12”Round-trip elimination across every workflow hot path (completion, dependent promotion, subworkflow child start) and claim-pass cap accounting; the per-workflow completion ceiling roughly 2.4x at remote-database RTTs. No schema change (still v8).
Performance
Section titled “Performance”- Claim-pass cap accounting is one statement. The worker-local claimed
and in-flight counts, the optional cluster-wide count, and every capped
queue’s hard/soft count arrive in a single FILTER-aggregate read
(
CLAIM_PASS_COUNTS_SQL) instead of 2 + Q (+1) sequential statements under the claim advisory locks: an empty pass over Q=3 capped queues drops from 11 statements to 7. Count predicates are unchanged (verified column-for-column against the single-purpose statements, which remain for health snapshots); all counts now share one now() instant instead of one per statement. - Subworkflow child start is bulk-inserted. Starting a SubWorkflowNode’s
child workflow now writes all child node rows in one executemany over
the same bulk statement the batched workflow start uses, and child
TaskNode roots are inserted directly as ENQUEUED with their task rows
in a second executemany (child roots cannot carry
args_from/workflow_ctx_from— spec validation requires both to referencewaits_for). The parent workflow’s name rides the enqueue CAS RETURNING instead of a separate read. Statements: 4 + C + 3R -> 5 flat; measured against a remote (~33-45ms RTT) Postgres, a 50-child flat child start dropped from ~9.3s to ~0.3s. SubWorkflowNode roots still recurse; a root whose task_options fail to parse demotes to the per-node path so corruption keeps failing that child root, not the parent. A slow root’s failure that pauses the workflow (itson_error=PAUSE) now gates the fast roots: their task rows are inserted only if the workflow is still RUNNING after slow roots ran and any queued child-to-parent propagation was drained; otherwise they revert to READY with task_id cleared — a paused workflow gains no runnable task rows (the same strengthened pause contract as batched promotion). This also closes the identical pre-existing corner in the batched workflow START (shipped in 0.1.9), where fast-root task rows landed before the slow-root loop and before synchronously failed child propagation. - Dependent promotion is batched per skip-cascade level. Completing a
task that unblocks F plain-TaskNode dependents (args_from included)
now runs a fixed pipeline — one grouped config+dependency-status
evaluation, batched PENDING->READY / ->SKIPPED CAS writes, one grouped
dependency-results read, one batched READY->ENQUEUED CAS returning the
insert payloads, one bulk task INSERT and one bulk LINK — instead of 7
statements per dependent under the workflow lock. Statements are flat
in F (8 vs 6+7F+2); measured against a remote (~33-45ms RTT) Postgres,
a 1-root -> 119-dependent promotion dropped from ~34s to ~0.6s of
lock-held time. SubWorkflowNode and
workflow_ctx_fromdependents keep the per-node path. Join semantics, skip cascades, pause/cancel guards, per-row CAS, and per-node failure isolation are unchanged; the payload builder is shared with the per-node path so they cannot diverge. One strengthening vs the sequential path: underon_error=PAUSE, a payload-build failure in a promotion level (a horsies bug path — corrupt persisted rows) pauses the workflow, stops the level, and reverts the level’s already-CAS’d-but-uninserted siblings to READY — a paused workflow gains no new runnable task rows, and the siblings sit in the recovery-covered shape (resume re-enqueues READY nodes). The sequential loop’s post-pause state depended on processing order (siblings processed before the failure were already enqueued). Pinned by test_pause_policy_build_failure_reverts_siblings_to_ready. - Task completion runs in half the round trips. The completion path’s
locate -> lock -> CAS-update triple is one statement
(
COMPLETE_WORKFLOW_TASK_SQL: locate the node by backing task id, take the workflow row’s FOR UPDATE lock, CAS to terminal status, return the progression context), the post-update status/depth reads are gone (the held lock freezes the workflow row, so the locked row’s values are authoritative), the completion check no longer re-acquires the already-held lock, and the failure path no longer re-acquires it a third time or re-readson_error. Per success completion with one pending dependent: 10 statements -> 5; failed completion (on_error=FAIL): 14 -> 7. Measured against a remote (~33-45ms RTT) Postgres: 491ms -> 328ms per completion (failure path 654ms -> 335ms). Completions of the same workflow serialize on the workflow row lock, so the shorter lock-held window raises the per-workflow completion ceiling from ~2.3-3/s to ~3.2-4.3/s.
Changed
Section titled “Changed”on_workflow_task_completerequires a keyword-onlytask_name(callers read it from the task row they already hold); the result envelope is encoded before the first statement. The worker threadstask_namethrough dispatch and both finalize retry stages; the phase2 pre-flight workflow-task existence check is removed (the merged statement self-detects non-workflow tasks at the same cost).- The completion-encode failure fallback (a horsies bug path: the result
envelope cannot be encoded) now stores
TaskError.data={'task_id','task_name'}instead of{'workflow_id','task_index'}— the workflow context is not known before the merged statement runs.
0.1.9 — 2026-06-11
Section titled “0.1.9 — 2026-06-11”Workflow-start batching and per-queue claim-lock scoping: round-trip elimination on the two hot multi-statement paths.
Performance
Section titled “Performance”- Workflow start is batched: node rows and fast-path root tasks (plain
TaskNodes without
args_from/workflow_ctx_from) are built in memory and inserted in a fixed handful of pipelined statements instead of one INSERT per node plus three statements per root. Measured against a remote (~33ms RTT) Postgres: starting a 119-root workflow dropped from ~16s to ~0.35s; statement count is flat in workflow size. Subworkflow roots andargs_from/ctx_fromroots keep the per-row path. Start semantics are unchanged:Ok(handle)still means durably persisted, one transaction, idempotent restart by workflow_id, whole-start rollback on failure.
Changed
Section titled “Changed”- The claim advisory lock is scoped per capped queue (cluster_wide_cap keeps the single global key): workers claiming disjoint capped queues no longer serialize against each other. During a rolling deploy, old and new workers do not contend with each other, so a per-queue cap can briefly overshoot by up to one pass’s batch until the fleet is on one version.
0.1.8 — 2026-06-11
Section titled “0.1.8 — 2026-06-11”Workflow-completion performance redesign, supervisor-contract fixes, scheduler state self-healing, and the close of the raise-contract documentation track. Schema migrates v7 → v8 automatically on first broker start.
Changed
Section titled “Changed”- Workflow completion at scale: terminal-set resolution rewritten as a
payload-free edge read plus in-process set difference (finalizing
completion at 1000 tasks: 168.7ms → 11.7ms under the workflow lock);
child-workflow info collapsed to a single-pass query; new composite
index
(workflow_id, status, task_index)(schema v8) removes the per-failure first-failed scan. - Subworkflow parent propagation is de-nested: each ancestor level now
advances in its own transaction instead of recursing root-ward while
holding every descendant’s
FOR UPDATElock. A child workflow’sworkflow_doneNOTIFY is therefore visible slightly before its parent node advances (waiters re-read their own workflow’s status on wake). A crash between propagation levels is healed by workflow recovery; full self-healing requiresrecovery_config(the CLI wires it; programmatic workers should too).
Breaking
Section titled “Breaking”TaskSchedule.timezoneis validated at scheduler startup: an invalid IANA name now exits 1 at boot (CONFIG_INVALID_SCHEDULE) instead of leaving the schedule silently dormant with a per-tick init failure.
- A worker whose executor restart failed from a background finalizer path now exits non-zero for supervisor restart instead of running on as an executorless zombie that claims nothing.
horsies workerexits 1 (was 0) when startup times out after exhausting the resilience retry budget — a clean exit suppressed supervisor restarts.- Listener
UNLISTENfailures during unsubscribe no longer raise into result-waiter cleanup paths, and the channel is always untracked so a reconnect cannot resurrect a ghostLISTEN. app.check(): a workflow builder whose signature cannot be introspected now folds into the validation report instead of crashing the check phase.- Schedule state rows self-heal: a schedule whose state-row creation failed at scheduler startup (or whose row was deleted externally) was invisible to the tick loop and stayed dormant until a restart. Each tick now recreates missing rows for enabled schedules.
Removed
Section titled “Removed”ScheduleStateManager.delete_state— dead since schedulers stopped deleting foreign schedule-state rows at startup; no production caller.
Documentation
Section titled “Documentation”- Raise-contract docstrings across the worker package, workflow engine, scheduler, app/CLI boundary, and listener: every fallible function now names the seam that recovers from its failure and how (the fallible-audit Result-conversion track is closed).
0.1.7 — 2026-06-10
Section titled “0.1.7 — 2026-06-10”Correctness and performance hardening from a full-project review, plus task timeouts, child-process hooks, and uncapped queue concurrency. Schema migrates from v2 to v7 automatically on first broker start.
CustomQueueConfig.max_concurrencyacceptsNoneas an explicit uncapped sentinel (mirroringcluster_wide_cap=None): no per-queue limit is enforced and the claim pass skips that queue’s in-flight count query.0remains valid (pauses claiming); negative values are now rejected at config validation.- Per-child-process hook:
@app.on_child_process_startregisters sync zero-argument functions that run once in every worker child, after task imports and before horsies opens its own child pool. The supported seam for disposing fork-inherited app engines and setting worker-specific pool policy (Celeryworker_process_init/ Dramatiqafter_process_bootparity). Fail-closed: a raising or hung hook (10 s budget) exits the child with a dedicated code and the worker stops with the hook named instead of restart-looping. - Per-task execution timeout:
@app.task(..., timeout_ms=...)(minimum 1000 ms, measured from dispatch). On expiry the worker records aTASK_TIMEOUTattempt, fails the task — or schedules a retry when"TASK_TIMEOUT"is inauto_retry_for— and kills the child process. The kill restarts the worker’s process pool; sibling tasks in flight recover through crash recovery. A deadline that fires before user code starts requeues the task instead.
Breaking
Section titled “Breaking”catch_up_missed=Falsenow matches its documentation: after scheduler downtime, only the most recent due slot fires (skipped slots are logged) and the schedule resumes strictly in the future. The previous behavior accidentally replayed the entire backlog one run per tick; deployments relying on that replay must setcatch_up_missed=True.PostgresConfig.database_urlandsession_database_urlare now pydanticSecretStr—repr()/model_dump()mask credentials. Code reading these fields must call.get_secret_value(). String inputs validate as before.- Producer pool defaults dropped from
30 + 30to SQLAlchemy’s5 + 10(pool_size/max_overflow); raise them explicitly for high-throughput producers. - Scheduler startup no longer deletes
horsies_schedule_staterows absent from its config (this broke rolling deploys and shared-database topologies); orphan rows are kept and logged. TaskSchedulepositionalargsare rejected at scheduler startup (they always failed at enqueue; now they fail fast).- Spec validation rejects
join='any'/'quorum'nodes whoseargs_fromtargets a parameter without a default (WORKFLOW_INVALID_JOIN), and@app.taskregistration rejects Enums with non-JSON-native member values.
- Worker finalize/retry SQL now requires claim ownership: a stale finalizer (its task reaper-requeued and re-claimed by another worker) could overwrite the new owner’s in-flight attempt, corrupt the attempt history, or trigger a third execution.
- Workflow terminal marks require
status='RUNNING': a task left running throughcancel()could flip a CANCELLED workflow to COMPLETED/FAILED on completion and cascade the resurrection into parent workflows. resume_workflow/cascade_resume_to_childrendecode stored dependency results with the app registry again — resumedargs_fromconsumers receivedRESULT_DESERIALIZATION_ERRORsentinels instead of the real upstream results.stop()shuts the executor down before draining finalizers: a task finishing after the drain timeout had its completed result discarded and was recorded (and possibly re-executed) asWORKER_CRASHED.- Subworkflow completion takes the parent workflow lock before promoting dependents, closing a fan-in race that left nodes PENDING with all dependencies terminal until the next reaper sweep.
TaskHandle.get()no longer caches transient errors: one broker hiccup or a not-found racing the enqueue poisoned the handle permanently.- Schedules no longer wedge on permanent enqueue errors (e.g.
PAYLOAD_MISMATCHafter a deploy changed kwargs): the doomed slot is skipped and the schedule keeps running. _schedule_retryreads the queue name from the already-locked row instead of a second pooled session (pool-starvation deadlock under mass failure), and finalize retryability is keyed on the recovery DB outcome instead of the child-future exception type.import_file_pathrolls backsys.modulesand its cache when module execution fails (a broken module was silently returned as success on the next import) and no longer registers basename aliases that shadow other importable modules.- A deterministic result-encode failure during workflow completion now degrades to a FAILED node with a serialization-error envelope instead of looping phase-2 finalize retries forever.
Performance
Section titled “Performance”- Claim path: partial composite indexes for both eligibility arms plus
a split-arm
CLAIM_SQL— measured ~430× faster claim passes at a 50k-row pending backlog. The pending arm walks its composite inORDER BYorder and stops at the limit; the expired arm carries two complementary partial indexes (expiry filter for the few-expired steady state, ordered composite for deep expired backlogs — measured 30.7ms → 0.11ms at 50k expired rows) with the planner choosing per data distribution. The cluster-wide claim advisory lock is taken only when cluster/queue caps require serialized accounting, and its key is a fixed constant (DSN-derived keys silently split the lock between workers using different DSN spellings of the same database). - Workers subscribe to their queue channels only: the global
task_newchannel woke every worker for every insert cluster-wide (thundering herd). The trigger still emitstask_newfor external observers. - Notify triggers split into INSERT/UPDATE pairs gated by
WHEN (OLD.status IS DISTINCT FROM NEW.status)— lease renewals no longer invoke plpgsql per row. - Result waiters get payload-keyed dispatch on
task_done(one shared LISTEN, per-task delivery) and the wait loop polls a slim status probe instead of the full row with TOASTed payload columns. - Reaper passes are gated by a cluster-wide try-advisory-lock (one executing reaper per interval instead of one per worker), stale-claim requeue locks only genuinely stale rows, pending expiry runs in bounded SKIP LOCKED batches, and two per-iteration session-churn sites are gone.
- Six write-amplifying single-column indexes on
horsies_tasksdropped (every lifecycle UPDATE wrote entries into all of them); dependency lookups use@>so the GIN index actually applies; heartbeat and worker-state timeseries PKs widened to BIGINT (int4 sequences exhausted in months at heartbeat rates). - Listener notification connections enable TCP keepalives so silently dropped connections surface within ~60s instead of hanging the dispatcher.
Changed
Section titled “Changed”- Global workflow-recovery passes are capped at 200 rows per candidate query per pass; resume-scoped passes remain uncapped. Successive passes converge on large backlogs without one pass holding its session and transaction throughout.
- In-process task calls return the lax-coerced ok value (e.g.
Ok('5')for a declaredintreturns5), matching what wire consumers decode. WorkerConfig.__repr__masks its DSN fields.- New docs: datetime round-trip caveats, scheduler DST behavior, exception-mapper exact-class matching, and the database trust boundary.
0.1.6 — 2026-06-04
Section titled “0.1.6 — 2026-06-04”- Schema v2 task lifecycle metadata:
is_workflow_task,finalizing_at, andfinalizing_by_worker_id. Existing workflow-linked task rows are backfilled once during migration; new direct task sends and workflow enqueues write the flag explicitly. RecoveryConfig.finalizing_stale_threshold_msto protect the child-to-parent finalization handoff without disabling stale child recovery.- Worker-specific broker pool settings:
worker_pool_size,worker_max_overflow,worker_child_pool_min_size, andworker_child_pool_max_size.
Changed
Section titled “Changed”- Worker claim batching now defaults to filling available local/global capacity
(
max_claim_batch=0), while a positivemax_claim_batchremains an explicit per-queue fairness cap. - Soft-prefetch local budgeting counts already-owned
CLAIMEDrows, preventing a worker from hoarding beyondprocesses + prefetch_buffer. - Plain tasks skip workflow-specific child preflight/finalize checks using the
persisted
is_workflow_taskflag. - Worker processes now use a smaller coordinator pool by default (
3 + 2overflow) while producer/web broker defaults remain unchanged (30 + 30). - Child worker psycopg pools now default to
min_size=0,max_size=2instead of1..5, and are configurable fromPostgresConfig. - Worker child processes are warmed before the parent opens long-lived database sockets, and replacement executors use a non-inheriting start method after startup, avoiding fork inheritance of psycopg connections.
BrokenProcessPooland child-future failures now distinguishCLAIMEDwork fromRUNNINGwork.RUNNINGtasks respectWORKER_CRASHEDretry policy or persist a terminalWORKER_CRASHEDresult instead of being blindly requeued.- Finalization failure handling now preserves queue/workflow context across
retries, schedules phase-2 retries after terminal state is committed, and
returns synthetic
TaskResultpayloads for worker-failure/corrupt-result terminal paths so workflow advancement and capacity notifications are not skipped. - The reaper skips recent
finalizing_athandoffs, recovers stale finalizing rows afterfinalizing_stale_threshold_ms, and still recovers hung child processes even when the parent worker coordinator is alive. - Workflow pause now cancels claimed-but-not-started internal task rows and
resets their workflow nodes to
READY, including retry-window nodes already markedRUNNING, so resume enqueues fresh task rows instead of leaving orphan claims.
0.1.5 — 2026-06-02
Section titled “0.1.5 — 2026-06-02”ping_workers(min_responses=N)/ping_workers_async(min_responses=N): return as soon asNdistinct workers reply instead of waiting the fulltimeout_seconds.min_responses=1is a fast fail-open liveness gate — a healthy fleet answers in milliseconds; only a degraded fleet pays the timeout. Removes the latency floor for high-frequency/healthprobes. Pongs are de-duplicated byworker_id. See Worker & Database Health.
0.1.4 — 2026-06-02
Section titled “0.1.4 — 2026-06-02”0.1.4 adds a typed worker & database health API: active ping-pong liveness for
workers, a database reachability probe, and typed reads over the worker-state
timeseries (including idle workers). It retires the untyped get_worker_stats.
- Database reachability probe:
app.ping_database_async()/ping_database()runSELECT 1through the live broker pool and returnBrokerResult[DatabasePing]with measured round-trip latency. Callable from a running event loop. - Active worker ping-pong:
app.ping_workers_async(target_worker_id=None, timeout_seconds=2.0)/ping_workers()broadcast a ping over LISTEN/NOTIFY and collectWorkerPongreplies within the window. A reply proves the worker’s event loop is responsive and that it can reach Postgres. Passtarget_worker_idto probe one worker. - Typed worker-state reads over the
horsies_worker_statestimeseries:app.list_worker_states_async()(latest snapshot per worker, including idle workers),get_worker_state_async(worker_id), andget_worker_state_history_async(worker_id, limit=None)returningWorkerStateSnapshot. See Worker & Database Health. - New exports:
DatabasePing,WorkerPong,WorkerStateSnapshot. - New broker error codes:
DB_PING_FAILED,WORKER_PING_FAILED.
Removed
Section titled “Removed”broker.get_worker_stats()(untypedlist[dict], RUNNING-tasks only, missed idle workers). Useapp.list_worker_states_async()— typed and inclusive of idle workers.
0.1.3 — 2026-05-31
Section titled “0.1.3 — 2026-05-31”0.1.3 adds CronSchedule, a typed 5-field cron-style schedule pattern. It brings
wall-clock alignment and minute-offset load staggering that IntervalSchedule
cannot express — without cron strings.
CronSchedule: a typed 5-field cron-style schedule pattern (minute,hour,monthterm lists plus aDaySelector). Provides wall-clock alignment and minute-offset staggering thatIntervalSchedulecannot express, with no cron strings. The day-of-month vs day-of-week ambiguity is explicit throughEitherDay(OR) andBothDays(AND). See Schedule Patterns. New exports:CronSchedule,Month,CronEvery,CronStep,CronValues,CronRange,CronEnumValues,CronEnumRange,CronEnumStep,DaySelector,EveryDay,ByMonthDay,ByWeekday,EitherDay,BothDays.
0.1.2 — 2026-05-29
Section titled “0.1.2 — 2026-05-29”0.1.2 is a breaking release headlined by the strict-serde redesign: the wire stops carrying class identity, and every task parameter and return type must classify into a concrete declared shape. It also closes a large batch of workflow, worker, scheduler, and broker correctness bugs found in the 2026-05-24 audit.
For mechanical, copy-paste upgrade steps, see the Migrating to 0.1.2 guide.
Breaking Changes
Section titled “Breaking Changes”Serialization (strict-serde)
- The wire format no longer encodes class identity. The receiver’s declared
type drives every decode through
pydantic.TypeAdapter; values carry only a single envelope marker, not per-value class tags. Banned types (Any,object, baredict/list/tuple,TypeVar, bareBaseModel,TypedDict,bytes,set/frozenset,Callable,pathlib.PurePathsubclasses) are rejected at@app.taskregistration. (#54) - Legacy
codec/serde.py(to_jsonable/rehydrate_value/ class-tag envelopes) is removed. The strictcodec/json_io(dumps_json/loads_json) is the only JSON boundary;serialize_error_payloadandserialize_task_optionsmoved to dedicated modules. (#84) broker.get_result/broker.get_result_asyncare removed. Useapp.get_result(_async)for a typed decode (returns an outerBrokerResultwrappingTaskResult[Any, TaskError]) orbroker.get_raw_result_record_asyncfor the raw envelope. (#84)dumps_jsonrejects tuples and non-string dict keys instead of silently coercing them to lists / string keys. Tuple-annotated parameters still round-trip through the typed path. (#82)dumps_jsonrejects non-UTF-8 output (lone surrogates) up front instead of failing later on the PostgresTEXTinsert. (#71)SubWorkflowNode.kwargsandSubWorkflowSummary.outputare now typed, and aSubWorkflowNode’s result type resolves by uniquedefinition_key. Child workflow definitions must declare a stabledefinition_key. (#63, #68)SubWorkflowSummary.from_jsonfails closed (raisesValueError) on a corrupt or unknown status instead of silently coercing toFAILED. (#70)
Workflow API
- Subworkflow enqueue now requires an explicit
brokerargument. (#55) WorkflowHandle.get()/get_async(timeout_ms=0)now returns immediately — the result if the workflow is already terminal, otherwiseWAIT_TIMEOUT— instead of blocking forever. Onlytimeout_ms=Nonedisables the timeout. (#86)
Tasks, scheduling & CLI
.schedule(delay=...)rejects negative and non-integer delays at the wrapper boundary (delay=0still means “enqueue now”). (#73)- Scheduled tasks validate kwargs-encodability at startup and reject positional arguments (kwargs-only contract), instead of failing on every tick. (#80)
- The CLI rejects a conflicting
-m/--moduleflag and positional module path instead of silently preferring the flag. (#76)
- Opt-in
RetryPolicy.max_delay_secondscap so exponential backoff no longer grows unbounded. (#44) encode_task_errorcodec helper and a publicdecode_task_errorthat polymorphically preservesSubWorkflowError. (#59, #60)- Err-only fast path in recovery and
get_task_info(failed tasks surface without a localok_type). (#61) __wrapped__on the task wrapper soinspect.signature/inspect.unwrapresolve the original function. (#75)
Workflow engine & recovery
- Duplicate
waits_foredges no longer falsely trip cycle detection. (#22, #30) - Partial child-workflow commits are prevented on a mid-loop validation
failure, so recovery can no longer miss an orphaned
RUNNINGchild. (#24) - Terminal-state CAS guards added to the workflow-task-failed and parent-node-result updates. (#25, #26)
- First-failure error semantics preserved: per-failure error selection is serialized and recovery recomputes the first-by-index error. (#27, #29)
on_error=PAUSEcascades the pause to running child workflows. (#28)- Subworkflow cycle detection is keyed on definition identity, not the display name. (#33)
WorkflowContextprivate result/summary state survives dump/restore. (#31)- Ready subworkflows stay ready without a broker, and recovery re-evaluates demoted nodes in the same pass. (#62)
- Workflow tasks are locked before cancellation, closing a worker-pickup race. (#65)
- All
args_fromtype errors are surfaced in a single pass. (#69) app.workflow()no longer mutates the caller’sTaskNode. (#79)WorkflowContext.result_foracceptsSubWorkflowNodein its type signature. (#81)
Worker
- Finalize retries are tracked as finalizers, so they are no longer cancelled at shutdown (workflow advancement is not dropped). (#35, #52)
RUNNINGtasks are requeued after aBrokenProcessPoolcrash. (#37)- Concurrent executor restarts are serialized (no leaked pools). (#49)
- Nonrunnable-task cleanup is guarded by worker ownership. (#51)
Broker, listener & scheduler
- The racing
add_readeron the dispatcher fd is dropped, and the dispatcher is paused during a health-disconnect reset. (#45, #47) - Unverifiable enqueue conflicts fail instead of assuming idempotent success. (#48)
- Normal-branch schedule advancement is anchored to slot time, eliminating interval drift on late ticks. (#46)
Persistence
- ORM
datetimedefaults are evaluated per row instead of being captured at import, soupdated_atadvances on ORM updates. (#23)
Models, codec & app
TaskError.model_dump(mode='json')flattens a live exception instead of raising. (#72)SubWorkflowErrorsubtype and fields are preserved through error round-trip. (#59)- Per-task and global
exception_mapperreserved-code collisions are detected (the check matchesMapping, not justdict). (#85) - Workflow package
__all__drift fixed; the init log no longer asserts a role. (#83)
CLI & utils
- The CLI installs signal handlers before schema init and closes brokers on a startup failure. (#41, #43)
- Tasks may close over helper functions without a false
TASK_PREDECORATED_NOT_SUPPORTEDrejection. (#40) - The docs tarball fetch has a download timeout and a verified prefix guard. (#74)
LoopRunner.stop()no longer leaves the runner half-stopped, and retry jitter is floored before the spread, then applied upward (no lower-half collapse). (#78)
Internal
Section titled “Internal”- Dead-code removal, regression-test additions, fixture migrations to the strict envelope, and review follow-up tightenings. (#32, #34, #36, #38, #39, #42, #67, #77, #87)
