Migrating to 0.7.0
Upgrade order
Section titled “Upgrade order”- Install horsies 0.7.0 with psycopg 3.3.7 or later. See Dependencies.
- Change the config: worker profiles, child pools, the running stale threshold and the snapshot interval.
- Change the code: matches on error codes, result waits inside a task and workflow limits.
- Run
horsies check <app>, andhorsies check <app> --liveagainst the target database. See New validation. - Stop every 0.6.x worker, then start the 0.7.0 processes. See Stop 0.6.x workers first.
- Update the start commands and the supervisor for the exit codes.
- Read After the upgrade for the changes in behaviour.
Dependencies
Section titled “Dependencies”horsies 0.7.0 requires psycopg 3.3.7 or later. The pg-binary and pg-c extras require psycopg[binary] or psycopg[c] 3.3.7 or later, which install the C build of the same version as psycopg. Before 3.3.6, psycopg waited with no limit for an interrupted query to end. 3.3.7 also sends the cancel request when the cancel connection is not ready at once. The 5-second limit on the cancel request also needs libpq 17 or later. See psycopg and libpq.
Worker settings move into profiles
Section titled “Worker settings move into profiles”horsies worker no longer takes execution flags. The settings live in named worker profiles in AppConfig.worker_profiles, and a worker starts with --profile <name>. --loglevel and the app locator stay.
- Add one profile for each set of flags that your deployment starts workers with.
- Replace the flags on each
horsies workercommand with--profile <name>. - Run
horsies check <app>, andhorsies check <app> --liveagainst the target database.
--profile is required. There is no default profile. A removed flag exits with code 2 and a message that names the field that replaces it:
horsies worker: error: --processes was removed in 0.7.0: worker settings live in a worker profile. Set WorkerProfile(isolated=IsolatedChildren(count=N)) in AppConfig.worker_profiles, and start the worker with --profile <name>.Flag to profile field
Section titled “Flag to profile field”| 0.6.x flag | Default | 0.7.0 profile field |
|---|---|---|
--processes N |
1 |
isolated=IsolatedChildren(count=N) (required, no default) |
--max-tasks-per-child N |
100 |
isolated=IsolatedChildren(count=..., recycle=IsolatedRecycle(after=TaskLimit(tasks=N))) |
--max-tasks-per-child 0 |
recycle=IsolatedRecycle(after=NoTaskLimit()) |
|
--max-memory-per-child-mb N |
off | recycle=IsolatedRecycle(above=MemoryLimit(mb=N)) |
no --max-memory-per-child-mb |
recycle=IsolatedRecycle(above=NoMemoryLimit()) (the default) |
|
--max-claim-batch N |
0 |
claim=ClaimLimits(per_queue_per_pass=TaskLimit(tasks=N)) |
--max-claim-batch 0 |
claim=ClaimLimits(per_queue_per_pass=NoTaskLimit()) (the default) |
|
--max-claim-per-worker N |
0 |
claim=ClaimLimits(claimed_per_worker=TaskLimit(tasks=N)) |
--max-claim-per-worker 0 |
claim=ClaimLimits(claimed_per_worker=NoTaskLimit()) (the default) |
|
--drain-timeout SECONDS |
300 |
drain_timeout=timedelta(seconds=SECONDS) |
--child-preload library|app|none |
library |
preload=ChildPreload.LIBRARY, ChildPreload.APP or ChildPreload.NONE |
The defaults of the profile fields are the defaults of the old flags, except the number of children: IsolatedChildren.count has no default.
Example
Section titled “Example”A 0.6.x command:
horsies worker myapp.instance:app --processes=8 --max-tasks-per-child=500 \ --max-memory-per-child-mb=1024 --drain-timeout=120 --child-preload=appThe 0.7.0 profile and command:
from datetime import timedelta
from pydantic import SecretStr
from horsies import ( AppConfig, ChildPreload, Horsies, IsolatedChildren, IsolatedRecycle, MemoryLimit, PostgresConfig, TaskLimit, WorkerProfile,)
app = Horsies(AppConfig( broker=PostgresConfig( database_url=SecretStr('postgresql+psycopg://user:pass@localhost:5432/mydb'), ), worker_profiles={ 'default': WorkerProfile( isolated=IsolatedChildren( count=8, recycle=IsolatedRecycle( after=TaskLimit(tasks=500), above=MemoryLimit(mb=1024), ), ), drain_timeout=timedelta(seconds=120), preload=ChildPreload.APP, ), },))horsies worker myapp.instance:app --profile defaultChild pools move into profiles
Section titled “Child pools move into profiles”PostgresConfig.worker_child_pool_min_size, worker_child_pool_max_size and worker_child_pool_check are removed. In 0.6.x one value applied to every child. In 0.7.0 each child kind has its own pool in the worker profile.
0.6.x PostgresConfig field |
Default | 0.7.0 profile field |
|---|---|---|
worker_child_pool_min_size=N |
0 |
IsolatedChildren(pool=IsolatedPool(min_size=N)), SharedAsyncChildren(pool=SharedPool(min_size=N)), SharedThreadChildren(pool=SharedPool(min_size=N)) |
worker_child_pool_max_size=N |
2 |
IsolatedChildren(pool=IsolatedPool(max_size=PoolSize(connections=N))), SharedAsyncChildren(pool=SharedPool(max_size=PoolSize(connections=N))), SharedThreadChildren(pool=SharedPool(max_size=PoolSize(connections=N))) |
worker_child_pool_check=False |
True |
IsolatedChildren(pool=IsolatedPool(check=False)), SharedAsyncChildren(pool=SharedPool(check=False)), SharedThreadChildren(pool=SharedPool(check=False)) |
An isolated child opens 0 to 1 connections by default (0.6.x: 0 to 2). It runs one task at a time, and its two writes run one after the other, so it never uses a second connection. The checkout check is off by default in 0.7.0 (it was on in 0.6.x): each connection of a child pool is in autocommit, and each write of a child is one statement, so the check doubled its round trips. Set check=True on the pool of a kind to keep it. A shared child derives its pool size from its seats by default (PoolSizeFromSeats()); see Sizes Derived from the Seats.
The worker process no longer pre-pings its pooled connections by default. PostgresConfig.worker_pool_pre_ping (default False) sets it; pool_pre_ping applies to the other engines only. Set worker_pool_pre_ping=True to keep the 0.6.x behavior. See Health Checks at High RTT.
pool_pre_ping and worker_pool_pre_ping ping a pooled connection only when it waited in the pool for more than 1 s (PRE_PING_AFTER_IDLE_S); 0.6.x pinged at every checkout. A connection that dies within 1 s of its check-in fails its next statement with a retryable connection error (TaskSendEnqueueFailed(retryable=True) for a send): retry it with retry_send, or set resend_on_transient_err=True.
A config that still sets a removed field fails when it is built, with every such field in one report:
error[HRS-213]: worker_child_pool_max_size was removed in 0.7.0 = note: each child kind of a worker profile has its own connection pool
= help: set the pool of each child kind in AppConfig.worker_profiles: IsolatedChildren(pool=IsolatedPool(max_size=PoolSize(connections=N))), SharedAsyncChildren(pool=SharedPool(max_size=PoolSize(connections=N))) or SharedThreadChildren(pool=SharedPool(max_size=PoolSize(connections=N)))A 0.6.x config with worker_child_pool_max_size=1 and worker_child_pool_check=False, and the 0.7.0 profile. Both values are the 0.7.0 defaults of IsolatedPool; the profile states them:
from horsies import IsolatedPool, PoolSize
single_connection_profile = WorkerProfile( isolated=IsolatedChildren( count=8, pool=IsolatedPool(max_size=PoolSize(connections=1), check=False), ),)The worker pool moves into profiles
Section titled “The worker pool moves into profiles”PostgresConfig.worker_pool_size and worker_max_overflow are removed. In 0.6.x they sized the pool of the worker process (3 + 2 by default), and every worker child copied them for the broker of its task code. In 0.7.0 two profile fields replace them:
0.6.x PostgresConfig field |
Default | 0.7.0 profile field |
|---|---|---|
worker_pool_size=N, worker_max_overflow=M (worker process) |
3, 2 |
WorkerProfile(pool=ParentPool(max_size=PoolSize(connections=N + M))); the default PoolSizeFromSeats() derives the size from the seats of the profile, 5 to 20 connections |
worker_pool_size=N, worker_max_overflow=M (task code that sends, in a child) |
3, 2 |
task_send_pool=PoolSize(connections=N + M) on IsolatedChildren, SharedAsyncChildren or SharedThreadChildren; default 2 |
The pool of the worker process has no overflow: it holds at most max_size connections, and the connection check counts it exactly. The task-send pool of each child has no overflow either. The producer pool (pool_size, max_overflow) is unchanged and does not apply in a worker child. In the worker process, app.get_broker() opens the producer pool; the worker process does its own work on WorkerProfile.pool.
A config that still sets a removed field fails when it is built, with every such field in one report:
error[HRS-213]: worker_pool_size was removed in 0.7.0 = note: the worker process opens its pool with the resolved pool.max_size of its worker profile = note: task code in a worker child that sends opens the task_send_pool of its child kind, not this pool
= help: set the pool of the worker process in AppConfig.worker_profiles: WorkerProfile(pool=ParentPool(max_size=PoolSize(connections=N))); the default, PoolSizeFromSeats(), derives it from the seats of the profileA 0.6.x config with worker_pool_size=8 and worker_max_overflow=2, and the 0.7.0 profile:
from horsies import ParentPool
ten_connection_profile = WorkerProfile( isolated=IsolatedChildren(count=8), pool=ParentPool(max_size=PoolSize(connections=10)),)running_stale_threshold_ms holds one heartbeat interval plus 2 s
Section titled “running_stale_threshold_ms holds one heartbeat interval plus 2 s”With auto_fail_stale_running=True (the default), a 0.7.0 worker stops when
no runner heartbeat lands for half of RecoveryConfig.running_stale_threshold_ms.
That half must hold one runner_heartbeat_interval_ms plus 2 s for the next
heartbeat to land. So the threshold is at least
2 × (runner_heartbeat_interval_ms + 2,000 ms): 64,000 ms at the default
30,000 ms interval. 0.6.x accepted 2 × runner_heartbeat_interval_ms. A
smaller value raises ConfigurationError with HRS-204 when the config is
built, and names the smallest valid value. With
auto_fail_stale_running=False, the 0.6.x bound stays. See
When the Worker Cannot Write Heartbeats.
worker_state_snapshot_interval_ms is at most 60,000 ms
Section titled “worker_state_snapshot_interval_ms is at most 60,000 ms”0.6.x accepted RecoveryConfig.worker_state_snapshot_interval_ms up to
300,000 ms. In 0.7.0 the cap is 60,000 ms, half of the 120 s window in which a
worker reads as live. A larger value raises ConfigurationError with
HRS-204 when the config is built. Set the interval to 60,000 ms or less. See
Recovery Config.
Matches on error codes
Section titled “Matches on error codes”An exhaustive match on these types needs a change:
| Type | Change |
|---|---|
ErrorCode |
Remove TASK_ASYNC_FUNCTION_NOT_SUPPORTED (HRS-106). @app.task() accepts an async def function, and the number is retired |
OperationalErrorCode |
Add SUBWORKFLOW_DEPTH_LIMIT_REACHED, TASK_BLOCKED_LOOP and NEIGHBOUR_BLOCKED_LOOP |
RetrievalCode |
Add WAIT_INSIDE_TASK |
BrokerErrorCode |
Add WAIT_INSIDE_TASK |
BrokerOperationError |
Add BrokerWaitInsideTask |
The new code strings are reserved. An exception_mapper or a default_unhandled_error_code with one of these values fails horsies check with HRS-212. A 0.6.x process cannot decode an error stored with one of the new OperationalErrorCode values, so upgrade every process that reads task results.
New fields without a default: WorkerStateSnapshot.slot_health, WorkerStateSnapshot.class_capacity and the monitoring LeafTaskInfo.execution. Code that builds one of these types, for example a test fake, must pass them. See Worker & Database Health.
A result wait inside a task is refused
Section titled “A result wait inside a task is refused”In a worker child, a wait for the result of another task or workflow returns an error at once, and waits for nothing:
| Call | Returns |
|---|---|
TaskHandle.get() and get_async() on a sent task, app.get_result() and get_result_async(), WorkflowHandle.get() and get_async() |
RetrievalCode.WAIT_INSIDE_TASK |
app.get_raw_result(), broker.get_raw_result_record() and their *_async forms |
Err(BrokerWaitInsideTask) |
The refusal also covers the threads that a task starts and the on_child_process_start hooks. send(), a workflow start, TaskHandle.info() and workflow_ctx.result_for() still work. Compose the steps as a workflow. See Waiting for a Result Inside a Task.
Workflow limits
Section titled “Workflow limits”- A child workflow has at most depth 32. A
SubWorkflowNodewhose child would have depth 33 fails withSUBWORKFLOW_DEPTH_LIMIT_REACHED, and theon_errorpolicy of its workflow applies. - A
WorkflowSpecholds at most 10,000 nodes. A larger spec raisesHRS-038when it is built. Split the nodes intoSubWorkflowNodes: each child workflow has its own limit. horsies checkreportsHRS-036for a nesting deeper than 32, andHRS-037for a definition that includes itself with the same kwargs, when noSubWorkflowNodeon the path haswaits_for.
See Limits.
New validation
Section titled “New validation”These were not checked before 0.7.0, because the values were command-line flags:
horsies checkvalidates every profile. A claim limit above the capacity of the worker (its seats plusprefetch_buffer; one seat per isolated child) is refused withHRS-218; before, it had no effect.horsies check --livereportsHRS-219when one worker of a profile opens more Postgres connections itself than the server has usable slots. The worker process counts the resolvedpool.max_sizeof the profile, and each child the resolvedpool.max_sizeof its mode. The error also states what task code that sends adds: thetask_send_poolof each child kind, per child and per event loop that it sends from.horsies checkrefuses a child pool whosemin_sizeis above its resolvedmax_sizewithHRS-218. In 0.6.x the same bound onworker_child_pool_min_sizeraisedHRS-213.- At boot the worker warns when the per-child memory limits add up to more than the memory that the worker can use: the container (cgroup) limit when it is below the host RAM, else the host RAM, and when one worker opens more connections itself than the server has usable slots.
Each bound of the old flags still applies: at least one child, a task limit of at least 1, a memory limit of at least 1 MB, and a drain timeout above 0. New upper bounds: 1024 isolated children, 2,147,483,647 tasks for a task limit, 16,777,216 MB for a memory limit, 24 hours for the drain timeout, and 1000 connections for a child pool, a task-send pool and the pool of the worker process.
See Worker Profiles for every field and for the validation table.
Stop 0.6.x workers first
Section titled “Stop 0.6.x workers first”The first 0.7.0 process that starts migrates the schema from v52 to v58:
- v53 adds
horsies_worker_states.slots_down. - v54 adds
horsies_tasks.execution, and buildsidx_horsies_tasks_claim_pending_classconcurrently. - v55 replaces the claim function with one that takes a budget per execution class, and drops the earlier one. It adds
horsies_worker_states.class_capacity, buildsidx_horsies_tasks_claim_expired_classconcurrently, and dropsidx_horsies_tasks_claim_pendingandidx_horsies_tasks_claim_expired_orderedconcurrently after the version row commits. An upgrade that fails before that commit leaves the earlier schema version and both indexes, so the earlier claim function keeps the indexes it reads. - v56 adds
horsies_finish_failed_attempt, which writes a failed attempt of a RUNNING task (the attempt row, then the retry or the terminal failure) in one statement, its row typehorsies_failed_attempt_outcome, andhorsies_lock_running_task, the locking read of a RUNNING task. Both take the task advisory lock before the row lock. No table changes. - v57 adds the function
horsies_complete_tasks_fused, which writes a batch of plain successes in one statement. - v58 adds
horsies_finish_workflow_success, which writes the success of a RUNNING workflow task (the attempt row, the move to the history, the phase-2 record of its node) in one statement, with the task advisory lock before the row lock, and its row typehorsies_workflow_success_outcome. It dropshorsies_lock_running_task, which has no caller left. No table changes.
A 0.6.x worker that still runs fails each claim pass and exits. Stop every 0.6.x worker first, then start the 0.7.0 workers.
A 0.6.x process refuses a schema newer than its own when it sets up its schema: database schema v58 is newer than this build supports (v52). Upgrade the producers, the schedulers and the web UI too. Rows that a 0.6.x process writes read ISOLATED.
Exit codes
Section titled “Exit codes”| Code | Process | Cause | In 0.6.x |
|---|---|---|---|
| 1 | horsies worker |
--profile is missing, or names no profile |
No such option |
| 2 | horsies worker |
A removed worker flag | The flag was valid |
| 1 | horsies worker |
The worker cannot write runner heartbeats, with auto_fail_stale_running=True |
The reaper failed the running tasks |
| 77 | horsies worker |
As above, and the worker has not stopped 120 seconds later | No such code |
| 74 | worker child | The child stopped while threads of the executor of its task loop still ran | Code 74 meant a forked child that does not reparent to the worker |
| 75 | worker child | The child failed its boot, after it reported the cause | No such code |
| 76 | worker child | A forked child that does not reparent to the worker | Code 74 |
See Exit Codes.
After the upgrade
Section titled “After the upgrade”Scheduled runs apply the options of their task
Section titled “Scheduled runs apply the options of their task”A scheduled run stores the options of its task: timeout_ms, retry_policy, execution and cancel_grace_ms. In 0.6.x a scheduled run had no timeout and no retries, and ran as ISOLATED. A scheduled task with timeout_ms now stops at its timeout, and a failure retries by its policy.
The enqueue digest of every slot changes. A slot that a 0.6.x scheduler enqueued and did not record gets PAYLOAD_MISMATCH when 0.7.0 fires it. This happens when the 0.6.x scheduler stopped before it saved the schedule state. The scheduler then counts the slot as fired: it logs one warning, records the run with the task id of the slot, and moves to the next slot. That run keeps its 0.6.x row: no timeout, no retries, ISOLATED. See Task Options.
A keyed send that a 0.6.x process sent is a conflict
Section titled “A keyed send that a 0.6.x process sent is a conflict”The stored task options have two more keys, execution and cancel_grace_ms, so the command of every send changes. A send with an idempotency_key that a 0.6.x process already sent returns IDEMPOTENCY_KEY_CONFLICT, not the task of the 0.6.x send. A send after the reservation of the key ends is accepted.
Enqueue refuses task options that are not an object
Section titled “Enqueue refuses task options that are not an object”enqueue_async refuses with INVALID_TASK_OPTIONS:
- task options that are not a JSON object;
- an
executionthat is not anExecutionvalue; - a
cancel_grace_msthat does not fit the class: from 100 to 10,000 ms forSHARED_ASYNC, absent or null for the other classes.
Sends of an @app.task function write valid options. This affects only code that passes its own options text to the broker.
A child that cannot start does not always stop the worker
Section titled “A child that cannot start does not always stop the worker”A child that fails its start for a cause that can clear takes only its slot down: EMFILE, ENFILE, ENOMEM or EAGAIN in a start step (the spawn and the event loop among them), or a thread that the OS refuses. The worker runs on degraded, and starts a new child for the slot after a growing wait. In 0.6.x each such failure counted one of three attempts, and the third stopped the worker. A worker with every slot down still stops after three failed starts in a row. Read WorkerStateSnapshot.slot_health to see the slots that are down. See Child Start Failures.
The worker process writes the runner heartbeats
Section titled “The worker process writes the runner heartbeats”One thread of the worker writes the runner heartbeats of its RUNNING tasks, over one connection of its own. A worker opens one more Postgres connection than in 0.6.x. A task that holds the GIL in C code no longer stops its heartbeats. See When the Worker Cannot Write Heartbeats.
A timed-out task moves after its child has exited
Section titled “A timed-out task moves after its child has exited”The worker writes TASK_TIMEOUT, the retry and the next workflow step only after the child of the task has exited. So a retry never runs next to the attempt that it replaces. When the child does not exit within 30 seconds of the kill, the worker writes nothing and logs at critical level, and stale-task recovery takes the task. With stale RUNNING recovery off (no RecoveryConfig, or auto_fail_stale_running=False), nothing else would move the row, so the worker waits for the exit and writes the row once it sees it; a stop of the worker ends that wait.
Execution modes are new
Section titled “Execution modes are new”A profile can start shared async children and shared thread children:
shared_async=SharedAsyncChildren(count=N, seats=K): each child runs up to KExecution.SHARED_ASYNCtasks at once, as coroutines.shared_thread=SharedThreadChildren(count=N, seats=K): each child runs up to KExecution.SHARED_THREADtasks at once, on threads.
They are new in 0.7.0 and have no flag before them. A profile without them runs every task in isolated children, as 0.6.x did, also a task marked SHARED_ASYNC or SHARED_THREAD. See Choosing an Execution Mode.
