Migrating to 0.6.0
What 0.6.0 changes
Section titled “What 0.6.0 changes”For an operator, 0.6.0 changes four things:
- Workers run each task in a child process that the worker owns. There is no process pool.
- SIGTERM drain has a bound:
--drain-timeout(default 300 seconds). - A workflow whose recovery raises on every reaper pass now fails after
workflow_recovery_failure_limitpasses. - Schema v52 adds one empty table.
For a developer, the type contract is strict at every boundary. See topic 13.
Many error types, return values and codes changed. See Required code changes.
Upgrade rule
Section titled “Upgrade rule”Do not run a rolling upgrade from 0.5.x to 0.6.0. Stop every process, upgrade all of them, then start them.
- Stop every worker, scheduler, producer and
horsies webprocess. - Upgrade the
horsiespackage in all of them. - Start the processes.
The reasons:
- A 0.5.5 broker checks the schema version once, at its first use. A 0.5.5 process that already runs does not stop on a v52 database, so it continues to write beside 0.6.0.
- A 0.5.5 process cannot decode a result that 0.6.0 stored with
WORKFLOW_OUTPUT_SKIPPEDorWORKFLOW_RECOVERY_LIMIT_REACHED. The read fails withUnknown built-in task code. - 0.6.0 cannot decode an error that a 0.5.5 process stores without an error code.
Schema step
Section titled “Schema step”The schema goes from v51 to v52. The change is one new table, horsies_workflow_recovery_failures. The table starts empty. There is no data step.
Any process that uses a PostgresBroker applies the migration the first time it calls an entry point that initializes the schema:
horsies workerat start.horsies schedulerat start.- The first workflow start (
spec.start(),start_workflow()). - The first send or schedule (
enqueue_async). - The first
get_task_infoorget_raw_resultread. ensure_schema_initialized()on the broker.
horsies web does not apply the migration. Until another process applies it, /api/meta reports schema_state: MISMATCH and the web actions are off. A broker built with run_schema_migrations=False or assume_initialized=True does not apply it either.
The migration is safe to run from many processes at once. A PostgreSQL advisory lock guards the DDL. See Database Schema.
After v52 is applied, a 0.5.5 horsies worker or horsies scheduler refuses to start on that database, with database schema v52 is newer than this build supports (v51). Decide your rollback before you upgrade.
New settings and flags
Section titled “New settings and flags”In 0.7.0 the worker flags below move into worker profiles. See Migrating to 0.7.0.
| Name | Default | Where | Page |
|---|---|---|---|
workflow_recovery_failure_limit |
25 (3-1,000) |
RecoveryConfig |
Recovery Config |
--drain-timeout SECONDS |
300 (finite, > 0) |
horsies worker |
Graceful Shutdown |
--child-preload MODE |
library (library, app, none) |
horsies worker |
Preload Parent |
At the default check_interval_ms and the default limit, the recovery limit is reached 24 intervals after the first pass, which is about 720 seconds. A workflow that is in a failing state when you upgrade reaches the limit and fails. Set a higher workflow_recovery_failure_limit to get more time for a fix.
Set --drain-timeout below the stop window of your supervisor. See CLI Reference.
What an operator sees
Section titled “What an operator sees”- SIGTERM drain. The worker stops claims and returns tasks that wait for a child to
PENDING. It waits up to--drain-timeoutseconds for running tasks. Then it kills the children that still run a task. Each of these tasks takes the child crash path and getsWORKER_CRASHED, or a retry whenWORKER_CRASHEDis inauto_retry_for. - Child loss. When a child dies, the worker starts a new child in that slot. Other children continue. A task in
CLAIMEDreturns toPENDING. A task inRUNNINGtakes the retry policy forWORKER_CRASHED. - Refusals at worker start. The worker exits non-zero in these cases:
- The app locator is
__main__:app. --child-preload appis set and the import leaves a Python thread or a socket.--max-memory-per-child-mbis at or below the warmed child baseline.- A child fails to start three times, or the operating system refuses to start a process.
- A child hook fails.
- The app locator is
- New code on a running worker. A recycled child is a new fork of the preload parent, so it keeps the library code that the parent imported. With
--child-preload=appit also keeps your app code. To load new code, restart the worker. - Process names. On Linux, a forked child shows the command line of the preload parent in
ps. A monitor that countsmultiprocessing-forkprocesses needs a change. - Memory.
children_memory_mbcounts shared pages once for each child. It reads above real use. Read the container or cgroup total.
See Worker Architecture for the process model.
Required code changes
Section titled “Required code changes”Each topic below lists what to change and how to find it. The changelog has the full text of each change.
Run horsies check first. The type contract refuses more shapes at definition, so a task or a model can stop registering. See topic 13.
- Worker process model.
- Rename
ExecutorRestartFailedErrortoChildStartFailedError. There is no alias.WarmupIncompleteErrorandMemoryRecycleUnsupportedErrorare removed, and the worker no longer raises them. - Set libraries that read a setting at import, such as
PSYCOPG_IMPL, in the process environment, not in a.envfile that your app loads. - Find uses of
multiprocessing.parent_process()in task code. It returnsNonein a child.
- Rename
- Workflow recovery limit.
- Add a
WORKFLOW_RECOVERY_LIMIT_REACHEDcase to each exhaustivematchonOperationalErrorCode. - The code string is reserved. An
exception_mapperordefault_unhandled_error_codewith that value failshorsies checkwithHRS-212.
- Add a
- Schedules.
- Run
horsies check. A schedule whose kwargs exceedpayload.reject_bytesfails at scheduler start withHRS-205. - A kwargs field with a
default_factory, or a validator that reads the clock, runs once for each scheduler start and not once for each fire.
- Run
- Sync calls and the event loop.
- Replace each sync call from a coroutine or a Jupyter cell with the
*_asyncform. A sync call in a running event loop returnsASYNC_CONTEXT. - Add an
ASYNC_CONTEXTcase to each exhaustivematchonBrokerErrorCode,WorkflowStartErrorCode,HandleErrorCodeandRetrievalCode.
- Replace each sync call from a coroutine or a Jupyter cell with the
- Send, enqueue and broker errors.
TaskSendError,BrokerOperationErrorandBrokerEnqueueErrorare unions of frozen variants. Match on the variant before you readexception,task_idorpayload. Pyright reports each site.- Replace
isinstance(err, TaskSendError)with amatchon the variants. ReplaceTaskSendError(code=...)in tests with a variant constructor. - Add cases for
LOOP_RUNNER_FAILED,INVALID_INPUTandLISTENER_DISABLEDwhere you match on codes. TaskSendValidationFailedis not retryable. Do not pass it to the retry methods.- Rename the environment variable
TASKLIB_SUPPRESS_SENDStoHORSIES_SUPPRESS_SENDS. - A failed broker construction raises
HRS-211from each send method. A defect in the enqueue write propagates.
- Workflow start.
WorkflowStartErroris a union of frozen variants. Match on the variant before you readexception.- Add cases for
WORKFLOW_ID_CONFLICTandLOOP_RUNNER_FAILED. RemoveWorkflowStartErrorCode.INTERNAL_FAILED. retry_start()takesWorkflowStartRetryableonly.- A task node with a
priorityoutside 1-100 raisesHRS-034. A naivegood_untilraisesHRS-035. Runhorsies check. - Add
WORKFLOW_INVALID_PRIORITY,WORKFLOW_NAIVE_GOOD_UNTIL,WORKFLOW_INVALID_DEFINITION_CONFIG,TASK_ASYNC_FUNCTION_NOT_SUPPORTEDandCHECK_LIVE_ASYNC_CONTEXTto each exhaustivematchonErrorCode.
- Workflow handle and result reads.
- Add an
INVALID_INPUTcase to each exhaustivematchonRetrievalCode. Aget()with a negativetimeout_ms, a bool, a value that is not an int, or a value aboveMAX_TASK_TIMEOUT_MSreturns it. A negative value no longer works like0. HandleOperationErroris a union of frozen variants. RemoveHandleErrorCode.INTERNAL_FAILED. Add cases forSTORED_DATA_INVALIDandRESUME_RECOVERY_FAILED.results()andtasks()returnHandleWorkflowNotFoundfor an unknown workflow id.get()returnsWORKFLOW_NOT_FOUNDwhen retention removed the row, andRESULT_DESERIALIZATION_ERRORfor stored data that does not parse.get()returnsWORKFLOW_OUTPUT_SKIPPEDfor a completed workflow whose output node is skipped. Code that read this state throughRESULT_DESERIALIZATION_ERRORmust match the new code.- Add a
WORKFLOW_OUTPUT_SKIPPEDcase to each exhaustivematchonOutcomeCode. The code string is reserved: anexception_mapperordefault_unhandled_error_codewith that value failshorsies checkwithHRS-212.
- Add an
- Subworkflow node outcome.
- A
SubWorkflowNodeisFAILEDwhen its child isCOMPLETEDwith an err output, or with a stored result that does not parse. - A dependent with
allow_failed_deps=Falseis nowSKIPPED. Keep the node out of the success cases of the parent, or setallow_failed_deps=Trueon the dependent.
- A
- Codec and JSON boundary.
encode_valueanddecode_valueraiseValueValidationError, not pydanticValidationError. Changeexcept ValidationErroraround these calls.- Strings with a lone surrogate, self-referencing values, dict keys without a JSON form, non-UTF-8
bytesand nesting above 300 levels are refused. The refusal is aTaskSendValidationFailedfrom a send,CONFIG_INVALID_SCHEDULEfrom a schedule, andHRS-031from a workflow spec. RETURN_TYPE_MISMATCHno longer stores the returned value inactual_value.
TaskErrorcontract and retired codes.TaskErrorrequireserror_codeand refuses a key that it does not declare. FindTaskError(message=...)calls without a code.- A stored error without a code does not decode. No migration rewrites old rows.
ContractCode.PYDANTIC_HYDRATION_ERRORandErrorCode.WORKFLOW_EXCESS_POSITIONAL_ARGSare removed. UseRESULT_DESERIALIZATION_ERROR,INVALID_JSON_PAYLOADorRETURN_TYPE_MISMATCH, andWORKFLOW_POSITIONAL_ARGS_NOT_SUPPORTED.ErrorCode.WORKFLOW_SUBWORKFLOW_BUILD_WITH_BINDINGis removed too: no code raised it, so remove its case.
- Monitoring, task actions and rerun.
MonitoringQueryErrorandTaskActionErrorare unions of frozen variants. Add cases forHISTORY_CONTRACT_VIOLATIONandSTORED_DATA_INVALID.get_task_detailreturnsTaskDetailNotFound, notOk(None), for an absent task. Replacedetail is Nonechecks.list_tasks,task_breakdownandlist_workflow_runstakeTaskPage,BreakdownLimitandWorkflowRunsLimit. Build each withparse.RerunSourceAbsent.predates_retained_flooris nowreason, aTaskAbsenceReason. Add aRerunSourceCorruptcase to each exhaustivematchonRerunOutcome.
- Definition-time refusals, config and tooling.
CustomQueueConfigrefuses an empty queue name, a name with whitespace at the start or the end, and a name over 100 characters (HRS-200).- Every config model refuses unknown keys. Remove misspelled keys.
@app.task()refusesasync deffunctions (HRS-106) and an empty task name (HRS-102).- A
WorkflowDefinitionsets options withconfig = WorkflowDefinitionConfig(...). A nestedclass MetaraisesHRS-033. - The agent skill files are in
horsies/.agents/skills/horsies/. Install them withhorsies skills install --target DIR. horsies get-docsdownloads the docs of the release tag. Pass--ref mainfor themainbranch.timeout_mstakes 1000 toMAX_TASK_TIMEOUT_MS(4,294,967,295 ms, about 49.7 days).@app.taskrefuses a larger value withHRS-102. A queued row from an earlier version with a larger stored value fails withWORKER_SERIALIZATION_ERRORwhen a worker claims it. Before the upgrade, drain the queues of a task with such a timeout.
- Strict type contract.
- Run
horsies check. A task or a model that uses a refused shape no longer registers, so a worker does not start until it changes. The list is in Shapes Refused at Definition:Field(exclude=True),exclude_if,extra='allow',PlainValidator,WrapValidator,SkipValidation, a schema hook,Json[T], an alias whose written key does not read back,Literalwith a plainEnum, a default that does not hold its type (n: int = None). - A conversion is refused at
.send(), in a schedule and on a return:"7"for anint, a str for adatetime,Enum,UUIDorDecimal. Pass the value of the declared type. Schedule kwargs hold Python values, so write the real value there too. .send()refuses a missing required argument and an unknown key at any depth. A subclass with fields that the declared type lacks is refused.- Each
build_withthat aSubWorkflowNodetargets follows the task rules:Anyand baredictparameters are refused, and its hints must resolve.WorkflowDefinition[Any]cannot feedargs_fromor be an output. - A return that is not a
TaskResult, includingNone, givesRETURN_TYPE_MISMATCH, notUNHANDLED_EXCEPTIONorTASK_EXCEPTION. A policy withauto_retry_for=['TASK_EXCEPTION']no longer retries a forgottenreturn. - Each key of
kwargsandargs_fromon a node binds to one named, typed parameter. A key that reached**paramsof abuild_withis refused when the spec is built. Name the parameter, or declareparams: dict[str, JsonValue]. A row from an earlier version with such a key fails when the parent reaches the node. TaskError.datamust be JSON-native. ADecimal, adatetimeor a tuple becomes{'data_refused': <cause>}. Convert withstr(),.isoformat()orlist().- Stored rows are read strictly. A result that an earlier version stored with a value outside a return constraint, with a serializer that changes the JSON type, or deeper than 200 levels, reads as
RESULT_DESERIALIZATION_ERRORuntil retention removes it (30 days by default). A queued task of that shape fails withWORKER_SERIALIZATION_ERROR. Before the upgrade, drain the queues of a task with such a type.
- Run
