Deploying Code Changes
Deploying Code Changes
Section titled “Deploying Code Changes”A task row stores the task name and the encoded kwargs. It does not store the code. The worker that claims the row resolves the task name and decodes the kwargs against the code that it has loaded at that time. The same rule holds for old and new workers that run side by side during a deploy.
Why and When
Section titled “Why and When”Read this page before a deploy that changes any of these:
- a task name
- a task parameter (added, removed, renamed or retyped)
- the return type of a task
- a field of a model or dataclass that a task takes or returns (added, removed, renamed or retyped)
- a workflow definition or a
build_withsignature
Rows that exist before the deploy are not changed by it. They run with the code that is loaded when they are claimed.
Which Code Reads Which Stored Data
Section titled “Which Code Reads Which Stored Data”| Stored data | Read against | When |
|---|---|---|
| Task name of a task row | The task registry of the claiming worker | At claim and execution |
| Kwargs of a task row | The current signature of the task | At execution |
| Workflow DAG (nodes, edges, node kwargs) | Nothing. The DAG is stored when the workflow starts and is not rebuilt | - |
| Subworkflow definition | The workflow registry, by definition_key. The child is built from the current code |
When the subworkflow node becomes ready |
| Stored result of an upstream task | The current return type of the upstream task | When a later node, a workflow context or a handle reads it |
Failure Modes
Section titled “Failure Modes”| Change after the row was stored | Error code | Where it appears |
|---|---|---|
| A parameter was removed or renamed. The row has a kwarg that the signature does not declare | WORKER_SERIALIZATION_ERROR |
The result of the task. The task function does not run |
| A parameter changed type, and the stored value does not fit the new type | WORKER_SERIALIZATION_ERROR |
The result of the task. The task function does not run |
| A required parameter was added. The row has no value for it | UNHANDLED_EXCEPTION |
The result of the task. Python raises TypeError at the call. This is the default default_unhandled_error_code |
| The task name is not registered in the worker | WORKER_RESOLUTION_ERROR |
The result of the task. The message names the task |
A subworkflow definition_key is not registered |
SUBWORKFLOW_LOAD_FAILED |
The subworkflow node ends FAILED, and failure handling of the parent runs |
The kwargs of a subworkflow node do not fit the current build_with signature |
WORKER_SERIALIZATION_ERROR |
The subworkflow node ends FAILED |
| The return type of an upstream task changed, and a stored result does not fit | RESULT_DESERIALIZATION_ERROR |
A TaskResult with this error replaces the stored result where it is read: downstream nodes, WorkflowContext and handles |
| An upstream task name was removed or renamed | RESULT_DESERIALIZATION_ERROR |
Same place. The error data has stage: 'source_not_registered' |
| Change to a model field after the data was stored | Result on read |
|---|---|
| A required field was added | Error. Field missing |
| A field changed type, and the stored value does not fit | Error |
| A field was added with a default | Reads. The default is filled in |
| A field was removed | Reads. The stored value is ignored |
| A field was renamed, and the new name has a default | Reads. The stored value is lost, with no error |
| A field was renamed, and the new name is required | Error. Field missing |
The error is RESULT_DESERIALIZATION_ERROR when a result is read, and WORKER_SERIALIZATION_ERROR when a queued task is decoded. A stored value is read strictly: an int reads as a float, a float does not read as an int, and a str does not read as an int.
An old worker follows the same rules. During a rolling deploy, a new producer can send a row that an old worker claims. An added kwarg that the old signature does not declare gives WORKER_SERIALIZATION_ERROR on the old worker.
How To
Section titled “How To”Add a Parameter
Section titled “Add a Parameter”- Add the parameter with a default value. Deploy all workers.
- Deploy the producers that send the new kwarg.
A default value lets old rows decode and run. A required parameter fails every old row with UNHANDLED_EXCEPTION.
from horsies import TaskError, TaskResult
from myapp.instance import app
@app.task('send_email')def send_email( *, to: str, subject: str, locale: str = 'en',) -> TaskResult[None, TaskError]: return TaskResult(ok=None)Remove or Rename a Parameter
Section titled “Remove or Rename a Parameter”- Deploy the producers so that they stop sending the old kwarg.
- Wait until no live row carries it (see Check for Remaining Rows).
- Deploy the workers with the new signature.
A rename is an add and a remove. Add the new parameter with a default, move the producers, drain, then remove the old parameter.
Rename or Remove a Task
Section titled “Rename or Remove a Task”Keep the old name registered until no row uses it. Register two tasks that call the same function body.
from horsies import TaskError, TaskResult
from myapp.instance import app
def _deliver_report(*, report_id: str) -> TaskResult[None, TaskError]: return TaskResult(ok=None)
@app.task('build_report')def build_report(*, report_id: str) -> TaskResult[None, TaskError]: return _deliver_report(report_id=report_id)
# Old name. Remove it after no row with this name remains.@app.task('make_report')def make_report(*, report_id: str) -> TaskResult[None, TaskError]: return _deliver_report(report_id=report_id)- Deploy workers that register both names.
- Deploy producers that send the new name.
- Wait until no row has the old name.
- Remove the old task.
A scheduled task (see Schedule Config) sends by name. Change the schedule to the new name in step 2.
Change a Return Type
Section titled “Change a Return Type”A task result that a workflow reads later is decoded with the type that the code declares at read time. Keep the new type able to decode the old stored shape, or wait until no workflow holds a stored result of the old shape. Add new fields with defaults.
Change a Model
Section titled “Change a Model”Stored results and queued tasks hold the fields of the model at the time they were written. Code that reads them uses the model of the current deploy. Results stay for the retention period, which is 30 days by default.
Unknown keys are refused when a value is sent. They are ignored when stored data is read, so that old rows stay readable after a normal model change.
Add a Field
Section titled “Add a Field”Give the field a default. Deploy all workers, then the producers.
from pydantic import BaseModel
class Car(BaseModel): model: str km: int colour: str = 'unknown'A required field fails every old row. Make it required only after no old row remains.
Rename a Field
Section titled “Rename a Field”Accept both names on read with AliasChoices. New rows are written under the new name, and old rows still read.
from pydantic import AliasChoices, BaseModel, Field
class Car(BaseModel): model: str km: int registration_year: int = Field( validation_alias=AliasChoices('registration_year', 'register_year'), )- Add the alias. Deploy all workers, then the producers.
- Wait until no live row and no result that you still read holds the old name (see Check for Remaining Rows).
- Remove the old name from
AliasChoices.
The new name must be in AliasChoices, or @app.task refuses the model.
Remove a Field
Section titled “Remove a Field”- Give the field a default. Deploy all workers, then the producers.
- Remove the field.
Old rows that hold the field still read. The stored value is ignored.
Change the Type of a Field
Section titled “Change the Type of a Field”Add a new field with the new type and a default, move the code to it, and remove the old field after no old row remains. An in-place change fails every old row that does not fit the new type.
Change a Workflow Definition
Section titled “Change a Workflow Definition”A started workflow keeps its stored DAG. Changing the class does not change a running workflow.
- Give a new shape a new
definition_key, for examplemyapp.pipeline.v2. - Keep the old
definition_keyregistered for as long as a running workflow can start a subworkflow with it. - A subworkflow node that is not ready yet is built from the current code when it becomes ready. If the
build_withsignature changed, its stored kwargs can fail to decode.
retry_start is best-effort idempotent by workflow_id and does not compare the spec. After a deploy, it can return the handle of a workflow that has the old shape. See Workflow API.
Check for Remaining Rows
Section titled “Check for Remaining Rows”The live table holds only live rows. Query it by task name:
SELECT task_name, status, count(*)FROM horsies_tasksWHERE task_name = 'make_report'GROUP BY task_name, status;Workflow nodes that are not enqueued yet are in horsies_workflow_tasks with status PENDING or READY. They read the code when they are enqueued:
SELECT task_name, status, count(*)FROM horsies_workflow_tasksWHERE status IN ('PENDING', 'READY')GROUP BY task_name, status;A PAUSED workflow keeps its unstarted nodes until it resumes or ends.
Worker Shutdown
Section titled “Worker Shutdown”On SIGTERM or SIGINT a worker stops new claims and waits for running tasks. See Worker Architecture. A task that is already running keeps the code that it loaded. A row that is still PENDING is claimed later by any worker, with the code of that worker.
A recycled child loads your app from disk, unless the worker profile sets preload=ChildPreload.APP. With that mode, a recycled child keeps the code that the preload parent imported, so new code needs a worker restart.
Things to Avoid
Section titled “Things to Avoid”Don’t rename a model field and give the new name a default, with no alias. Old rows read with no error, and the stored value is replaced by the default.
Don’t add a required model field in one step. Each old row fails.
Don’t add a required parameter in one step. Old rows have no value for it, and each one fails with UNHANDLED_EXCEPTION.
Don’t remove a task name while a row or a schedule uses it. Each row fails with WORKER_RESOLUTION_ERROR.
Don’t deploy a new producer before the workers when the producer sends a new kwarg. An old worker rejects the kwarg with WORKER_SERIALIZATION_ERROR.
