Skip to content

Serialization

Strict-serde. The wire carries values plus one envelope marker. It does not carry class identity. The receiver’s declared type drives every decode through pydantic.TypeAdapter.

Two consequences:

  • Tasks must declare every parameter and return type. Banned types (Any, object, bare dict/list/tuple, TypeVar, bare BaseModel, TypedDict, bytes, set/frozenset, Callable, and pathlib.PurePath subclasses) are rejected at @app.task registration.
  • JsonValue is the only untyped fence. Use it at task boundary positions or inside BaseModel/@dataclass fields when the payload is genuinely raw JSON. JsonInput holds the same values, for a position that only accepts them. See JsonInput.
Module Purpose
horsies.core.types.json_value JsonValue PEP 695 alias; JsonInput, JsonList, JsonDict
horsies.core.codec.json_value StrictJsonError; strict JSON-native validators; re-exports JsonValue
horsies.core.codec.json_io dumps_json / loads_json — the strict JSON-native stringification boundary (no class tags). loads_json_bytes is the same parse for stored UTF-8 payloads, and an empty payload is an error. dumps_canonical_json is the one canonical form for stored or hashed text. A unit test keeps every other JSON parse out of the library
horsies.core.codec.error_payload flatten_exception; serialize_error_payload for library-emitted error payloads
horsies.core.codec.task_options serialize_task_options — TaskOptions → persisted JSON
horsies.core.codec.signature_check validate_task_signature — rejects banned types at decorator time
horsies.core.codec.kwargs encode_kwargs_read_back(task_fn, kwargs) / decode_kwargs(...) for producer/consumer kwarg binding
horsies.core.codec.typed encode_value / decode_value / encode_task_result / decode_task_result / decode_task_error / validate_task_result_envelope

Re-exports: from horsies import JsonDict, JsonInput, JsonList, JsonValue, StrictJsonError.

Every stored task result is one shape:

{
"__h_task_result__": true,
"ok": ...payload...,
"err": null
}
  • Marker __h_task_result__: true distinguishes a Horsies envelope from any other JSON in the row.
  • Exactly one of ok / err is non-null.
  • The ok slot carries a JSON value matching the task’s declared ok_type. BaseModel/@dataclass instances are stored as their plain Pydantic JSON form (no per-value type tag) — the receiver reconstructs them by calling decode_task_result(envelope, ok_type).
  • The err slot, when populated, carries the JSON form of TaskError, including error_code, message, data, and a flattened exception (a FlattenedException TypedDict, never a live BaseException on the wire).

Outputless-workflow envelopes wrap an additional marker:

{
"__h_task_result__": true,
"__h_outputless_terminals__": true,
"ok": {
"results_by_id": {"<node_id>": <per-node envelope>, ...},
"task_name_by_id": {"<node_id>": "<task_name>", ...}
},
"err": null
}

Each results_by_id entry is itself a __h_task_result__ envelope; the consumer dispatches per-node decode using the task name to look up task_ok_type in the local registry.

The worker calls encode_task_result(result, ok_type) after the user function returns. The codec:

  1. Validates result.ok against ok_type in strict mode (raises ValueValidationError on a wrong value or a serializer failure; the text holds no input value).
  2. Reads the value back with the decode that a reader uses, and refuses a value that a reader cannot read or that loses data. See Strict Values.
  3. Emits the envelope.
  4. Flattens any live BaseException carried on TaskError.exception into a FlattenedException (type, module, message, repr, traceback).

An ok value that fails surfaces as WORKER_SERIALIZATION_ERROR. An error never fails here: see The Error Path.

from horsies.core.codec.typed import encode_task_result
from horsies import TaskResult, TaskError
envelope = encode_task_result(
TaskResult[int, TaskError](ok=42),
ok_type=int,
)
# {"__h_task_result__": True, "ok": 42, "err": None}

The consumer holds the declared ok_type (from the task registry) and calls decode_task_result(envelope, ok_type):

  1. validate_task_result_envelope checks the marker and structural shape.
  2. If err is populated, decode_task_error runs first — no ok_type is required to surface failed tasks, including those whose task name is unknown locally. It refuses a key that the error type does not declare, and an error without error_code. The refusal names the cause.
  3. Otherwise the ok slot is read through its JSON text in strict mode, against ok_type.

Distinct decode-error surfaces:

Code Source When
BrokerErrorCode.INVALID_JSON_PAYLOAD broker/app result read the stored result does not parse as JSON, is not a JSON object, has a malformed envelope, or fails typed slot decode through an app-level API
BrokerErrorCode.NO_TYPE_AVAILABLE app.get_result_async task_name is not in the local task registry, so no ok_type available to decode the ok slot
OperationalErrorCode.RESULT_DESERIALIZATION_ERROR handle / workflow / engine wrappers around typed decode a wrapper catches a strict decode failure and folds it into a TaskResult(err=...)
ContractCode.NO_TYPE_AVAILABLE outputless workflow handle decode per-node lookup failed inside WorkflowHandle.get() for a terminal task name not in the local registry

Values flowing through a task signature must classify as one of:

  • A JSON primitive (None, bool, int, float, str).
  • datetime.datetime / date / time, uuid.UUID, decimal.Decimal. These are written as JSON strings and read from them. A str at input is refused. Datetime caveats: a ZoneInfo-aware datetime decodes with a fixed-offset timezone (same instant, zone semantics lost — DST arithmetic on the consumer differs), and naive datetimes round-trip naive (ambiguous across workers in different timezones — prefer UTC-aware values).
  • JsonValue or JsonInput (at boundary positions only).
  • An Enum subclass.
  • A concrete BaseModel subclass (not bare BaseModel).
  • A @dataclass.
  • list[T], tuple[T, ...], dict[str, T], Literal[...], Annotated[T, ...], T | None, or a discriminated union — recursively classified.
  • TaskResult[T, TaskError] or TaskResult[T, TaskError] | None, as a whole parameter only. The engine fills it through args_from. err_t must be TaskError, and a default must be None.

Anything outside this surface raises SignatureValidationError at @app.task time. The error names the banned type and prints the documented fix.

The same rules apply to each parameter of a build_with that a SubWorkflowNode targets, because its kwargs are stored and read back. See Subworkflows.

from horsies import Horsies, TaskResult, TaskError, JsonValue
from pydantic import BaseModel
app = Horsies(config)
class Order(BaseModel):
id: int
items: list[str]
@app.task("process_order")
def process_order(*, order: Order) -> TaskResult[Order, TaskError]:
return TaskResult(ok=order)

The wire stores the Pydantic JSON form ({"id": 1, "items": ["widget"]}), with no per-value class tag. The worker decodes it back to Order because the registered task declared Order as the parameter type.

For raw JSON payloads where the inner shape is genuinely dynamic:

@app.task("validate_input")
def validate_input(
*,
data: dict[str, JsonValue],
) -> TaskResult[dict[str, JsonValue], TaskError]:
return TaskResult(ok=data)

The codec validates in strict mode at every boundary. A wrong value is refused, not converted.

Value Declared type Result
"7", 7.0, True int Refused
"yes", 1 bool Refused
1700000000, "2026-01-01" datetime Refused
"red" an Enum Refused. Pass the member
(1, 2), {1, 2} list[int] Refused
1 float Accepted, as 1.0
{"n": 1} a BaseModel Accepted, checked field by field
{"n": 1} a dataclass Refused. Pass an instance

A stored value is read through its JSON text, also in strict mode. A datetime reads from its string, and "7" does not read as an int.

Read-back. .send(), a schedule, a workflow definition and a task result decode what they encoded, with the decode that the worker uses. A value that a reader cannot read is refused before it is stored. This finds a value that validation does not see, such as a mutated instance or model_construct.

Lost data. The value and its read-back are compared. These are refused:

  • An unknown key in a dict, at any depth.
  • A subclass instance with fields that the declared type does not have. A subclass with no new field is accepted.
  • A model field that a serializer drops, when its value differs from the default.

Unknown keys on read. A stored key that the model no longer declares is ignored, so old rows stay readable after a model change. See Change a Model.

Omitted parameters. An omitted parameter crosses no wire, so its default is validated before the task body runs, and the function gets the returned value. A list or a dict default is a fresh copy for each run. A model default stays the same object.

Validators. A validator can run more than once on one value, because the wrapper, the encode and the read each run it. Write validators that give the same result when they run again.

@app.task and each SubWorkflowNode refuse a shape that cannot cross the wire, or that cancels the check. Each spelling gets the same answer: Annotated, a Field default, a decorator, a stdlib or a pydantic dataclass, nested or generic.

Shape Why Instead
Field(exclude=True), exclude_if The field is not written, so the reader gets the default Keep the field out of this type
alias, alias_generator or AliasPath with a key that the read does not accept The written key does not read back Set populate_by_name=True or serialize_by_alias=True, or add the field name to AliasChoices
extra='allow' Extra fields have no type Declare each field, or add a dict[str, T] field
PlainValidator, WrapValidator, mode='plain', mode='wrap' The validator replaces or can skip the check AfterValidator or BeforeValidator
SkipValidation, any position that accepts any value No check runs A concrete type, or JsonValue
A class or metadata with its own __get_pydantic_core_schema__, GetPydanticSchema The hook replaces the check A type from the allow list
computed_field with extra='forbid' The field is written, and the read refuses it Remove one of the two
Json[T] The value is written as JSON, not as a JSON string Declare T
Literal with a plain Enum member The member is written as its value, and Literal refuses that The Enum type, or a str or int Enum
A dataclass with revalidate_instances other than never Revalidation drops the fields of a subclass Remove the setting
A default that does not hold its declared type The reader or the function would get a wrong type A default that is a value of the type

A default is checked against the type only. No validator runs at import, because a validator can need a model, a context or a service.

A self-check runs each rule once per process. The rules read core schema keys that pydantic owns, so a pydantic that renames a key crashes loud, with the rule, the version and the fix. It does not switch a rule off.

  • PrivateAttr and an attribute set outside the declared fields are not part of the data, so they do not cross the wire.
  • A validator that replaces an instance on purpose, such as AfterValidator that builds a base class, keeps its result.
  • A serializer on a dataclass that drops a field with a default is not found. A required field is refused at read-back.
  • default_factory is not called at definition, because it may have side effects. A wrong type is refused on each write. A read of an old row with the key missing is not checked.
  • A field default is not validated at read, as in pydantic. Write it in its final form, or set validate_default=True.
  • A constraint on a union, such as Annotated[int | float, Field(gt=0)] = -1, is not checked on a default.

An error is the report of a failure, and auto_retry_for and outcome routing read its code. So an error always encodes, and its code and message stay.

  • A lone surrogate in the code, the message or the exception text is escaped.
  • TaskError.data must be JSON-native. A value that is not, or that a reader cannot read, is replaced with {'data_refused': <cause>}. The cause names a type, never the value. A writer gives JsonInput, and a reader gets JsonValue, so pyright flags these at the call and takes a list[str].
  • A SubWorkflowError keeps its class, its id, its status and its counts. If its output cannot cross, the output is None.
  • A TaskError subclass with extra fields does not cross the wire, and comes back as a plain TaskError. Put the values in data, and match on error_code.

A str or dict subclass in data passes and reads as a plain str or dict. A StrEnum member reads as its value.

Models flowing through args_from ride through the same envelope. The downstream task declares TaskResult[Model, TaskError] (or an Optional / Union containing it) as the parameter type; the engine decodes the upstream envelope and binds a typed TaskResult into the consumer’s kwargs.

from pydantic import SecretStr
from datetime import datetime, timezone
from pydantic import BaseModel
from horsies import (
Horsies,
AppConfig,
PostgresConfig,
TaskResult,
TaskError,
TaskNode,
)
config = AppConfig(
broker=PostgresConfig(
database_url=SecretStr("postgresql+psycopg://user:password@localhost:5432/mydb"),
),
)
app = Horsies(config)
class Order(BaseModel):
item: str
total: float
created_at: datetime
@app.task("create_order")
def create_order() -> TaskResult[Order, TaskError]:
return TaskResult(ok=Order(
item="widget",
total=9.99,
created_at=datetime.now(timezone.utc),
))
@app.task("process_order")
def process_order(
*,
order_result: TaskResult[Order, TaskError],
) -> TaskResult[str, TaskError]:
if order_result.is_err():
return TaskResult(err=order_result.err_value)
order: Order = order_result.ok_value # typed Order, not a dict
return TaskResult(ok=f"Processed {order.item}")
node_create: TaskNode[Order] = TaskNode(fn=create_order)
node_process: TaskNode[str] = TaskNode(
fn=process_order,
waits_for=[node_create],
args_from={"order_result": node_create},
)

encode_task_result(result, ok_type) -> dict[str, Json]

Section titled “encode_task_result(result, ok_type) -> dict[str, Json]”

Encode a TaskResult[T, TaskError] to the wire envelope.

Parameter Type Description
result TaskResult[T, TaskError] Domain result returned by the task
ok_type TypeAnnotation Declared T from the task signature

Raises: ValueValidationError if result.ok does not match ok_type or pydantic cannot serialize it; StrictJsonError for non-JSON-serializable structures. ValueValidationError (horsies.core.codec) is a SerializationError and a ValueError; its text names each failing field, the message and the error type, not the value.

decode_task_result(envelope, ok_type) -> TaskResult[T, TaskError]

Section titled “decode_task_result(envelope, ok_type) -> TaskResult[T, TaskError]”

Decode a wire envelope into a typed TaskResult.

Parameter Type Description
envelope dict[str, Json] The __h_task_result__ envelope
ok_type TypeAnnotation Declared T from the task signature (only used when the ok slot is populated)

Decode the err slot independently of ok_type. Used by consumers reading failed tasks whose task name is unknown locally.

validate_task_result_envelope(envelope) -> dict[str, Json]

Section titled “validate_task_result_envelope(envelope) -> dict[str, Json]”

Cheap shape check before per-slot decode. Returns the envelope as dict[str, Json] on success; raises StrictJsonError if the marker is missing, the key set is wrong, or both slots are populated.

encode_value(value, type) / decode_value(payload, type)

Section titled “encode_value(value, type) / decode_value(payload, type)”

Primitive entry points for non-TaskResult values (workflow kwargs, success policies, args_from blocks). Same TypeAdapter-driven semantics.

Raises: encode_value raises ValueValidationError for a dict key without a JSON form (a frozenset, or an Enum member whose value is a list, a dict or a set), for a value that holds itself, for a value that is nested too deep, and for bytes that are not UTF-8. Any other TypeError or ValueError from the dump propagates, for example from a computed field.

Validators: a validator refuses a value with ValueError or AssertionError. Any other exception from a validator is a bug: it propagates, and encode_value and decode_value do not catch it.

type JsonValue = (
None | bool | int | float | str
| list[JsonValue]
| dict[str, JsonValue]
)

PEP 695 type alias. Re-exported as horsies.JsonValue. This is what a reader gets: an exact list or dict, so a match over them is exhaustive.

A type checker does not accept a list[str] for a list[JsonValue], because a list is mutable and the receiver could add an int to it. JsonInput is what a position accepts. It holds the same values as JsonValue, and a list[str] or a dict[str, int] fits it. It exists only for a type checker: at run time JsonInput is JsonValue, and every check of JsonValue applies.

Value Type checker Run time
A list or dict of JSON values, at any depth, with str keys Accepted Accepted
A subclass of dict, such as OrderedDict or Counter Accepted Accepted, read as a dict
A tuple, bytes, a deque or a set Flagged Refused
A Mapping that is not a dict, such as mappingproxy, UserDict or ChainMap Flagged Refused
dict[int, str], list[Decimal], or any other bad value inside a container Flagged Refused

JsonList and JsonDict are the protocols behind it, and they appear in hover text and in errors. JsonList has pop with an index, because only a list has that form. JsonDict has __reversed__ and popitem, because only a dict has both.

TaskError.data takes JsonInput and gives JsonValue, and so does a subclass of TaskError.

A task parameter has one annotation, which serves .send() and the task body. A return type has one too. Choose by what the task does with the value:

Annotation Use it when Result
A concrete type The shape is known The best choice: .send() and the body both get the exact type
JsonValue The task reads the value An exact read. Pyright flags a list[str] variable at .send()
JsonInput The task passes the value on .send() takes a list[str]. A read in the body gives partly unknown types in strict mode

Pyright flags an open TypedDict as JsonInput, because it can hold keys that it does not declare. Pass a dict, or use a model. Pyright also flags a TaskError that is passed to a user’s read-only Protocol with a data property. mypy accepts that form.

Raised by strict codec primitives and the JSON parse guards for strict JSON violations. The parse guards reject the constants NaN, Infinity and -Infinity, and a numeric literal outside the finite binary64 range, such as 1e9999. A literal that underflows, such as 1e-9999, parses to 0.0. They also reject a string with a lone surrogate, such as \ud800; a surrogate pair in JSON text is one valid character and stays accepted. encode_value refuses each string that dumps_json refuses: a lone surrogate in a string or in a dict key raises StrictJsonError at any depth. In a Python str, a surrogate pair is two code points, so it is refused too. encode_value and decode_value also raise StrictJsonError when the error text of a validator holds a lone surrogate, because pydantic cannot build that text. The text names the kwarg and holds no input value. dumps_json / loads_json return SerdeResult and wrap stringification or parse failures as SerializationError. Re-exported as horsies.StrictJsonError.

MAX_TYPED_JSON_DEPTH (200) is the deepest nesting of a value under a declared type, the limit of the strict JSON read. A deeper value is refused at read-back. MAX_JSON_DEPTH (300) is the deepest nesting of lists and dicts at the JSON boundary, which also holds the envelopes around a value. dumps_json and the fence of encode_value at a JsonValue position refuse a deeper value and a value that holds itself. loads_json refuses a deeper text, so each walker that gets a parsed value stays inside the recursion limit of Python. The bound is above the limit of a pydantic dump, which refuses at 255 levels: a value that encode_value accepts also passes the bound inside its envelope.

Don’t use Any, object, or bare containers in task signatures. They are rejected at registration time.

# Wrong — raises SignatureValidationError
@app.task("bad")
def bad(*, data: dict) -> TaskResult[dict, TaskError]:
return TaskResult(ok=data)
# Correct — declare the contents
@app.task("good")
def good(*, data: dict[str, JsonValue]) -> TaskResult[dict[str, JsonValue], TaskError]:
return TaskResult(ok=data)

Don’t define result types in __main__ or inside functions. Workers import types by module path; classes without an importable qualname cannot participate in typed decode.

Don’t smuggle reserved keys into TaskError.error_code or TaskError.data. TaskError.error_code accepts the built-in OperationalErrorCode / RetrievalCode / ContractCode enums or any non-reserved string. Reserved built-in codes passed as strings (e.g. "WORKER_RESOLUTION_ERROR") are rejected — pass the enum member directly.

Don’t expect class identity to survive the wire. Two BaseModel subclasses with structurally identical fields decode interchangeably when the consumer’s declared type matches either. The wire stores values, not classes.