Skip to content

Defining Tasks

Define tasks by decorating functions with @app.task() and returning TaskResult[T, TaskError]. A task function is a def or an async def function. For argument serialization rules, see sending-tasks.

from pydantic import SecretStr
from horsies import Horsies, AppConfig, PostgresConfig, TaskResult, TaskError, JsonValue
config = AppConfig(
broker=PostgresConfig(database_url=SecretStr("postgresql+psycopg://...")),
)
app = Horsies(config)
@app.task("process_order")
def process_order(*, order_id: int) -> TaskResult[str, TaskError]:
# Do work
return TaskResult(ok=f"Order {order_id} processed")

Define a task with async def to await I/O in its body. @app.task() checks it as it checks a def task, and it returns a TaskResult in the same way.

import asyncio
@app.task("fetch_price", timeout_ms=30_000)
async def fetch_price(*, symbol: str) -> TaskResult[float, TaskError]:
await asyncio.sleep(0.1) # stands in for an awaited client call
return TaskResult(ok=101.5)

With Execution.ISOLATED (the default), a worker child runs one task at a time, def or async def. A SHARED_ASYNC task runs next to other tasks on the event loop of a shared async child instead: see Shared Async Tasks. An isolated child keeps one event loop until the child stops, and an async task runs on that loop. An async client that a module keeps stays open across the tasks of one child.

The loop runs only while an async task runs. A def task runs with no running event loop, so it can call asyncio.run(). While the child runs a def task or waits for its next task, the timers of a module-level async client, for example a keepalive or a pool maintenance task, do not run.

An async task must await the work that it starts. When the body of an async task returns (a value, or an exception that the task maps to a result) and asyncio tasks that it started, for example with asyncio.create_task(), are still on the loop:

  1. The child waits up to 1 second for that work, also for the asyncio tasks that it starts meanwhile.
  2. Work that is still alive gets a cancel, and two more cancels 0.1 seconds apart.
  3. When the work has ended, the worker writes what the task returned. Only then do the retry and the next workflow node start, so they never run next to that work. The worker then replaces the child, and the next task runs in a new child: cancelled code can leave broken state in the process.
  4. Work that is still alive 0.3 seconds after the first cancel, for example a task that catches every cancel, is stuck. The worker kills the child, and writes what the task returned after it saw the exit. A child that reports neither the end of the work nor that it is stuck within 26.3 seconds of the return, for example because that work blocks the event loop, is killed the same way. When the exit is not seen within 30 seconds of the kill, the rule of a timeout kill applies (see Set an Execution Timeout).

The worker writes runner heartbeats for the row while the child waits for that work: the child marks the row finalizing only after the work ended, and not at all when it did not stop. So stale RUNNING recovery does not fail the row while the work runs.

The return ends the deadline of the task: timeout_ms bounds the body, and what the task returned stands while the child waits for the work. The outcome is never TASK_TIMEOUT then. A deadline that fires before the worker has read the return still kills the child, as for a result that arrives after the deadline. A crash of the child in that wait also keeps what the task returned, once the worker saw the exit.

  • The worker sees asyncio tasks only. A callback that loop.call_soon() or loop.call_later() scheduled, and work that loop.run_in_executor() started and the task did not await, are not waited for and cause no replacement. They can still run during a later task of the same child.
  • A client that keeps its own background tasks on the loop, for example a keepalive or a consumer task, delays the outcome of each async task by about 1 second, and causes a child replacement after it.
  • A thread that asyncio.to_thread() starts can outlive its task, for example when a timeout ends the wait for it. The thread is not an asyncio task, so it is not waited for and does not cause a replacement. When the worker stops the child, the child does not wait for that thread past the close, which takes at most 7 seconds: it exits, and its atexit handlers do not run. An idle executor thread does not count: the close of the async generators and the shutdown of the executor each get at least 1 second.
  • The worker logs one line for each task that left work: the task, the coroutine of each asyncio task that was left, for up to 10 tasks, and how that work ended, for example Task watch_price (<id>) returned and left 1 task(s) on the event loop of child pid=4242 (refresh_session); that work ended with no cancel; the outcome is what the task returned, and the child is replaced.

A task that cancels a helper task in finally must also await it. cancel() only asks the helper to stop. A helper that awaits while it stops, for example to close a session, is still on the loop when the task returns: the outcome waits for it, and the worker replaces the child. Await the helper after the cancel:

async def refresh_session() -> None:
try:
while True:
await asyncio.sleep(30) # stands in for a keepalive call
finally:
await asyncio.sleep(0) # stands in for closing the session
@app.task("watch_price")
async def watch_price(*, symbol: str) -> TaskResult[float, TaskError]:
helper = asyncio.create_task(refresh_session())
try:
await asyncio.sleep(0.1) # stands in for the work of the task
return TaskResult(ok=101.5)
finally:
helper.cancel()
# gather returns the CancelledError of the helper instead of raising it.
await asyncio.gather(helper, return_exceptions=True)

A child creates its event loop at boot, before it reports ready, when the app has at least one async def task. A child of an app with no async def task creates no loop. When the loop cannot be created:

  • The child fails its boot, and no task reaches it.
  • An OSError with EMFILE, ENFILE, ENOMEM or EAGAIN (the process has no file descriptor left, for example) can clear, so the slot is down: it takes no task, and the worker starts a new child for it after a growing wait. Other children continue. When every slot is down, the worker exits non-zero after three failed attempts, so its supervisor restarts it. The log names the slot and the cause, for example Slot 1 is down: the OS ran short of a resource (EMFILE): OSError: [Errno 24] Too many open files.
  • Any other failure, such as an event loop policy that raises, is the same on every start, so it repeats (TaskLoopCreateFailed): each failed start counts one of three attempts, also while other children serve, and the third stops the worker. See Child Start Failures.

In an async task, send with .send_async() or .schedule_async(). The sync .send() and .schedule() refuse a running event loop: they return Err(TaskSendAsyncContext) and enqueue nothing. See Sending Tasks.

from horsies import Ok, Err
@app.task("refresh_price")
async def refresh_price(*, symbol: str) -> TaskResult[str, TaskError]:
match await fetch_price.send_async(symbol=symbol):
case Ok(handle):
return TaskResult(ok=handle.task_id)
case Err(send_err):
return TaskResult(err=TaskError(error_code="SEND_FAILED", message=send_err.message))

timeout_ms works as for a def task: on expiry, the worker kills the child. An async def function that uses yield is refused with HRS-101.

Every task parameter must be keyword-only: declare them after a bare *,. The leading * forces callers to pass arguments by name.

@app.task("send_email")
def send_email(*, to: str, subject: str) -> TaskResult[None, TaskError]:
...

The queue transmits arguments by name only — the signature is the wire contract. A positionally-bindable parameter (def f(order_id: int)) is rejected at registration, so the error surfaces when the app loads, not at send time. With keyword-only parameters, a positional call (process_order.send(1)) also fails the type checker at the call site rather than at runtime.

@app.task("compute")
def compute(*, x: int) -> TaskResult[int, TaskError]:
return TaskResult(ok=x * 2)

For void tasks:

@app.task("fire_and_forget")
def fire_and_forget() -> TaskResult[None, TaskError]:
do_something()
return TaskResult(ok=None)

Domain errors (expected failures):

@app.task("validate_input")
def validate_input(*, data: dict[str, JsonValue]) -> TaskResult[dict[str, JsonValue], TaskError]:
if not data.get("email"):
return TaskResult(err=TaskError(
error_code="MISSING_EMAIL",
message="Email is required",
data={"field": "email"},
))
return TaskResult(ok=data)

Unhandled exceptions are wrapped automatically:

@app.task("might_crash")
def might_crash() -> TaskResult[str, TaskError]:
raise ValueError("Something went wrong")
# Becomes TaskResult(err=TaskError(
# error_code=OperationalErrorCode.UNHANDLED_EXCEPTION,
# message="Unhandled exception in task might_crash: ...",
# ))

With explicit try/except:

from horsies import RetryPolicy, TaskResult, TaskError, JsonValue
@app.task(
"some_api_call",
retry_policy=RetryPolicy.exponential(
base_seconds=30,
max_retries=5,
auto_retry_for=["TIMEOUT", "CONNECTION_ERROR"],
),
)
def some_api_call() -> TaskResult[dict[str, JsonValue], TaskError]:
try:
result = call_external_api()
return TaskResult(ok=result)
except TimeoutError:
return TaskResult(err=TaskError(error_code="TIMEOUT", message="Request timed out"))
except ConnectionError:
return TaskResult(err=TaskError(error_code="CONNECTION_ERROR", message="Connection failed"))

Or with the exception mapper (no try/except needed):

@app.task(
"some_api_call",
retry_policy=RetryPolicy.exponential(
base_seconds=30,
max_retries=5,
auto_retry_for=["TIMEOUT", "CONNECTION_ERROR"],
),
exception_mapper={
TimeoutError: "TIMEOUT",
ConnectionError: "CONNECTION_ERROR",
},
)
def some_api_call() -> TaskResult[dict[str, JsonValue], TaskError]:
result = call_external_api()
return TaskResult(ok=result)

For backoff strategies, jitter, and auto-retry triggers, see Retry Policy.

@app.task("urgent_task", queue_name="critical")
def urgent_task() -> TaskResult[str, TaskError]:
...

timeout_ms bounds task execution time, measured from the moment a worker gives the task to a child process. Time that a task waits for a free child does not count. The value is from 1000 (1 second) to MAX_TASK_TIMEOUT_MS (4,294,967,295, about 49.7 days), which is also the limit of the Rust port. @app.task refuses a value outside that range with TASK_INVALID_OPTIONS. A queued row that holds a value outside it fails with WORKER_SERIALIZATION_ERROR when a worker claims it, and no user code runs.

@app.task("resize_video", timeout_ms=120_000)
def resize_video(*, video_id: str) -> TaskResult[str, TaskError]:
...

On expiry, the worker kills the child process, waits for it to exit, and then fails the task with TASK_TIMEOUT. A result that reaches the worker after the deadline does not change this. An async def body that returned before the deadline is not timed out while the child waits for the asyncio tasks that it left (see Async Tasks). If the deadline elapses before user code starts, the task is requeued instead. Timeouts are terminal by default; opt into retries by adding "TASK_TIMEOUT" to auto_retry_for:

@app.task(
"resize_video",
timeout_ms=120_000,
retry_policy=RetryPolicy.fixed([60, 300], auto_retry_for=["TASK_TIMEOUT"]),
)
def resize_video(*, video_id: str) -> TaskResult[str, TaskError]:
...

The kill goes to the child and to the subprocesses in its session. Other tasks on the same worker continue. A subprocess that calls setsid itself leaves the session, and the kill does not reach it. The worker starts a new child in that slot, so size timeout_ms as a hang backstop well above normal runtime, not as routine control flow.

The worker writes the timeout only after the child has exited. So a retry never runs next to the attempt that it replaces, and a workflow node that runs after a failure does not start while that attempt runs. If the child does not exit within 30 seconds of the kill, the worker writes nothing and logs at critical level. Then stale-task recovery handles the task: a started task as a crashed worker’s task, and a task that had not started when its claim lease ends. With stale RUNNING recovery off (no RecoveryConfig, or auto_fail_stale_running=False), nothing else would move the row, so the worker waits for the exit and writes the row once it sees it; a stop of the worker ends that wait.

A task with Execution.SHARED_ASYNC on a shared async child is different. On expiry its child cancels it: the task gets asyncio.CancelledError at its next await. The task fails with TASK_TIMEOUT when it has stopped, with the work that it started, also when it returns a value after the cancel. If it has not stopped cancel_grace_ms after the cancel (default 1 second), the child cancels it a second time, and takes no new task. A task that still has not stopped is stuck: the worker kills the child when every task left in it is stuck, so no other task in that child loses time that it still had. Each of those tasks then fails with TASK_TIMEOUT. A task that blocks the event loop of its child for 5 seconds makes the worker kill the child at once: that task fails with TASK_BLOCKED_LOOP, each other task before its deadline with NEIGHBOUR_BLOCKED_LOOP. See Shared Async Tasks.

A SHARED_THREAD task on a shared thread child cannot be cancelled. On expiry its child closes: it takes no new task. When no task in the child has time left, the worker kills the child, and each task that was still in it fails with TASK_TIMEOUT. A task alone in its child is killed at its deadline. See Shared Thread Tasks.

execution declares how a worker runs the task. The value is an Execution member, and the default is Execution.ISOLATED. Every task row stores the class. To choose a class for the work of a task, see Choosing an Execution Mode.

execution Runs Function timeout_ms cancel_grace_ms
Execution.ISOLATED (default) one task at a time in its own child process def or async def optional not allowed
Execution.SHARED_ASYNC many tasks as coroutines in one child process async def required optional
Execution.SHARED_THREAD many tasks on threads in one child process def required not allowed
@app.task(
"fetch_quote",
execution=Execution.SHARED_ASYNC,
timeout_ms=10_000,
cancel_grace_ms=2_000,
)
async def fetch_quote(*, symbol: str) -> TaskResult[str, TaskError]:
...
@app.task("render_thumbnail", execution=Execution.SHARED_THREAD, timeout_ms=30_000)
def render_thumbnail(*, image_id: str) -> TaskResult[str, TaskError]:
...

A SHARED_ASYNC task runs on a seat of a shared async child when the worker profile has shared_async=SharedAsyncChildren(...), and in an isolated child otherwise. See Shared Async Tasks. A SHARED_THREAD task runs on a seat thread of a shared thread child when the profile has shared_thread=SharedThreadChildren(...), and in an isolated child otherwise. See Shared Thread Tasks.

cancel_grace_ms is the time that a SHARED_ASYNC task gets to stop after its cancel. After it, the child cancels the task a second time. A thread cannot be cancelled, so only SHARED_ASYNC takes it. The value is from MIN_CANCEL_GRACE_MS (100) to MAX_CANCEL_GRACE_MS (10,000). A SHARED_ASYNC task that sets none gets DEFAULT_CANCEL_GRACE_MS (1,000). The stored options of the task hold the value in effect when the task is sent, so a later change of the default does not change a queued task.

@app.task refuses a declaration that breaks a rule with a TaskDefinitionError, one code per rule:

Code Name Rule
HRS-107 TASK_SHARED_ASYNC_REQUIRES_ASYNC_DEF SHARED_ASYNC needs an async def function
HRS-108 TASK_SHARED_THREAD_REQUIRES_DEF SHARED_THREAD needs a def function
HRS-109 TASK_SHARED_EXECUTION_REQUIRES_TIMEOUT SHARED_ASYNC and SHARED_THREAD need timeout_ms
HRS-110 TASK_CANCEL_GRACE_REQUIRES_SHARED_ASYNC only SHARED_ASYNC takes cancel_grace_ms
HRS-111 TASK_CANCEL_GRACE_OUT_OF_RANGE cancel_grace_ms is from 100 to 10,000

When a declaration breaks more than one rule, the first rule in this order is reported. A type checker refuses the same declarations when execution is written as a member: the overloads require timeout_ms for a shared class, and the decorator that they return accepts only the function kind of the class. With cancel_grace_ms and a member other than SHARED_ASYNC, the decorator accepts no function. A class that is known only at run time requires timeout_ms and takes cancel_grace_ms, and the decorator checks the other rules at run time.

# Dotted module paths (recommended)
app.discover_tasks(["myapp.tasks", "myapp.jobs.worker_tasks"])
# Or file paths
app.discover_tasks(["tasks.py", "more_tasks.py"])

discover_tasks imports the exact entries listed. Dotted module paths use importlib.import_module(), while .py entries are imported by file path — it does not recursively scan submodules. To discover tasks in myapp.tasks.scraping, either list it explicitly or export the decorated functions from myapp.tasks.__init__.py.

Don’t omit the return type annotation.

# Wrong - raises TypeError at definition time
@app.task("bad_task")
def bad_task():
return {"status": "done"}
# Correct
@app.task("good_task")
def good_task() -> TaskResult[dict[str, JsonValue], TaskError]:
return TaskResult(ok={"status": "done"})

Don’t reuse task names.

@app.task("duplicate_name")
def task_one() -> TaskResult[str, TaskError]:
...
@app.task("duplicate_name") # Raises error
def task_two() -> TaskResult[str, TaskError]:
...
Parameter Type Required Description
task_name str Yes Unique task identifier. An empty name raises TaskDefinitionError (HRS-102)
queue_name str No Target queue (CUSTOM mode only)
retry_policy RetryPolicy No Retry timing, backoff, and auto-retry triggers
exception_mapper dict[type[BaseException], str] No Maps exception classes to error codes
default_unhandled_error_code str No Error code for unmapped exceptions (overrides global)
timeout_ms int No Execution time limit in milliseconds, from 1000 to MAX_TASK_TIMEOUT_MS (4,294,967,295), measured from the moment a child takes the task. On expiry the child process is killed and the task fails with TASK_TIMEOUT
execution Execution No How a worker runs the task: ISOLATED (default), SHARED_ASYNC or SHARED_THREAD. See Choose the Execution Class
cancel_grace_ms int No SHARED_ASYNC only. The time to stop after a cancel, from 100 to 10,000 ms. Default 1,000 ms

Returns: For a def function, a TaskFunction: a direct call returns the TaskResult. For an async def function, an AsyncTaskFunction: a direct call returns a coroutine, and await gives the TaskResult. Both send with .send() / .send_async().

good_until is not accepted here — task expiry deadlines are a per-send concern. See Sending Tasks — Set a Task Deadline.