Skip to content

Child-Process Hooks

@app.on_child_process_start registers a sync function that runs once in every worker child process: after task-module imports (so app engines exist) and before the child opens horsies’ own connection pool. It is the supported seam for application-owned per-child setup — the equivalent of Celery’s worker_process_init signal and Dramatiq’s after_process_boot middleware.

A horsies worker runs task code in N child processes. Application code commonly holds its own module-level SQLAlchemy engine, separate from horsies’ broker. SQLAlchemy’s QueuePool has no idle eviction, so each child retains its pool’s connections for the life of the process: the app-owned connection floor plateaus at one pool per child and never drains.

Use the hook to set a worker-specific pool policy (e.g. a warm pool for web traffic, NullPool or a tiny pool in worker children). The hook runs in the child, so the child can build its own engine with a policy that differs from the web process.

Task code must read the engine through indirection — an Engine’s pool class is fixed at create_engine() time, so the hook replaces the engine object rather than mutating it:

app/databases/postgres.py
from sqlalchemy import Engine, create_engine, QueuePool
_app_engine: Engine | None = None
def _build_engine(*, worker_child: bool) -> Engine:
if worker_child:
# worker default: tiny bounded pool (NullPool behind PgBouncer)
return create_engine(DB_URL, poolclass=QueuePool, pool_size=1, max_overflow=0)
return create_engine(DB_URL, pool_size=10)
def get_engine() -> Engine: # task code reads via this, never the global
global _app_engine
if _app_engine is None:
_app_engine = _build_engine(worker_child=False)
return _app_engine
app/configs/horsies.py
from app.databases import postgres
@app.on_child_process_start
def reset_db_for_child() -> None: # idempotent, sync
if postgres._app_engine is not None:
postgres._app_engine.dispose() # close connections opened during import
postgres._app_engine = postgres._build_engine(worker_child=True)

Hooks run in registration order and fire on every child start, including replacement children — bodies must be idempotent. Register hooks in the app module or a discovered task module; a module the child never imports cannot fire its hooks.

With preload=ChildPreload.APP in the worker profile, your app module runs once, in the preload parent, before any child exists. Hooks still run in every child, so build per-child engines in the hook, and do not open a connection at import: the worker refuses to start if the import leaves a socket. Build clients of native libraries, such as gRPC, in the hook too: the guard does not see their threads.

The hook is the mechanism; the pool policy is the application’s choice:

Situation Recommended app-engine pool
General default QueuePool(pool_size=1, max_overflow=0) — warm reuse, floor capped at 1 per child
PgBouncer / transaction pooler with short DB touches NullPool — floor ~0; connects to the pooler are cheap
DB-heavy tasks against direct Postgres Small QueuePool; measure latency against connection count

NullPool trades idle economy for a fresh connect (TCP/TLS/auth) per checkout and has no active-connection cap. It applies to the application’s engines only — horsies’ own child pool is not configurable through this hook.

Fail-closed: a hook that raises, or runs past its 10-second budget, terminates the child with a dedicated exit code and the hook’s name in the worker log. The parent recognizes that exit code and stops the worker instead of restart-looping — the hook re-runs in every replacement child, so retrying cannot succeed. For best-effort work, catch exceptions inside the hook body.

Don’t dispose without rebinding. dispose() alone does not change the floor: a QueuePool refills on the next checkout and retains idle connections again.

Don’t read the engine through a captured reference.

# Wrong — task captured the engine before the hook replaced it
from app.databases.postgres import get_engine
engine = get_engine() # module-import time, parent's engine
@app.task("report")
def report() -> TaskResult[int, TaskError]:
with engine.connect() as conn: # rebind never took effect here
...
# Correct — resolve at call time
@app.task("report")
def report() -> TaskResult[int, TaskError]:
with get_engine().connect() as conn:
...

Don’t register async functions. Hooks run during child boot, before any event loop exists; registration rejects coroutine functions (HRS-214). Dispose async engines via async_engine.sync_engine.dispose(close=False).

  • Engines created lazily after the hook runs, raw psycopg clients outside any pool, and pools hidden in C extensions are out of reach — only app-side close-per-use helps there.
  • A sync hook cannot set pool policy for an async engine driven via asyncio.run inside a task; choose that policy at construction.

@app.on_shared_async_child_start registers an async def function that runs once in every shared async child, on the event loop of that child, before the child takes its first task. Use it to build long-lived async clients: an HTTP client or an async database pool that it builds belongs to the loop that every task of the child uses.

import asyncio
from horsies import Execution, TaskError, TaskResult
shared_state: dict[str, asyncio.AbstractEventLoop] = {}
@app.on_shared_async_child_start
async def remember_loop() -> None:
shared_state['loop'] = asyncio.get_running_loop()
@app.task('loop_check', execution=Execution.SHARED_ASYNC, timeout_ms=5_000)
async def loop_check() -> TaskResult[bool, TaskError]:
return TaskResult(ok=shared_state['loop'] is asyncio.get_running_loop())
  • The hooks run after the on_child_process_start hooks and after the child opens its async connection pool, in registration order.
  • Each hook runs for at most 10 seconds. A hook that raises or runs longer ends the child and stops the worker, as for on_child_process_start. A hook that blocks the loop is ended by the same bound.
  • They run in every shared async child, also in a replacement child, so bodies must be idempotent.
  • Work that a hook starts and leaves running, such as a background task of a client, belongs to no task.
  • Isolated children do not run these hooks.
Parameter Type Required Description
fn ChildProcessStartHook (Callable[[], None]) Yes Sync, zero-argument function. Registered once per identity; duplicate registrations are ignored.

Returns: the function unchanged (usable as a decorator).

Raises: ConfigurationError (HRS-214) — fn is not callable or is an async function.

app.get_child_process_start_hooks() -> list[ChildProcessStartHook]

Section titled “app.get_child_process_start_hooks() -> list[ChildProcessStartHook]”

Snapshot of registered hooks in registration order.

Parameter Type Required Description
fn SharedAsyncChildStartHook (Callable[[], Coroutine[Any, Any, None]]) Yes A callable with no arguments whose call returns a coroutine. Registered once per identity; duplicate registrations are ignored.

Returns: the function unchanged (usable as a decorator).

Raises: ConfigurationError (HRS-214) — fn is not callable, or the registration can read that its call does not return a coroutine.

The registration does not call fn. It accepts:

  • an async def function;
  • an object whose __call__ is async def;
  • a function, or an object whose __call__ is a function, whose return annotation is an awaitable type, such as Coroutine[Any, Any, None];
  • a lambda, which can be neither async def nor annotated. The child awaits its value; when that value is not awaitable, the hook fails when the child starts, and the worker stops.

app.get_shared_async_child_start_hooks() -> list[SharedAsyncChildStartHook]

Section titled “app.get_shared_async_child_start_hooks() -> list[SharedAsyncChildStartHook]”

Snapshot of registered shared async child start hooks in registration order.