Child-Process Hooks
@app.on_child_process_start registers a sync function that runs once in every worker child process: after task-module imports (so app engines exist) and before the child opens horsies’ own connection pool. It is the supported seam for application-owned per-child setup — the equivalent of Celery’s worker_process_init signal and Dramatiq’s after_process_boot middleware.
Why and When
Section titled “Why and When”A horsies worker runs task code in N child processes. Application code commonly holds its own module-level SQLAlchemy engine, separate from horsies’ broker. SQLAlchemy’s QueuePool has no idle eviction, so each child retains its pool’s connections for the life of the process: the app-owned connection floor plateaus at one pool per child and never drains.
Use the hook to set a worker-specific pool policy (e.g. a warm pool for web traffic, NullPool or a tiny pool in worker children). The hook runs in the child, so the child can build its own engine with a policy that differs from the web process.
How To
Section titled “How To”Task code must read the engine through indirection — an Engine’s pool class is fixed at create_engine() time, so the hook replaces the engine object rather than mutating it:
from sqlalchemy import Engine, create_engine, QueuePool
_app_engine: Engine | None = None
def _build_engine(*, worker_child: bool) -> Engine: if worker_child: # worker default: tiny bounded pool (NullPool behind PgBouncer) return create_engine(DB_URL, poolclass=QueuePool, pool_size=1, max_overflow=0) return create_engine(DB_URL, pool_size=10)
def get_engine() -> Engine: # task code reads via this, never the global global _app_engine if _app_engine is None: _app_engine = _build_engine(worker_child=False) return _app_enginefrom app.databases import postgres
@app.on_child_process_startdef reset_db_for_child() -> None: # idempotent, sync if postgres._app_engine is not None: postgres._app_engine.dispose() # close connections opened during import postgres._app_engine = postgres._build_engine(worker_child=True)Hooks run in registration order and fire on every child start, including replacement children — bodies must be idempotent. Register hooks in the app module or a discovered task module; a module the child never imports cannot fire its hooks.
With preload=ChildPreload.APP in the worker profile, your app module runs once, in the preload parent, before any child exists. Hooks still run in every child, so build per-child engines in the hook, and do not open a connection at import: the worker refuses to start if the import leaves a socket. Build clients of native libraries, such as gRPC, in the hook too: the guard does not see their threads.
Pool policy
Section titled “Pool policy”The hook is the mechanism; the pool policy is the application’s choice:
| Situation | Recommended app-engine pool |
|---|---|
| General default | QueuePool(pool_size=1, max_overflow=0) — warm reuse, floor capped at 1 per child |
| PgBouncer / transaction pooler with short DB touches | NullPool — floor ~0; connects to the pooler are cheap |
| DB-heavy tasks against direct Postgres | Small QueuePool; measure latency against connection count |
NullPool trades idle economy for a fresh connect (TCP/TLS/auth) per checkout and has no active-connection cap. It applies to the application’s engines only — horsies’ own child pool is not configurable through this hook.
Failure Semantics
Section titled “Failure Semantics”Fail-closed: a hook that raises, or runs past its 10-second budget, terminates the child with a dedicated exit code and the hook’s name in the worker log. The parent recognizes that exit code and stops the worker instead of restart-looping — the hook re-runs in every replacement child, so retrying cannot succeed. For best-effort work, catch exceptions inside the hook body.
Things to Avoid
Section titled “Things to Avoid”Don’t dispose without rebinding. dispose() alone does not change the floor: a QueuePool refills on the next checkout and retains idle connections again.
Don’t read the engine through a captured reference.
# Wrong — task captured the engine before the hook replaced itfrom app.databases.postgres import get_engineengine = get_engine() # module-import time, parent's engine
@app.task("report")def report() -> TaskResult[int, TaskError]: with engine.connect() as conn: # rebind never took effect here ...
# Correct — resolve at call time@app.task("report")def report() -> TaskResult[int, TaskError]: with get_engine().connect() as conn: ...Don’t register async functions. Hooks run during child boot, before any event loop exists; registration rejects coroutine functions (HRS-214). Dispose async engines via async_engine.sync_engine.dispose(close=False).
Limits
Section titled “Limits”- Engines created lazily after the hook runs, raw psycopg clients outside any pool, and pools hidden in C extensions are out of reach — only app-side close-per-use helps there.
- A sync hook cannot set pool policy for an async engine driven via
asyncio.runinside a task; choose that policy at construction.
Shared async child start hooks
Section titled “Shared async child start hooks”@app.on_shared_async_child_start registers an async def function that runs once in every shared async child, on the event loop of that child, before the child takes its first task. Use it to build long-lived async clients: an HTTP client or an async database pool that it builds belongs to the loop that every task of the child uses.
import asyncio
from horsies import Execution, TaskError, TaskResult
shared_state: dict[str, asyncio.AbstractEventLoop] = {}
@app.on_shared_async_child_startasync def remember_loop() -> None: shared_state['loop'] = asyncio.get_running_loop()
@app.task('loop_check', execution=Execution.SHARED_ASYNC, timeout_ms=5_000)async def loop_check() -> TaskResult[bool, TaskError]: return TaskResult(ok=shared_state['loop'] is asyncio.get_running_loop())- The hooks run after the
on_child_process_starthooks and after the child opens its async connection pool, in registration order. - Each hook runs for at most 10 seconds. A hook that raises or runs longer ends the child and stops the worker, as for
on_child_process_start. A hook that blocks the loop is ended by the same bound. - They run in every shared async child, also in a replacement child, so bodies must be idempotent.
- Work that a hook starts and leaves running, such as a background task of a client, belongs to no task.
- Isolated children do not run these hooks.
API Reference
Section titled “API Reference”@app.on_child_process_start
Section titled “@app.on_child_process_start”| Parameter | Type | Required | Description |
|---|---|---|---|
fn |
ChildProcessStartHook (Callable[[], None]) |
Yes | Sync, zero-argument function. Registered once per identity; duplicate registrations are ignored. |
Returns: the function unchanged (usable as a decorator).
Raises: ConfigurationError (HRS-214) — fn is not callable or is an async function.
app.get_child_process_start_hooks() -> list[ChildProcessStartHook]
Section titled “app.get_child_process_start_hooks() -> list[ChildProcessStartHook]”Snapshot of registered hooks in registration order.
@app.on_shared_async_child_start
Section titled “@app.on_shared_async_child_start”| Parameter | Type | Required | Description |
|---|---|---|---|
fn |
SharedAsyncChildStartHook (Callable[[], Coroutine[Any, Any, None]]) |
Yes | A callable with no arguments whose call returns a coroutine. Registered once per identity; duplicate registrations are ignored. |
Returns: the function unchanged (usable as a decorator).
Raises: ConfigurationError (HRS-214) — fn is not callable, or the registration can read that its call does not return a coroutine.
The registration does not call fn. It accepts:
- an
async deffunction; - an object whose
__call__isasync def; - a function, or an object whose
__call__is a function, whose return annotation is an awaitable type, such asCoroutine[Any, Any, None]; - a lambda, which can be neither
async defnor annotated. The child awaits its value; when that value is not awaitable, the hook fails when the child starts, and the worker stops.
app.get_shared_async_child_start_hooks() -> list[SharedAsyncChildStartHook]
Section titled “app.get_shared_async_child_start_hooks() -> list[SharedAsyncChildStartHook]”Snapshot of registered shared async child start hooks in registration order.
