Broker Config
Basic Usage
Section titled “Basic Usage”from pydantic import SecretStrfrom horsies import PostgresConfig
broker = PostgresConfig( database_url=SecretStr("postgresql+psycopg://user:password@localhost:5432/mydb"),)Fields
Section titled “Fields”| Field | Type | Default | Description |
|---|---|---|---|
database_url |
SecretStr |
required | SQLAlchemy connection URL |
session_database_url |
SecretStr | None |
None |
Direct/session-capable URL for schema setup and LISTEN/NOTIFY |
pgbouncer_transaction_mode |
bool |
False |
Disable prepared statements for transaction-pooled PgBouncer |
pool_size |
int |
5 | Connection pool size (raise for high-throughput producers) |
max_overflow |
int |
10 | Additional connections beyond pool_size |
pool_timeout |
int |
30 | Seconds to wait for connection |
pool_pre_ping |
bool |
True |
Ping a pooled connection at checkout when it waited in the pool for more than 1 s (PRE_PING_AFTER_IDLE_S), in every engine but the one of the worker process; see Remote PostgreSQL |
worker_pool_pre_ping |
bool |
False |
Ping a pooled connection of the worker process at checkout when it waited in the pool for more than 1 s. Off: the worker sends a one-statement write in one round trip, and a dead pooled connection fails its first statement with a connection error that the worker retries; see Remote PostgreSQL |
tcp_keepalives |
bool |
True |
Enable libpq TCP keepalives on broker and child-process connections; see Remote PostgreSQL |
tcp_keepalives_idle |
int |
30 | Idle seconds before the first keepalive probe (libpq keepalives_idle) |
tcp_keepalives_interval |
int |
10 | Seconds between keepalive probes (libpq keepalives_interval) |
tcp_keepalives_count |
int |
3 | Unacknowledged probes before the connection is dropped (libpq keepalives_count) |
tcp_user_timeout_ms |
int |
30000 | Milliseconds that sent data may stay unacknowledged before the connection is closed (libpq tcp_user_timeout); 0 uses the operating system default; see Remote PostgreSQL |
retain_rerun_input_default |
bool |
False |
App-level default for rerun-input retention at enqueue; a per-task value overrides it |
echo |
bool |
False |
Echo SQL statements to logs. Bound values are not shown |
pool_recycle |
int |
1800 | Recycle connections after N seconds |
The pools of a worker are not PostgresConfig fields. Each child kind has its own pool and task_send_pool in the worker profile, and the worker process has WorkerProfile.pool. See Connection Pools and the Default Executor. worker_child_pool_min_size, worker_child_pool_max_size, worker_child_pool_check, worker_pool_size and worker_max_overflow were removed in 0.7.0; a config that sets one fails with HRS-213, and the error names the profile field that replaces it.
Connection URL Format
Section titled “Connection URL Format”postgresql+psycopg://user:password@host:port/databaseComponents:
postgresql+psycopg- Driver (psycopg3)user:password- Credentialshost:port- Server location (default port: 5432)database- Database name
psycopg and libpq
Section titled “psycopg and libpq”horsies needs psycopg 3.3.7 or later. The install sets where the libpq of psycopg comes from:
| Install | psycopg loads |
|---|---|
horsies |
the libpq of the system |
horsies[pg-binary] |
the libpq that psycopg-binary bundles |
horsies[pg-c] |
the libpq of the system, through psycopg-c |
When a cancelled asyncio task, KeyboardInterrupt or SystemExit interrupts a running query, psycopg:
- Sends a cancel request to the server, for at most 5 seconds.
- Waits for the query to end, for at most 5 seconds.
- Closes the connection, when the query has not ended.
The 5-second limit of step 1 needs libpq 17 or later. With an older libpq, the cancel request has no limit, and in async code psycopg sends it from a thread of the default executor. The 5-second limit of step 2 applies with every libpq. Read the version that psycopg loaded with psycopg.pq.version(): libpq 17.0 reads 170000.
In a shared async child, that thread is work of the task: the attempt ends only when the cancel request returns. A second cancel does not stop a thread.
Examples
Section titled “Examples”Local Development
Section titled “Local Development”PostgresConfig( database_url=SecretStr("postgresql+psycopg://postgres:postgres@localhost:5432/horsies_dev"),)Production with Connection Pool
Section titled “Production with Connection Pool”PostgresConfig( database_url=SecretStr("postgresql+psycopg://app:secret@db.example.com:5432/production"), pool_size=10, max_overflow=20, pool_timeout=30, pool_recycle=3600,)From Environment Variable
Section titled “From Environment Variable”import os
PostgresConfig( database_url=SecretStr(os.environ["DATABASE_URL"]),)PgBouncer / Transaction Pooling
Section titled “PgBouncer / Transaction Pooling”Use a split configuration when the main SQL URL points to PgBouncer transaction
pooling. database_url is used for ordinary SQL work. session_database_url is
used for schema setup, partition maintenance, and PostgreSQL LISTEN/NOTIFY.
These operations need persistent session semantics.
from horsies import AppConfig, PostgresConfig
config = AppConfig( broker=PostgresConfig( database_url=SecretStr(os.environ["DATABASE_URL_POOLED"]), session_database_url=SecretStr(os.environ["DATABASE_URL_DIRECT"]), pgbouncer_transaction_mode=True, ),)Some managed providers, including PlanetScale Postgres, expose separate pooled
and direct connection strings. Treat provider-specific ports and hostnames as an
example, not a Horsies requirement. Check your provider dashboard or current
provider docs and copy the pooled transaction URL into database_url and the
direct/session-capable URL into session_database_url.
Rules:
- Workers require
session_database_urlwhenpgbouncer_transaction_mode=True. - The two URLs must be different in transaction-pool mode.
- Worker startup checks session behavior before partition maintenance starts.
- Syce requires the session URL for real-time updates.
- PgBouncer-only worker deployment is unsupported.
- Horsies does not infer PgBouncer mode from port numbers.
Horsies runs SET jit = off once on each new broker connection that keeps
session settings. Some partition-maintenance statements have cost estimates
above the default jit_above_cost, so a JIT-enabled server would compile
them on every call.
- Direct connections and the
session_database_urlpool always run without JIT. - With
pgbouncer_transaction_mode=True, thedatabase_urlpool keeps the server’s JIT setting, because server sessions are shared between clients. To disable JIT there too, set it for the role:ALTER ROLE <horsies_role> SET jit = off; - Your
database_urloptions are not changed.
Connection Pooling
Section titled “Connection Pooling”The broker uses SQLAlchemy’s async connection pool:
pool_size: Base number of persistent connectionsmax_overflow: Additional connections created under load (temporary)pool_timeout: How long to wait if all connections are busypool_recycle: Close and recreate connections after this many seconds
Sizing Guidelines
Section titled “Sizing Guidelines”pool_size and max_overflow size the general broker used by producers,
web processes, schedulers, and app-level APIs. They do not size the pools of a
worker: the worker process opens WorkerProfile.pool, and task code in a worker
child opens the task_send_pool of its child kind.
| Deployment | pool_size | max_overflow |
|---|---|---|
| Development | 2-5 | 5 |
| Small production | 5-10 | 10-20 |
| High traffic | 10-20 | 20-40 |
For direct PostgreSQL connections, budget a worker roughly as:
pool max_size (the pool of the worker process) + 2 session/LISTEN connections + 1 runner heartbeat connection + isolated children * isolated.pool max_size + shared async children * shared_async.pool max_size + shared thread children * shared_thread.pool max_size + task code that sends: children * task_send_pool of their kind, per event loopEach max_size is the resolved value: a PoolSize, or the size that
PoolSizeFromSeats() derives from the seats; see
Connections. With the defaults
and a profile of 8 isolated children, the framework ceiling is about
5 + 2 + 1 + 8 * 1 = 16 direct connections, and 8 * 2 = 16 more when every
child sends from one event loop.
Idle child processes keep no database connection by default because the
min_size of each pool is 0; connections are opened lazily when task
lifecycle work starts. An isolated child uses one connection at a time, for the
RUNNING update and the finalizing mark: the worker process writes the runner
heartbeats over its own connection. So the default
IsolatedPool(max_size=PoolSize(connections=1)) is enough for the own work of
an isolated child. The seats of a shared child share the pool of their child;
see Connections for its
size.
When using PgBouncer transaction pooling, those client-side pool slots are multiplexed onto fewer server backends. Connections are still not safe to share between OS processes; Horsies creates child pools inside each child process, and a child inherits no database socket from the worker.
Event Loops
Section titled “Event Loops”A broker keeps one resource set per open event loop that calls it. A set is a
runtime engine with its own pool, a session engine when session_database_url
differs from database_url, and a LISTEN/NOTIFY listener. Sync calls
(send(), TaskHandle.get(), spec.start(), the sync WorkflowHandle
methods, app.ping_workers()) run on the broker’s own sync runner loop, which
has its own set. One exception: in a process that runs no task bodies, a sync
send without an idempotency key writes on the calling thread, through the
sync send pool of the broker, once the broker checked the schema. See
Connection Ceiling.
- A set is built on the first call from its loop. Its pools open connections on demand, so a set that makes no database call opens none.
- When a loop closes, the next broker call from any loop releases that loop’s set: its idle connections and its listener connections close. A connection that a task of the closed loop still held is closed by garbage collection.
- A broker serves at most 5 open loops other than its sync runner loop and, in
a worker child, the task loop that runs
async deftasks. A call from one more open loop returnsEVENT_LOOP_LIMITin the error type of that call, and nothing runs. The error clears when one of those loops closes. The limit covers pytest-asyncio, which keeps one loop open per fixture scope (session, package, module, class, function). close_async()andclose()close every set: the calling loop’s set directly, the set of a closed loop by release, and the set of another open loop on that loop. When that loop does not run the close within 5 s, the result isCLOSE_FAILEDwith the loop’s thread in the message, and the close stays scheduled on that loop. A later call builds a new set.
Async Code and Jupyter
Section titled “Async Code and Jupyter”A sync call blocks its thread until the sync runner loop completes the call. On a thread that runs an event loop, this wait stops every coroutine on that loop. Coroutines and Jupyter notebook cells run on such a thread.
On such a thread, a sync call returns ASYNC_CONTEXT in the error type of
that call. Nothing runs, and the sync runner does not start.
| Sync call | Returns |
|---|---|
send(), schedule(), retry_send(), retry_schedule() |
TaskSendAsyncContext (TaskSendErrorCode.ASYNC_CONTEXT); the error keeps the payload for retry_send_async() / retry_schedule_async() |
app.get_result(), app.get_raw_result(), app.get_task_info(), app.ping_database(), app.ping_workers(), app.list_worker_states(), app.get_worker_state(), app.get_worker_state_history(), TaskHandle.info(), broker.enqueue(), broker.get_task_info(), broker.get_raw_result_record(), broker.close() |
BrokerAsyncContext (BrokerErrorCode.ASYNC_CONTEXT) |
TaskHandle.get(), WorkflowHandle.get(), WorkflowHandle.result_for() |
A result with RetrievalCode.ASYNC_CONTEXT, not cached |
WorkflowHandle.status(), .results(), .tasks(), .cancel(), .pause(), .resume(), pause_workflow_sync(), resume_workflow_sync() |
HandleAsyncContext (HandleErrorCode.ASYNC_CONTEXT) |
spec.start(), spec.retry_start() |
WorkflowStartAsyncContext (WorkflowStartErrorCode.ASYNC_CONTEXT); the error keeps the workflow id for retry_start_async() |
app.check(live=True) |
HRS-217 (CHECK_LIVE_ASYNC_CONTEXT); the live check does not connect |
On such a thread, use the *_async form:
# In a coroutine or a notebook cellhandle = (await my_task.send_async(value=1)).unwrap()result = await handle.get_async(timeout_ms=5000)- A
TaskHandlethat already holds its result returns it fromget(), because that call does not wait. await spec.retry_start_async(err)completes a start that returnedASYNC_CONTEXT, with the error’s workflow id.- To run a sync call from async code, run it on a thread without an event
loop, for example
await asyncio.to_thread(app.check, live=True).
Connection Ceiling
Section titled “Connection Ceiling”Each set can open:
pool_size + max_overflow # runtime engine+ pool_size + max_overflow # session engine, only when session_database_url differs+ 2 # listener, from the first result wait, workflow wait or worker pingIn a process that runs no task bodies, the broker has one more pool, the
sync send pool: up to pool_size + max_overflow connections, with the same
URL, connect args and idle ping as the runtime engine. A sync send without an
idempotency key (send(), schedule(), retry_send(), retry_schedule())
writes on the calling thread through this pool, once the broker checked the
schema. The first send of a broker (it checks the schema), a send with an
idempotency key, and the other sync calls use the set of the sync runner.
Task code in a worker child sends through the set of its task_send_pool
only, and has no sync send pool.
connections of one process = sets × (the ceiling of one set) + pool_size + max_overflow # sync send pool, when the table says Yes| Process | Sets | Sync send pool |
|---|---|---|
| Sync calls only, for example a WSGI app or a script | 1 (the sync runner loop) | Yes, when it sends without an idempotency key |
| Async calls from one loop only | 1 | No |
| ASGI app with async endpoints and sync calls in a thread pool | 2 | Yes, when the sync calls send without an idempotency key |
| pytest run with a session-scoped broker and function-scoped async tests | 2 (session loop, current test loop), plus 1 when tests make sync calls | Yes, when tests send with send() without an idempotency key |
| Worker child whose task code sends | 1 for the sync tasks (the runner loop), plus 1 for the async tasks (the task loop) | No |
At the defaults (pool_size=5, max_overflow=10, no session_database_url),
a process that sends only with send() opens up to 15 + 15 = 30
connections, and 2 more when it waits for a result. Each pool opens a
connection only when a call finds none free.
Things to Avoid
Section titled “Things to Avoid”Don’t leave event loops open after their work is done. Each open loop
keeps its set and its idle connections, and the sixth open loop gets
EVENT_LOOP_LIMIT.
# Wrong: each call leaves one more open looploop = asyncio.new_event_loop()loop.run_until_complete(broker.ping_database_async())
# Correct: asyncio.run() closes its loopasyncio.run(broker.ping_database_async())Don’t call sync close() inside a coroutine. It returns ASYNC_CONTEXT
and closes nothing. Use await broker.close_async().
Multiple Components
Section titled “Multiple Components”The broker creates three connection types:
- Runtime async engine (SQLAlchemy): For task and workflow SQL.
- Maintenance async engine (SQLAlchemy): For schema and partition DDL.
- LISTEN/NOTIFY (psycopg): For real-time notifications.
By default all three use database_url. With split PgBouncer configuration, ordinary
SQL uses database_url. Schema setup, partition maintenance, and
LISTEN/NOTIFY use session_database_url.
Health Monitoring
Section titled “Health Monitoring”The broker includes automatic health checks:
- Proactive connection monitoring
- Auto-reconnect on connection loss
- Exponential backoff for reconnection attempts
Schema Initialization
Section titled “Schema Initialization”On first use, the broker creates required tables:
horsies_tasks- Task storagehorsies_heartbeats- Liveness trackinghorsies_worker_states- Worker monitoringhorsies_schedule_state- Scheduler state
Uses PostgreSQL advisory locks to prevent race conditions during schema creation.
