Skip to content

Architecture

Horsies is a PostgreSQL-backed distributed task queue and workflow engine with real-time dispatch via LISTEN/NOTIFY. Producers enqueue tasks, workers claim and execute them in isolated processes, and results flow back through the database. Async APIs (send_async, get_async, start_async, etc.) are producer-side I/O helpers; they do not change where tasks execute.

The system consists of four main components:

  1. Producer: Application code that sends tasks via .send() or .send_async()
  2. PostgreSQL Broker: Stores tasks, results, and coordinates via LISTEN/NOTIFY
  3. Worker: Claims tasks and executes them in child processes
  4. Scheduler: (Optional) Enqueues tasks on a schedule
Producer PostgreSQL Worker(s)
│ │ │
│ .send() / .send_async()│ │
├────────────────────────>│ INSERT task │
│ │ │
│ │ NOTIFY task_new ───────────>│
│ │ │
│ │<──────── CLAIM task ────────┤
│ │ │
│ │ │ Execute
│ │ │
│ │<──────── UPDATE result ─────┤
│ │ │
│ .get() / .get_async() │ NOTIFY task_done │
│<────────────────────────│ │

Unlike Celery (which typically uses Redis or RabbitMQ), horsies uses PostgreSQL for everything:

  • Task storage: The tasks table holds all task data
  • Message passing: LISTEN/NOTIFY replaces a separate message broker
  • Coordination: Advisory locks prevent race conditions
  • State tracking: Heartbeats and worker states are database tables

Workers run tasks in child processes that they own, rather than threads:

  • By default, each task runs in an isolated child process
  • GIL contention is avoided for CPU-bound tasks
  • A crashed or timed-out isolated task takes down only its own child
  • A task can opt in to a shared child, which runs many tasks as coroutines or on threads. See Choosing an Execution Mode
  • On Linux, children fork from a preload parent, so they start in milliseconds and share memory
  • The worker process writes the runner heartbeats of its children’s tasks, so a task that holds the GIL cannot stop them

All tasks must return TaskResult[T, TaskError]:

  • Forces explicit error handling at definition time
  • Distinguishes domain errors (returned by task) from infrastructure errors (thrown by library)
  • Enables typed result retrieval

Workers don’t poll for tasks:

  • PostgreSQL triggers send NOTIFY on task INSERT
  • Workers subscribe and wake immediately
  • Result waiters listen on task_done channel
  • Fallback polling handles edge cases (lost notifications)

Transaction-pooled PgBouncer is safe for ordinary SQL, but not for persistent session features. When pgbouncer_transaction_mode=True, Horsies uses this split:

Producer/Worker SQL --> database_url --> PgBouncer transaction pool
Worker LISTEN --> session_database_url --> direct/session Postgres
Result wait LISTEN --> session_database_url --> direct/session Postgres
Syce LISTEN --> session_database_url --> direct/session Postgres
Schema setup --> session_database_url --> direct/session Postgres
Partition maintenance --> session_database_url --> direct/session Postgres

Workers check session behavior before partition maintenance starts. Startup fails if the session URL is not session-capable. Producer-side result waits may fall back to polling when notifications are unavailable.

Component Description
Horsies Application instance - configure and register tasks here
@app.task Decorator to define tasks
TaskHandle Returned by .send() - use to retrieve results
TaskResult Return type of all tasks - contains success or error
Worker (CLI) Started via horsies worker <app> --profile <name>
Scheduler (CLI) Started via horsies scheduler command
  1. Producer calls task.send(**kwargs) (keyword-only)
  2. Queue and arguments are validated
  3. Task row inserted into tasks table with status PENDING
  4. PostgreSQL trigger fires NOTIFY to wake workers
  5. TaskHandle returned to producer
  1. Worker receives NOTIFY and wakes
  2. Worker claims task atomically
  3. Task dispatched to the child process of a free slot
  4. Child process executes task function
  5. Result stored in database, status updated to COMPLETED or FAILED
  1. Producer calls handle.get()
  2. If result cached on handle, return immediately
  3. Otherwise, listen for task_done notification or poll database
  4. Return TaskResult with success value or error
  • Cluster level: Optional cluster_wide_cap limits total RUNNING tasks
  • Worker level: the children of the worker profile limit concurrent executions per worker
  • Queue level: max_concurrency per custom queue (CUSTOM mode only)
  • Claiming: ClaimLimits.per_queue_per_pass prevents one worker from starving others

See Concurrency for details.