Remote workers

By default, trimus-server runs the worker pool inside its own process — heartbeats are claimed and executed by the same pod that serves HTTP. For most installs that's fine. Remote workers are an opt-in mechanism for moving that execution out: a separate process (or fleet of them) claims tasks from postgres and runs agent heartbeats on its own host.

This page is the operator-facing tour. The wire-level protocol detail lives at REMOTE_WORKERS.md.

When to use them

The in-process worker pool is the default. Reach for remote workers when:

• You want to scale heartbeat execution independently of HTTP. E.g., a burst of agent runs shouldn't slow down the board UI.
• claude_local agents need a beefier host. Move them to a worker box with the right CPU/RAM, leave the HTTP server tiny.
• docker agents need Docker on the worker host but you don't want Docker on the HTTP host. Tag-based routing handles this cleanly.
• You want geographic / network isolation between the HTTP face and the LLM-calling layer (e.g., HTTP in a public DMZ, workers in a private VPC with provider egress).
• Multi-region deploys where you want a worker close to a specific provider region.

When you don't need them:

• Single-host install. The in-process worker pool already scales via WORKER_POOL_SIZE.
• Low traffic. The added complexity isn't worth it.

Architecture sketch

┌──────────────────┐   HTTP / SSE     ┌──────────────────────┐
│  trimus-server   │ ───────────────→ │  user browsers       │
│   (HTTP + UI)    │                  └──────────────────────┘
└────────┬─────────┘
         │ SQL (the server owns the queue)
         ↓
┌──────────────────┐
│   postgres       │
└──────────────────┘
         ▲  HTTP: register / heartbeat / claim / complete|fail
         │
┌────────┴─────────┐    HTTP
│ trimus-worker(s) │ ─────→ LLM provider
│ (one or many)    │
└──────────────────┘

The worker is a pure HTTP client — it does NOT touch the database

trimus-worker connects only to trimus-server over HTTP. It holds no DB connection and needs no DATABASE_URL. The server owns the Postgres task queue; workers register, heartbeat, claim, and report results entirely through the worker API. They authenticate with their own bearer key (issued at registration) and hold no local state beyond the task they're currently executing.

The lifecycle

1. Register the worker. POST /api/workers/register returns an API key (shown once).
2. Worker process boots with that API key.
3. Worker heartbeats — POST /api/workers/heartbeat every few seconds to mark itself live.
4. Worker claims tasks — POST /api/workers/tasks/claim, filtered by tags. One-at-a-time.
5. Worker runs the agent's heartbeat, calling whichever adapter the agent is configured for.
6. Worker reports completion — POST /api/workers/tasks/{id}/complete or /fail.

If a worker stops heartbeating, its in-flight task is reaped after a timeout and re-claimable by another worker.

Registration is idempotent — restarts don't create duplicates

The shipped trimus-worker binary performs step 1 itself on every boot: it presents its TRIMUS_WORKER_KEY to register, and the server resumes the existing worker (same worker_id, HTTP 200, no new key) rather than inserting a duplicate row. Restarting a worker is therefore safe and idempotent — it never accumulates orphan rows. (Requires server v3.1.0+; earlier versions both 401'd all worker auth and re-registered a fresh row on every boot.)

Tags

Workers register with a list of tags (gpu, claude-cli, docker, region-us-east, etc.). Tasks carry required tags, resolved at dispatch from the agent's adapter_config.tags, then the project's, then the company's default_worker_tags (#261). A worker only claims tasks whose required tags are a subset of its own tag set.

This is how you route specific agents to specific worker fleets.

A worker's tags are server-authoritative and editable after registration: on Remote Workers (/admin/workers), click edit in a worker's Tags cell, adjust the comma-separated list, and Save (#336). The change takes effect on the worker's next claim — no restart. Because the stored set wins, TRIMUS_WORKER_TAGS in the worker's env matters only at first registration; editing it later and restarting will not change a worker's tags (retag it here instead). A claim request may only narrow a worker to a subset of its stored tags, never broaden it past them.

Setting up a remote worker

1. Register

The easy way — from the admin UI. Open Sidebar → Remote Workers (/admin/workers) and use the Register a worker form: give it a name, optional capability tags, and an optional concurrency. Trimus mints the worker and shows a one-time credentials page with:

• the api_key (displayed once — Trimus stores only its hash),
• a ready-to-paste env file with this instance's URL and the key already filled in,
• step-by-step setup instructions for the worker machine.

Workers registered this way are approved immediately — your click is the admission decision, so the approval queue below never applies to them. If you navigate away without saving the key, delete the worker row and register again. The instance-wide TRIMUS_MAX_REMOTE_WORKERS cap applies here too.

The API way. The registration endpoint is unauthenticated — anyone who can reach the server can register a worker (lock down the endpoint at the network layer if that's a concern). Total registrations are capped by TRIMUS_MAX_REMOTE_WORKERS (default 10000; over it → 503), so a flood can't exhaust rows/keys (#153).

For stronger admission control, set TRIMUS_REQUIRE_WORKER_APPROVAL=true (#210). A newly-registered worker then sits pending: it receives an inert api_key but no operating prompt, and its heartbeat/claim calls return 403 until an instance-admin approves it under the Pending approval section of /admin/workers. The bundled trimus-worker binary handles this gracefully — it logs awaiting admin approval and re-registers on a backoff until admitted, then proceeds. Once approved, the worker's next register delivers the operating prompt and it starts claiming. Rejecting a worker fences its key permanently. The flag defaults false (auto-onboard); existing workers are unaffected.

Managing the queue

Approval decisions live on the worker row (approval_status), so the /admin/workers Pending approval controls work regardless of the flag. Turning the flag off later only stops new registrations from being gated — a worker already pending stays pending (an un-reviewed worker is never silently admitted); approve or delete it from the UI. A rejected worker's process keeps polling register on a backoff until you stop it.

From the worker host or any client:

curl -s -X POST -H 'Content-Type: application/json' \
  -d '{"name":"gpu-worker-1","tags":["gpu","heavy"]}' \
  "$TRIMUS_SERVER_URL/api/workers/register"

Response:

{
  "worker_id":        "550e8400-e29b-41d4-a716-446655440000",
  "api_key":          "sk_abc123...",
  "operating_prompt": "# Trimus Remote Worker Operating Contract\n\nYou are a remote worker...",
  "prompt_version":   "1.4.0"
}

Save the api_key (a standard sk_… bearer key) — it's shown once. The bundled operating_prompt is the canonical worker contract: the bundled trimus-worker binary uses it implicitly; a custom worker should plumb it into its model's system prompt so the model knows the lifecycle, the payload shape, and the action vocabulary.

POST /api/workers/register is public (unauthenticated) — a fresh worker has no key yet, and register is what issues it. The prompt is delivered in the register response, so bootstrap needs nothing else. As of #148 the standalone GET /api/workers/operating-prompt sits behind worker auth (the full contract shouldn't be disclosable anonymously): re-fetch it after a server upgrade with your (approved) worker key — curl -H "Authorization: Bearer $TRIMUS_WORKER_KEY" $TRIMUS_SERVER_URL/api/workers/operating-prompt — or read it on /admin/workers (Copy-to-clipboard next to the worker table).

2. Run the worker

The trimus-worker binary connects back to trimus-server over HTTP and starts claiming tasks. It needs exactly two required env vars (TRIMUS_SERVER_URL + TRIMUS_WORKER_KEY) — no DATABASE_URL:

export TRIMUS_SERVER_URL="https://trimus.example.com"
export TRIMUS_WORKER_KEY="sk_..."          # from the register response
# optional:
export TRIMUS_WORKER_NAME="gpu-worker-1"   # defaults to hostname
export TRIMUS_WORKER_TAGS="gpu,heavy"      # CSV
export TRIMUS_WORKER_CONCURRENCY=2         # parallel task slots
export TRIMUS_WORKER_POLL_SEC=5            # claim-loop interval
trimus-worker

The worker logs each task it claims, the agent it's running, and the result. See environment.md for the full worker var list.

3. Verify it shows up

Sidebar → Remote Workers (instance admin). The new worker should appear with status online, last-seen timestamp updating every few seconds. (A worker with no heartbeat for ~5 minutes flips to offline and is skipped by the dispatcher until it heartbeats again.)

If it doesn't:

• Check TRIMUS_SERVER_URL is reachable from the worker host.
• Check TRIMUS_WORKER_KEY matches what registration returned.
• Check the worker's logs for auth errors.

The table shows each worker's reported Version (binary build) and Contract (operating-contract revision it was built against), with a behind badge when either is older than the server's own — the caption above the table states the server's current version and contract. A worker that doesn't report a value (an older binary, or a custom worker that omits the fields) shows —; the value populates the next time that worker registers.

Distributing work

The default in-process worker pool inside trimus-server also participates by default — it's effectively a worker with no tags, so it claims any task whose tags are empty (the default for most agents). When you add remote workers:

• Untagged tasks are first-come-first-served between in-process and remote workers.
• Tagged tasks only go to workers with matching tags.

To make the in-process pool a fallback only, you can disable it via environment (currently requires source-side change) or set WORKER_POOL_SIZE=0. The latter is the cleanest.

Operations

Adding capacity

Just register more workers and start more trimus-worker processes. No coordinator, no leader election — postgres is the source of truth.

Draining a worker

Stop the trimus-worker process. Any in-flight task is reclaimed after the timeout. New tasks bypass the now-dead worker.

Killing a stuck task

A task whose worker died mid-execution stays claimed until the heartbeat timeout reaps it (after which another worker can re-claim). To intervene sooner, update the row directly in postgres — the /api/workers/tasks/{id}/fail endpoint requires the owning worker's key (it's worker-authenticated, not admin-authenticated), so it isn't an admin override path.

Rotating, disabling, or removing a worker

Registration is idempotent: presenting an existing worker's key just resumes it (same worker_id, no new key). To issue a fresh key, register a new worker (a new identity, no auth header), point the worker process at the new key, then remove the old worker.

Manage workers from Sidebar → Remote Workers (instance admin):

• edit (Tags cell) — replace the worker's capability tags. Server- authoritative and applied on the next claim, no restart (#336; see Tags).
• Disable / Enable — a reversible fence. A disabled worker's key is rejected with 403 (the registration and its history are kept). Use it to park a misbehaving worker without re-provisioning; Enable restores it.
• Drain / Undrain — graceful decommissioning (#77): a draining worker finishes its in-flight task but is offered no new work until you Undrain. Drain → wait → Delete to retire a worker cleanly.
• Delete — permanently removes the worker and its API key. Tasks it had claimed keep their history (their worker_id is nulled).

There is no public DELETE /api/workers/{id} JSON endpoint — use the admin UI buttons above (or, for scripting, DELETE FROM remote_workers WHERE id = '…' in postgres; the key rows cascade).

Failure modes

• All workers down. Tasks queue in postgres. As soon as one worker comes up, the queue drains in claim order.
• Postgres down. Workers can't claim or report. Tasks stall. Standard postgres monitoring should catch this first.
• Worker authenticated against the wrong trimus-server. It thinks it's claiming tasks but the requests 404. Check TRIMUS_SERVER_URL.
• Network partition between worker and provider. Worker claims the task, fails to call the LLM, marks the task failed. Agent's Run tab shows the error.

Deployment topologies

Remote workers exist so you can split Trimus into a central control plane (the part you host: API, UI, and database) and a worker plane (the part that runs agent work) placed wherever the work needs to be. The two planes talk over a single outbound HTTPS channel, which lets you put the workers close to the data and behind a firewall while keeping only the control plane internet-facing.

A — All-in-one (default)

TRIMUS_ROLE=all (or an unset role): the server runs the HTTP/UI tier and the background workers in one process. Simplest to operate; every tier shares one host and one network plane. Good for a single-node install or a trusted network.

B — Central control plane + remote workers near the data

The exposure-reducing topology:

• Control plane (trimus-server + Postgres, optionally split into TRIMUS_ROLE=web and TRIMUS_ROLE=scheduler — see Kubernetes): the only internet-facing tier. It owns the database, the task queue, tenant isolation, and all human-facing surfaces.
• Worker plane (trimus-worker): deployed near the data / behind the firewall. Each worker is database-free and outbound-only — it opens no inbound ports and holds no DATABASE_URL; it reaches the control plane over HTTPS (register / heartbeat / claim / complete) and calls LLM/tool endpoints from its own network. This keeps the data-adjacent tier's attack surface minimal: nothing dials into it.
• Routing: tag workers and target agents at a worker pool so the right work runs on the right fleet (see Tags; company/project-level worker affinity resolves agent → project → company default_worker_tags, #261).

What each tier needs to reach

Tier	Inbound	Outbound
trimus-server (control plane)	HTTPS from browsers and workers	Postgres; notification providers (Slack/Telegram)
Postgres	from the control plane only	—
trimus-worker (worker plane)	none	HTTPS to TRIMUS_SERVER_URL; LLM/tool endpoints

Because the worker plane is outbound-only and DB-less, you can place it inside a restricted network segment (near sensitive data, behind a corporate firewall) without exposing that segment to inbound traffic.

Security considerations

• The worker's API key gives the bearer the right to claim and complete any task. Treat it like a service account credential.
• Workers should be on the same network plane as the providers they call (or via the same egress path).
• Workers don't need browser access; lock the human-facing trimus-server behind your usual auth, and keep workers on a separate cluster / VPC if your environment expects it.

Trust boundary

The control plane is the trusted tier — it owns the database, tenant isolation, and every human-facing surface. Keep it locked down (auth, TLS, restricted registration).

A worker is a lower-trust, outbound-only compute tier. What a compromised worker can and can't do:

• Can: claim tasks and see the payloads of tasks it claims, and report results, using its own scoped bearer key.
• Cannot: reach the database directly, browse the UI/API, or obtain standing access to other tenants' data — it has no DB connection and no session, only the worker API scoped by its key and tags.

So the blast radius of a compromised worker is bounded to the work it was already running. Rotate or disable a worker's key (see Rotating, disabling, or removing a worker) if you suspect compromise.

Where to next

• remote-worker-smoke-test.md — executable curl + jq walkthrough that validates the full protocol end-to-end. Run it after every upgrade, or when building a new worker (openclaw, openfang, hermes, custom) to confirm which step you broke.
• REMOTE_WORKERS.md — wire-level protocol reference.
• adapters.md — when each adapter type benefits from workers.
• security.md — locking down the registration endpoint.