Skip to content

Monitoring

What to watch on a running trimus-server, and how to wire it into your observability stack. Three surfaces: health probes (/healthz, /readyz), Prometheus metrics (/metrics), and structured logs (stdout JSON). The /admin/operations panel ties them together for a human.

Liveness vs readiness

Trimus exposes orchestrator‑style probes. Use the right one for the right job:

Endpoint Format Semantics
/healthz JSON Liveness — 200 whenever the HTTP server responds. Body {status, version, commit}.
/readyz JSON Readiness — 200 ready / 503 not‑ready, with a per‑check breakdown.
/healthz/live text/plain Liveness alias — always 200, body OK.
/healthz/ready text/plain Readiness alias — 200 ready / 503 not ready: <reasons>.
/api/health JSON Legacy DB‑ping health ({status, db}). Prefer /healthz + /readyz.

Wire the load balancer to /readyz, the liveness probe to /healthz

/healthz says "the process is up" — restart on failure. /readyz says "this node should take traffic" — drain on 503. The text/plain aliases emit a one‑line body that's cheap to log in HAProxy/ALB access logs; the JSON variants are the truth source for dashboards that want per‑check detail.

curl -sS http://localhost:3100/healthz
# {"status":"alive","version":"v3.0.0","commit":"abc1234"}

curl -sS http://localhost:3100/readyz | jq '.status, .checks[].name'

What /readyz checks

/readyz runs a set of checks and returns 503 if any one escalates to not‑ready. Each carries a duration_ms, and the response ends with a recent_check_timings map (p50/p95/p99/count per check). The rows:

Check Not‑ready when Threshold env (default)
db_ping Postgres unreachable, or pool utilisation over threshold TRIMUS_READYZ_POOL_UTILIZATION_PCT (90.0)
workers (aggregate) + worker:<name> a worker's error rate over threshold (min >10 ticks) TRIMUS_READYZ_WORKER_ERROR_RATE_PCT (50.0)
channel:<name> a configured Slack/Telegram channel hasn't seen upstream traffic TRIMUS_READYZ_CHANNEL_STALENESS_SEC (60)
disk root‑fs usage over threshold, or Statfs failed TRIMUS_READYZ_DISK_PCT (90.0)
backup last successful backup older than threshold (or none recorded) TRIMUS_READYZ_BACKUP_MAX_AGE_HOURS (36)
(any check) recent p99 over threshold with ≥10 samples TRIMUS_READYZ_CHECK_P99_MS (1000)

Out‑of‑range / unparseable threshold values silently fall back to the default; each is read once at handler construction so probes stay fast.

db_ping — pool stats

The db_ping row carries pgxpool.Stat() fields: pool_acquired, pool_total, pool_max_open, pool_idle, and pool_utilization_pct (acquired / max × 100). MaxConns is the honest denominator — a ticking‑but‑saturated pool is a classic Postgres bottleneck, so the probe drains the node before queries start timing out.

workers — per‑worker error signal

Each worker:<name> row carries optional ticks, errors, and error_rate_pct. A worker that's ticking on schedule but failing every tick flips to not‑ready once its error rate crosses TRIMUS_READYZ_WORKER_ERROR_RATE_PCT. The text/plain alias contributes worker:<name> error rate <X.X>% to the not‑ready body.

There is a boot‑grace window: with ticks == 0 and uptime under 2× the worker interval, the aggregate workers row reports OK so a fresh boot doesn't flap.

channel:<name> — chat‑bridge reachability

When Slack (Socket Mode) and/or Telegram (long‑poll) are configured, /readyz emits one row per channel from the channel‑health registry:

{ "name": "channel:slack",    "ok": true,  "detail": "healthy (last_poll=14s ago)" }
{ "name": "channel:telegram", "ok": false, "detail": "no successful poll for 5m20s; threshold 1m0s" }

Slack health = any Socket Mode frame (Hello/keepalive count); Telegram health = any successful getUpdates (an empty [] counts). Unconfigured channels are omitted. A down channel blocks readiness like a stuck worker — for chat‑only customers, a broken bridge means their agents are unreachable, so drain user traffic.

disk — root‑filesystem pressure

{ "name": "disk", "ok": true,
  "detail": "root usage 47.0% < threshold 90.0% (free=18.2 GiB of 29.0 GiB; journal=1.4 GiB; backups=480.1 MiB)",
  "pct_used": 47.0, "free_bytes": 19575808000, "journal_bytes": 1503238553, "backup_dir_bytes": 503316480 }

A failed root Statfs is treated as not‑ready ("signal missing beats all‑clear with zero"). The 90% default sits below the 95% critical band on /admin/operations so the orchestrator drains the host while in‑flight requests can still complete. See Disk hygiene below.

backup — backup freshness

The backup row reads GetLastSuccessfulBackupRun:

{ "name": "backup", "ok": true,
  "detail": "healthy (last success 2.1h ago; threshold 36h)",
  "since_hours": 2.1, "threshold_hours": 36.0 }

A DB/query error or "no successful backup ever" is not‑ready (missing signal is a problem, not an all‑clear). The 36 h default is 1.5× a 24 h cron. Operators running external backup tooling (Litestream, host pg_dump) can opt out with TRIMUS_READYZ_BACKUP_CHECK=disable — the row is still emitted, OK, with a "disabled via env" detail.

Prometheus metrics

GET /metrics

Unauthenticated — gate it at the network/proxy layer. Notable series:

Metric Meaning
staple_disk_used_pct, staple_disk_free_bytes, staple_journal_bytes host disk gauges (cached 30 s; prefer these to scraping /readyz JSON for alerts)
staple_backup_last_success_timestamp_seconds last successful backup time (alert on staleness)
staple_readyz_check_duration_ms{check,pct="50\|95\|99"} per‑check probe latency
staple_readyz_check_samples{check} sample count behind the latency gauge

Worker tick/error counters come from internal/worker/metrics.go; the recent‑error ring is internal/alerts/ring.go.

OpenTelemetry traces

Set OTEL_EXPORTER_OTLP_ENDPOINT to enable OTLP/HTTP trace export; the standard OTel vars are honoured:

OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.example.com:4318"
OTEL_SERVICE_NAME="trimus-server"
OTEL_RESOURCE_ATTRIBUTES="environment=production,version=v3.0.0"

Traced: HTTP handlers (route + method), pgx queries, LLM provider calls (provider + model + token attributes), and heartbeat lifecycle events. Custom business metrics (agents per company, cost per company) are not emitted — build those from Postgres or instrument and PR back.

Logs

trimus-server logs structured JSON to stdout via log/slog (LOG_LEVEL = debug/info/warn/error). A RedactingHandler (internal/middleware/redact.go) scrubs secrets and keys from log lines.

{"time":"2026-05-31T16:32:11Z","level":"INFO","msg":"heartbeat run started","agent_id":"abc","run_id":"def","issue_id":"ghi"}

Ship stdout from wherever you run it — Docker/k8s capture it; systemd sends it to the journal (cap the journal — see Disk hygiene); plain‑process deployments should redirect + logrotate.

The /admin/operations panel

Instance‑admin only. One place to watch the live instance:

  • Recent alerts — the last 10 worker DM alerts (cost_budget_alerter, backup_health, gepa_notifier), severity‑badged. In‑process ring (zeroed on restart); for durable records use /admin/audit or journalctl.
  • Background workers table — per‑worker tick counters, error rate, last/next‑expected tick, mode + last action. Click a worker for the drill‑down (/admin/operations/workers/{name}) with its recent‑error ring; Tick now forces an immediate sweep (handy for the hours‑long retention/verify/backup workers — coalesces if double‑clicked).
  • Instance flags — effective values + toggles, with flag‑change history. Several env knobs (subagent rate, compression threshold, embeddings, local‑adapter gate) are editable here without a restart.
  • Disk / spend / backup / deploy tiles.
  • Auto‑refresh toggle (30 s / 60 s / 5 min, persisted to localStorage) so a long migration or a stuck worker can be watched without hammering reload.

What to alert on

Critical (page someone)

  • /readyz returns 503 for >1–2 min — inspect the failing check(s): db_ping, workers, channel:*, disk, or backup.
  • /healthz 5xx / no response — process down.
  • Postgres down — a separate alert from your DB layer; symptoms also surface in db_ping.
  • Disk full on root / TRIMUS_HOME / TRIMUS_STORAGE_DIR — agents fail with ENOSPC; the disk check should fire first.
  • All workers stuck — the workers aggregate or "no run started in N minutes".

Important (business hours)

  • Climbing 5xx or heartbeat‑failure rate (often provider auth: revoked key, suspended account).
  • Cost burn over policy — set a company budget; the cost_budget_alerter DMs admins and the backup/budget signals surface in the UI. See costs.md.
  • Approval queue backlog growing (humans not keeping up).
  • A stale backup check that hasn't yet crossed the readiness threshold.

Informational

New deploy succeeded; new company/agent; migration applied.

Heartbeat‑run debugging

When a single run fails, look at the Issue Timeline (run.failed with the error inline), the agent's Runs tab (status/model/cost/ duration), the run detail page (streaming event log + error), and the server logs (search the run_id). For broader patterns:

-- Failure rate by adapter type, last 24h
SELECT a.adapter_type, count(*) AS runs,
       sum(CASE WHEN hr.status = 'failed' THEN 1 ELSE 0 END) AS failures
FROM heartbeat_runs hr
JOIN agents a ON a.id = hr.agent_id
WHERE hr.started_at > now() - interval '24 hours'
GROUP BY a.adapter_type;

SSE liveness

Browsers on the board connect via GET /api/live/events; the UI shows a connection indicator (red = SSE dropped, usually an LB idle timeout). Bump your proxy's read timeout for /api/live/events to ~5 minutes to avoid false alarms on long‑lived connections.

Postgres‑side monitoring

Watch the usual signals: pg_stat_activity connection count (Trimus holds a small pool per server process), pg_stat_statements for slow queries (issue‑list / dashboard rollups should dominate — anything else is worth a look), and disk usage (see storage.md and database.md). Autovacuum handles VACUUM/ANALYZE; trust it unless you see bloat. Trimus does not read from replicas.

Disk hygiene

The /readyz disk check drains the host at TRIMUS_READYZ_DISK_PCT (default 90%). On small hosts the recurring growth offenders are usually not Trimus itself:

  • systemd journal grows unbounded by default — cap it:
# /etc/systemd/journald.conf.d/trimus-cap.conf
[Journal]
SystemMaxUse=300M
SystemKeepFree=1G
MaxRetentionSec=2week

then systemctl restart systemd-journald.

  • Go build cache (~/.cache/go-build) is repopulated on every rebuild — safe to drop. Docker layers accumulate from old image builds (docker image prune, docker builder prune).
  • Backups under TRIMUS_BACKUP_DIR — bounded by TRIMUS_BACKUP_RETENTION_DAYS (default 30) once the retention worker runs. See backup-and-restore.md.

A weekly maintenance cron (Go cache clean + Docker prune + journal vacuum) keeps these bounded; running containers and in‑use images are never touched.

What's NOT a great signal

  • Process CPU / memory — Trimus is IO‑bound on Postgres and LLM calls; spikes usually mean a stuck loop. Trace before alerting.
  • Absolute query/request rate — varies with user activity. Alert on derivatives ("doubled in 5 minutes"), not raw counts.

Where to next