LLM cost dashboard¶

The instance-admin LLM cost dashboard at /admin/costs answers "how much have we spent across every tenant in the last N hours or days?". It exists so you do not have to pick a company, look at its per-company costs page, and repeat the exercise for every tenant during a billing-spike investigation. Instance‑admin only.

What it surfaces¶

The dashboard aggregates LLM cost and token figures from three substrate tables — the data is already there, the page just surfaces it in one place:

chat_messages — agent replies inside the chat REPL (migration 066). Reads cost_usd, tokens_in, tokens_out, model, and provider. Rows where every cost / token column is NULL are skipped — they don't represent a billed LLM call.
heartbeat_runs — cron-driven agent ticks (migration 025). Reads cost_usd, prompt_tokens, completion_tokens, model, and provider. The token column names differ from the other two sources because the heartbeat SDK uses the vendor's terminology; the dashboard normalises them so prompt_tokens feeds the "tokens in" total.
evolution_traces — GEPA per-LLM-call learning record (migration 067). Same shape as chat_messages.

Why not just read `cost_events`?¶

Per-company cost_events is the budget pipeline's ledger. Late-bound aggregation against that ledger would inherit any label-mapping or backfill bug from the ingest path. The new dashboard reads the raw figures every LLM call wrote at the moment it ran — so if you discover the budget pipeline is missing a provider, you see the gap directly here.

Window selector¶

Three windows are available at the top of the page:

Chip	Trailing window
24h	last 24 hours
7d	last 7 days
30d	last 30 days

Default is 7d (the same as the audit insights page rationale — operators want a rolling burn view by default). Unknown values collapse to the default; the page always renders.

Tiles on the page¶

Total spend — the grand total across every source for the selected window. Total cost in USD with 4 decimal places, plus total input tokens, total output tokens, and total billed call count. A "billed call" is one row in one of the three source tables that contributed any non-NULL cost or token figure to the aggregate.
By provider — one row per distinct provider value across the three sources. Sorted by cost descending so the top spender lands at row 1. NULL providers fold under "unknown" so they don't disappear; investigate them.
By model — same shape but pivoted by model name. Useful for "what's gpt-4o costing us this week vs claude-3-5-sonnet?".
By source — chat_messages → heartbeat_runs → evolution_traces, in canonical order. This lets you see where the spend is happening (the chat surface vs the heartbeat surface vs the learning trace).

All amounts render to 4 decimal places. A $0.0001 figure is visible; a $0 cell means "no billed activity in this window for this row".

Per-company drill-down¶

The new dashboard intentionally aggregates only — it does not list per-tenant cost rows. To break a spike down by company, use the per-company page at /companies/<id>/costs which renders the cost_events ledger with the budget-policy overlay. The two pages complement each other:

/admin/costs answers "is the instance spend going up?"
/companies/<id>/costs answers "for THIS tenant, where?"

trimus-cli cost¶

For cron-style burn checks, an SSH-only shell, or any operator who prefers not to load the browser, the trimus-cli cost subcommand shares the same aggregator as the dashboard:

DATABASE_URL=postgres://... trimus-cli cost --window=24h

Flags:

Flag	Default	Notes
`--window`	`7d`	`24h` / `7d` / `30d`. Other values reject.
`--by`	`provider`	`provider` / `model` / `source`.
`--json`	off	Emit the `CostAggregate` struct as JSON.
`--quiet`	off	One-line `trimus-cost ok ...` summary.

Default table output:

💰 LLM spend — last 7 days

  Total:      $1.2345  (42 calls, 16,824 in / 8,420 out)

  By provider:
    openai      $0.8123  (28 calls)
    anthropic   $0.4222  (14 calls)

--json emits the full query.CostAggregate shape — Total, ByProvider, ByModel, BySource. Suitable for piping into jq:

trimus-cli cost --window=24h --json | jq '.Total.CostUSD'

--quiet collapses to the canonical one-line summary suitable for cron health checks:

trimus-cost ok window=24h total=$1.2345 calls=42

Composes with TRIMUS_QUIET=1 as a fleet-wide default (same pattern as backup-stats).

Requires $DATABASE_URL. Unknown / malformed --window and --by values reject loudly rather than silently collapsing to a default — a cron caller passing the wrong value should fail visibly.

Spend tile on /admin/operations¶

/admin/operations carries a compact "LLM spend (24h)" tile next to the Deploy / Environment / Disk cards. It summarises the trailing 24-hour spend from the same aggregation the full dashboard uses — so the two surfaces always agree to the decimal.

The tile shows:

Total USD spend over the last 24 hours (4 decimal places).
Call count contributing to the total.
Top provider (highest-spend row in the same window).
A drill-down link to /admin/costs?window=24h so the operator lands on the same window the tile summarised.

A left-border color band signals the spend level:

Band	Default trigger	Tunable via
green	`< $10`	`TRIMUS_OPS_SPEND_WARN_USD`
yellow	`$10 to <$100`	both
red	`>= $100`	`TRIMUS_OPS_SPEND_CRITICAL_USD`

Boundary policy: ties land in the more alarming band (a spend of exactly $warn renders yellow, a spend of exactly $critical renders red). Operators on a busier instance dial the thresholds up via the env vars; values are captured at server boot, so a change takes effect on the next restart. Malformed / negative values fall back to the defaults so the tile always renders.

When the underlying AggregateCosts call errors (DB hiccup, schema-skew during a migration window), the tile renders with Status="unavailable" and "—" placeholders instead of failing the entire /admin/operations page.

Adding new cost sources¶

If you wire a new substrate table that records LLM cost figures, add it to costSourceTables() in internal/db/query/costs.go. The dashboard picks it up automatically — both the per-source tile and the by-provider / by-model rows fold the new source in. The operations-panel spend tile also picks the new source up automatically since it reads the same aggregator.

The SQL fragment in each source returns a normalized shape:

(provider TEXT NULL, model TEXT NULL,
 tokens_in BIGINT, tokens_out BIGINT,
 cost_usd float8, call_count BIGINT)

— so the reducer doesn't need to know the underlying column names. Map the source's token columns into the tokens_in / tokens_out aliases inside the SQL fragment.

Per-company monthly budgets and alerts¶

An alerting layer sits on top of the dashboards. Each company can have a single row in company_cost_budgets (migration 127) with a monthly USD ceiling and two percentage thresholds (warning + critical). The cost_budget_alerter worker runs once per hour, aggregates month-to-date spend for every budgeted company across cost_events, chat_messages, heartbeat_runs, and evolution_traces, and dispatches a Slack + Telegram DM to every instance admin with a configured pairing when the spend crosses either threshold.

Setting a budget¶

The form lives on the company dashboard. When no budget is configured, an instance admin sees an inline "Monthly USD / Warning % / Critical %" form on the dashboard's LLM Cost Budget tile. Submitting POSTs to:

POST /admin/companies/{companyId}/budget
  monthly_usd  = 1000.00     # >= 0
  warning_pct  = 80          # 1..100
  critical_pct = 100         # 1..100, >= warning_pct

Validation is enforced both at the form layer (400 for bad inputs) and at the DB layer (CHECK constraints on the company_cost_budgets table). Subsequent submissions upsert the existing row in place — there's exactly one budget per company.

Read-back is on the same tile: the dashboard renders month-to-date spend, the bar, the band (ok / warning / critical), and the current configured thresholds. The bar's color matches the band so an operator skimming the dashboard catches over-budget companies without reading numbers.

Alerting behaviour¶

The worker's posture mirrors the GEPA auto-promote notifier: every DM dispatch is best-effort, every recipient is an instance admin — resolved from the TRIMUS_INSTANCE_ADMINS env allowlist (#142, not the users.instance_role column) — who is is_active=true and has at least one configured channel pairing. Operators without a pairing are silently skipped (NOT an error — a fresh instance is the expected zero state).

Re-fire is gated by last_warned_at / last_critical_at on the budget row. The worker only re-dispatches a level when its timestamp is NULL or older than 30 days. The 30-day window deliberately doesn't coincide with calendar boundaries — a budget written on the 15th gets one warning per month starting on its anniversary, not on the 1st.

Critical wins over warning when both are eligible — an over-budget company gets one DM at the critical level per tick, not two.

Environment knobs¶

Env var	Default	Notes
`TRIMUS_COST_BUDGET_ALERTER`	unset	Set to `disable` to opt out of the worker.
`TRIMUS_COST_BUDGET_ALERTER_INTERVAL_HOURS`	`1`	1..24. Lower values raise DB load.

When opted out the worker logs once and returns. The budget rows remain in the database; flipping the env back to default resumes alerting.

Checking the budget from the chat REPL¶

The /budget slash command in trimus-cli chat lets operators don't have to open the dashboard to see the current state. It fires a single GET /api/companies/{companyId}/budget/status, which returns the same numbers the dashboard tile renders:

> /budget
💰 Budget status (June 2026):
  Monthly budget:  $50.00
  Spent so far:    $12.34 (24.7%)
  Warning at:      $40.00 (80%)
  Critical at:     $50.00 (100%)
  Status:          OK

The Status label turns yellow at the warning band and red at the critical band when stdout is a TTY. When the company has no budget row configured, /budget prints a one-line hint pointing the operator at the /admin/companies/{id}/budget form instead of zero-padded numbers.

The endpoint is read-only and viewer+ gated (same access posture as /companies/{id}/costs). The REPL only needs the company ID it already cached at session bootstrap — no extra config knobs.

What the alerter does NOT do¶

It does not refuse cost events at the engine layer — a company can keep spending past its critical threshold. The alerter nudges; it does not throttle.
It does not surface the alert on the dashboard itself as a "danger banner" — the existing per-company budget tile already flips to the critical color band, which is the in-page signal.
It does not write a row into the legacy budget_incidents table (migration 069). That table belongs to the per-feature throttling pipeline and company_cost_budgets is a separate knob with a separate alerter.