Backup and restore¶

The complete operator runbook for backing up and restoring a full Trimus instance. Treat this as canonical — database.md has short backup notes that point back here.

If you're investigating a production incident, jump to Operator actions → Restore from a local backup or Restore from S3 below.

Table of contents¶

Overview
Architecture
First-time setup
Daily operations
Operator actions
Manual backup (CLI)
Manual backup (UI)
Restore from a local backup
Restore from S3
Verify a backup file
List backups
Detect drift / orphans
Partial restore
Backup verification (test restore)
Monitoring
Troubleshooting
Reference
Environment variables
CLI commands
backup_runs audit schema

1. Overview¶

What the backup system does¶

Produces encrypted Postgres dumps via pg_dump and the Trimus vault.
Writes optional copies to S3-compatible object storage.
Verifies each new dump end-to-end before declaring success.
Sweeps aged-out local files on a configurable retention window.
Re-verifies long-lived files in the background to catch bit rot.
Audits every operation (create / verify / reconcile / prune) into the backup_runs table for forensic review.
Surfaces the lifecycle on /admin/operations (latest backup widget, retention preview, backup worker tile) and /admin/audit?source=backup_runs (full row stream).

What the backup system does NOT do¶

No continuous WAL streaming. Each backup is a point-in-time dump; for ≤-minutes data loss SLAs use archive_command + a tool like pgBackRest or wal-g alongside Trimus's dumps.
No multi-version restore. The dump format is Postgres custom (-Fc); restoring requires the same major Postgres version (or a compatible newer one) and the same Trimus migration baseline.
No per-company restore via SQL. Tenant isolation is enforced at the application layer, not at the schema level — use the portability bundle for single-company recovery (see Partial restore).
No backup of TRIMUS_HOME or TRIMUS_STORAGE_DIR via the built-in workers. Those trees are filesystem state and need separate tarballs / snapshots — covered in First-time setup.
No encryption-key escrow. Lose TRIMUS_ENCRYPTION_KEY and every secret column is unrecoverable. See Encryption key contract.

What you must back up¶

A complete Trimus backup has four components. Missing any one turns the others into either dead data or unreadable ciphertext.

#	Component	Default location	Encrypted contents
1	Postgres database	varies (`$DATABASE_URL`)	Yes — secret values, provider API keys, custom headers, webhook signing secrets, OAuth tokens
2	`TRIMUS_HOME` directory	`$HOME/.trimus/`	Per-company / per-agent files, skills, project workspaces
3	`TRIMUS_STORAGE_DIR` directory	`./storage`	Uploaded issue attachments and company logos
4	`TRIMUS_ENCRYPTION_KEY`	environment variable on the server	Master key for #1's encrypted columns

Encryption key contract¶

⚠️ The encryption key is the single point of failure. Back up Postgres but lose the key, and every secret, provider API key, and webhook signing secret in the dump is unrecoverable. They're AES-GCM-encrypted with no recovery path.

Encrypted at rest with TRIMUS_ENCRYPTION_KEY:

company_secret_versions.encrypted_value — every operator secret
company_providers.api_key — per-company LLM provider credentials
company_providers.custom_headers — operator-supplied request headers
routine_triggers.signing_secret — webhook signing secrets
(Future encrypted columns will be documented here.)

Not encrypted by Trimus — relies on database / filesystem encryption-at-rest:

Issue bodies, comments, run events, activity logs (Postgres)
File attachments in TRIMUS_STORAGE_DIR (filesystem)
Files under TRIMUS_HOME (filesystem)

This is why backing up Postgres alone is not sufficient.

2. Architecture¶

The backup system is one CLI + four background workers + one audit table.

CLI: `trimus-cli backup`¶

The canonical entry point. The CLI runs pg_dump, wraps the output through the vault encryption envelope (TRIMUS_BACKUP_v1 magic + chunked AES-GCM with a wrapped per-file DEK), writes the local file, optionally uploads to S3, inline-verifies the result, and writes a backup_runs row. Every other backup pathway in the system ultimately calls internal/backup.Run — the same code the CLI exercises.

Background workers¶

All four workers register in the worker tick table and are visible on /admin/operations and as staple_worker_ticks_total{name="..."} on /metrics.

Worker	Cadence env	Default	Purpose
`backup_cron`	`TRIMUS_BACKUP_CRON_INTERVAL_HOURS`	24h	Creates a new encrypted dump on the configured cadence using the same pipeline as the CLI. Tags `kind='cron'`. Auto-disables when `TRIMUS_ENCRYPTION_KEY` is unset or `pg_dump` is not in `$PATH`. Hard-disable with `TRIMUS_BACKUP_CRON=disable`.
`backup_verify`	`TRIMUS_BACKUP_VERIFY_INTERVAL_HOURS`	24h	Picks the newest `.dump.enc` by mtime and streams it through the vault decrypt pipeline into `io.Discard`. Catches bit rot + KEK/DEK rotation issues on cold archives. Tags `kind='cron_verify'`. Disabled automatically when `TRIMUS_ENCRYPTION_KEY` is unset.
`backup_retention`	hourly	hourly	Removes local backup files older than `TRIMUS_BACKUP_RETENTION_DAYS`. Top-level only (no recursion). Stamps `pruned_at` on matching `backup_runs` rows. Set retention to `0` to disable the worker.
`backup_reconcile`	`TRIMUS_BACKUP_RECONCILE_INTERVAL_HOURS`	6h	Heals drift between the on-disk inventory and the `backup_runs` audit table. Inserts rows for orphan-on-disk files (`kind='reconciled'`) and stamps `pruned_at` on orphan-in-db rows. Hard-disable with `TRIMUS_BACKUP_RECONCILE=disable`.

Audit table: `backup_runs`¶

One row per operation (CLI create, manual UI create, cron create, cron verify, reconciliation). The kind column discriminates the source; the outcome, verify_outcome, pruned_at, and error_message columns capture the lifecycle. Full schema in backup_runs audit schema.

The dashboard at /admin/audit?source=backup_runs filters the consolidated audit feed to just backup activity.

3. First-time setup¶

0 · Confirm where each component lives¶

# 1. Postgres connection string — trimus-server reads $DATABASE_URL.
echo "$DATABASE_URL"

# 2. TRIMUS_HOME — look it up via the server env or the unit file.
echo "$TRIMUS_HOME"

# 3. TRIMUS_STORAGE_DIR — same.
echo "$TRIMUS_STORAGE_DIR"

# 4. TRIMUS_ENCRYPTION_KEY — the value, not the var name.
echo "$TRIMUS_ENCRYPTION_KEY" | wc -c   # base64(32 bytes) → 45 chars

If any of those are empty in the server's environment but data exists, the instance was likely started with hardcoded defaults or shell-script wrappers. Find them before configuring backups.

1 · Install `pg_dump`¶

The backup pipeline shells out to pg_dump for the Postgres custom-format dump. Install postgresql-client (or the matching package for your distro / container). The backup_cron worker auto-disables itself with a journal warning when pg_dump is not in the server's $PATH:

backup cron disabled reason="pg_dump not in PATH (install postgresql-client)" dir=/var/backups/trimus

2 · Provision the encryption key¶

TRIMUS_ENCRYPTION_KEY is the most fragile part of the system. Generate it once and store it somewhere durable AND access-controlled before producing any backups:

# Generate a base64-encoded 32-byte key.
openssl rand -base64 32

# Store in your secrets manager (1Password, AWS Secrets Manager,
# Vault, etc.) AND write into the server environment (systemd
# unit, docker-compose env, IRSA, whatever your deployment uses).

Treat the key with the same care as your TLS private keys. The backup_cron and backup_verify workers auto-disable when this env var is unset — a fresh deploy without a vault should not silently produce plaintext backups.

3 · Configure the backup directory¶

# All four workers share TRIMUS_BACKUP_DIR. Top-level only — no
# recursion. The directory must exist before the workers tick.
sudo mkdir -p /var/backups/trimus
sudo chown trimus:trimus /var/backups/trimus
sudo chmod 700 /var/backups/trimus

export TRIMUS_BACKUP_DIR=/var/backups/trimus

The /admin/operations retention preview tile renders a red dir missing badge if the configured path doesn't exist at server start; the retention worker logs a Warn line and skips its sweep in the same condition.

4 · (Optional) Configure S3 for off-host copies¶

trimus-cli backup can ship the encrypted dump to an S3-compatible bucket after it lands on disk. The local file is kept — the upload is an additional copy, not a move — so a transient S3 failure never costs you the dump.

The sink works with AWS S3, minio, Cloudflare R2, Backblaze B2, Wasabi, and anything else that speaks the S3 wire protocol.

Credentials come from the AWS SDK default chain — same as the Bedrock provider. Set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY in the environment, attach an instance role, use IRSA, or drop a profile in ~/.aws/credentials. Trimus doesn't introduce a separate credential channel.

Configuration¶

Variable	Default	Notes
`TRIMUS_BACKUP_S3_BUCKET`	—	Required to enable. Bucket name.
`TRIMUS_BACKUP_S3_PREFIX`	`trimus-backups/`	Key prefix. Include trailing `/` for a folder.
`TRIMUS_BACKUP_S3_REGION`	`us-east-1`	AWS region (or whatever your S3-compatible service expects).
`TRIMUS_BACKUP_S3_ENDPOINT`	(empty)	Custom endpoint URL. Leave empty for AWS S3.
`TRIMUS_BACKUP_S3_PATH_STYLE`	`false`	`true` forces path-style addressing (required for minio + most self-hosted gateways).

Each variable has a matching CLI override: --s3-bucket=..., --s3-prefix=..., --s3-region=..., --s3-endpoint=..., --s3-path-style=true|false. Pass --no-s3 to skip the upload for a single invocation even when the env vars are set.

AWS S3¶

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export TRIMUS_BACKUP_S3_BUCKET=ops-trimus-backups
export TRIMUS_BACKUP_S3_REGION=us-west-2
export TRIMUS_BACKUP_S3_PREFIX=prod/trimus/

minio (or any self-hosted S3 gateway)¶

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export TRIMUS_BACKUP_S3_BUCKET=trimus-backups
export TRIMUS_BACKUP_S3_ENDPOINT=https://minio.internal:9000
export TRIMUS_BACKUP_S3_PATH_STYLE=true     # required for minio
export TRIMUS_BACKUP_S3_REGION=us-east-1    # minio ignores it, SDK requires *some* value

Cloudflare R2¶

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export TRIMUS_BACKUP_S3_BUCKET=trimus-backups
export TRIMUS_BACKUP_S3_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com
export TRIMUS_BACKUP_S3_REGION=auto         # R2 expects "auto"

What gets uploaded¶

Only the encrypted dump file (.dump.enc). The TRIMUS_HOME and TRIMUS_STORAGE_DIR trees still need to be shipped off-host via your existing tooling — see Filesystem trees.

The upload is a single PUT for small dumps and a multipart upload for anything over 5 MiB; the SDK handles the split automatically.

5 · Back up filesystem trees (outside the built-in workers)¶

TRIMUS_HOME and TRIMUS_STORAGE_DIR are NOT touched by the backup workers. Snapshot them via your existing ops tooling:

# tar with -p preserves permissions (skills' executable bits).
tar -cpzf /backups/trimus-home-$(date +%Y%m%d-%H%M%S).tar.gz \
    -C "$(dirname "$TRIMUS_HOME")" "$(basename "$TRIMUS_HOME")"

tar -cpzf /backups/trimus-storage-$(date +%Y%m%d-%H%M%S).tar.gz \
    -C "$(dirname "$TRIMUS_STORAGE_DIR")" "$(basename "$TRIMUS_STORAGE_DIR")"

If these directories live on a network filesystem (NFS, EFS) and you already have filesystem-level snapshots, you can rely on those instead — but verify your snapshot tool captures POSIX permissions and symlinks. Skills directories sometimes contain executable scripts; losing the executable bit breaks them silently.

6 · (Optional) One-shot full backup script¶

For combined Postgres + filesystem + key snapshots, put it all together:

#!/usr/bin/env bash
# trimus-backup.sh — full backup of a Trimus instance.
set -euo pipefail

source /etc/trimus/trimus.env

STAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR=/backups/trimus/$STAMP
mkdir -p "$BACKUP_DIR"
chmod 700 "$BACKUP_DIR"

# 1. Encryption key (most fragile first).
echo "$TRIMUS_ENCRYPTION_KEY" > "$BACKUP_DIR/trimus_encryption_key.key"
chmod 600 "$BACKUP_DIR/trimus_encryption_key.key"

# 2. Postgres — prefer the Trimus CLI so the run is audited.
trimus-cli backup
# The encrypted dump lands in $TRIMUS_BACKUP_DIR; the manifest
# below captures the size + version for forensic purposes.

# 3. TRIMUS_HOME.
tar -cpzf "$BACKUP_DIR/trimus_home.tar.gz" \
    -C "$(dirname "$TRIMUS_HOME")" "$(basename "$TRIMUS_HOME")"

# 4. TRIMUS_STORAGE_DIR.
tar -cpzf "$BACKUP_DIR/trimus_storage.tar.gz" \
    -C "$(dirname "$TRIMUS_STORAGE_DIR")" "$(basename "$TRIMUS_STORAGE_DIR")"

# 5. Manifest.
cat > "$BACKUP_DIR/MANIFEST.txt" <<EOF
Trimus backup created: $STAMP
Hostname: $(hostname)
Trimus version (from binary): $(trimus-server -version 2>/dev/null || echo unknown)
Postgres version: $(psql "$DATABASE_URL" -tAc 'SELECT version()' 2>/dev/null || echo unknown)
trimus_home.tar.gz: $(stat -c %s "$BACKUP_DIR/trimus_home.tar.gz") bytes
trimus_storage.tar.gz: $(stat -c %s "$BACKUP_DIR/trimus_storage.tar.gz") bytes
EOF

Run from cron / systemd timer. Ship the directory off-host to your backup store (S3, Backblaze, an offsite rsync target).

4. Daily operations¶

Once setup is complete, the system runs on its own. Operators can ignore most of it most of the time.

What runs automatically¶

Worker	Default cadence	What you see
`backup_cron`	24h	A new `*.dump.enc` lands in `$TRIMUS_BACKUP_DIR` with a matching `backup_runs` row (`kind='cron'`). When S3 is configured, the same row carries the `s3_bucket` / `s3_key` columns.
`backup_verify`	24h	The newest existing dump is re-decrypted into `io.Discard`. Successful re-verify writes a `kind='cron_verify'` row; failures (bit rot, KEK rotation) also write a row with `verify_outcome='verification_failed'`.
`backup_retention`	1h	Files older than `TRIMUS_BACKUP_RETENTION_DAYS` are deleted from local disk. The matching `backup_runs` rows get `pruned_at` stamped. S3 lifecycle rules are NOT touched — that's your bucket's responsibility.
`backup_reconcile`	6h	The on-disk inventory and the `backup_runs` table are cross-checked. Drift in either direction produces a `kind='reconciled'` row or stamps `pruned_at` on an orphan-in-db row.

What to glance at periodically¶

/admin/operations — backup widget shows the most recent successful backup's age, encryption flag, size, duration. The retention preview tile shows the next prune count. The backup workers tile shows the health of all four workers.
/admin/audit?source=backup_runs — the audit dashboard filtered to backup activity. Read this when an alert fires or when you're investigating a missed backup.

A typical operator interaction with the system is once-a-month: the weekly Slack / email summary mentions the backup widget is green, and you move on.

5. Operator actions¶

Manual backup (CLI)¶

The canonical way to take a backup on demand:

DATABASE_URL=postgres://... TRIMUS_ENCRYPTION_KEY=... \
    trimus-cli backup
# → Backup complete:
#     file: ./trimus-backup-20260529T140112.dump.enc
#     size: 142850304 bytes
#     encryption: enc:v2 (dek=01HM...)
#     uploaded: s3://ops-trimus-backups/prod/trimus/trimus-backup-...dump.enc (136.23 MiB)
#     s3-etag: 9f86d081884c7d659a2feaa0c55ad015-29
#
# Verifying backup ...
#   verified: dek=01HM...  chunks=0  decrypted=126.30 MiB  in 4.21s

Auto-verify runs after the local file write AND after the S3 upload (if any). On verify failure the local file is DELETED. It's corrupt by definition; leaving it on disk would just confuse the next retention sweep into thinking a recent backup exists. The exit code is 1. If an S3 upload happened we print the URI but refuse to delete the remote object — operators may have S3 versioning or lifecycle rules on the bucket that we'd interfere with. The remote-cleanup decision is yours.

Skipping auto-verify¶

Two escape hatches for operators who already validate backups elsewhere:

# Per-invocation:
trimus-cli backup --no-verify

# Fleet-wide standing policy:
TRIMUS_BACKUP_NO_VERIFY=1 trimus-cli backup

Truthy TRIMUS_BACKUP_NO_VERIFY values are anything strconv.ParseBool accepts (1, true, yes). A typo (TRIMUS_BACKUP_NO_VERIFY=on) is treated as "verify normally" — better to over-verify than silently skip on a misconfigured env file.

--no-encrypt backups (EMERGENCY-USE-ONLY plaintext path) intentionally skip auto-verify: there's nothing to decrypt-verify, and validating a custom-format dump structurally would mean running pg_restore --list which is heavier than what fits in the inline check.

Cron usage (--quiet)¶

The CLI prints a multi-line "Backup complete:" summary plus a per-table row-count tally so an operator at a terminal can eyeball the result. In a cron job that output pollutes log digests; for those callers, --quiet collapses it to one line:

# In /etc/cron.d/trimus-backup
0 2 * * * trimus-cli backup --quiet >> /var/log/trimus-backup.log 2>&1

Quiet semantics across the backup family:

trimus-cli backup --quiet → suppresses the preamble, the "Backup complete:" multi-line block, and the row-count tally. Emits one line on success: trimus-backup ok <path> <size-bytes> <duration>.
trimus-cli restore --quiet --yes <file> → suppresses the "Restoring …", "(this will DROP existing data)" lines and the row-count tally. Emits one line: trimus-restore ok <source> <duration>.
trimus-cli backup-verify --quiet <file> → suppresses the decorative ✓ verified: … line. Emits one line on success: trimus-verify ok <source> dek=<id> chunks=<n> in <duration>.
trimus-cli backup-list --quiet [--dir=…] [--s3] → suppresses the table (and the --json array — quiet wins). Emits one line: trimus-backup-list ok <N> entries.

Errors always go to stderr regardless of --quiet. A cron operator running --quiet still sees the one-line "this is what broke" message in the log when something fails — quiet only suppresses the operationally-decorative success output.

Fleet-wide default via TRIMUS_QUIET=1:

# /etc/trimus/trimus.env on a cron-driven instance
TRIMUS_QUIET=1

TRIMUS_QUIET honours the same strconv.ParseBool rule as TRIMUS_BACKUP_NO_VERIFY — 1, true, TRUE, t are truthy; empty, unset, 0, false, on, yes, parse errors fall through to off. The "operator typo doesn't silently mute a release runbook" posture matches the TRIMUS_BACKUP_NO_VERIFY convention.

An operator with TRIMUS_QUIET=1 standing in the env can still run a one-time verbose invocation by passing --quiet=false:

# Investigate today's backup output without unsetting the env var:
TRIMUS_QUIET=1 trimus-cli backup --quiet=false

That's the standard Go-flag "explicit beats default" rule applied to quiet mode.

Manual backup (UI)¶

Instance admins can trigger a backup from /admin/operations by clicking Run backup now in the backup widget. The button hits POST /admin/operations/backup/run and the page redirects back with a green / red flash banner.

The UI trigger:

Is instance-admin gated. A board-admin (company-scoped) cannot trigger an instance-wide backup.
Reuses the same internal/backup.Run pipeline as the CLI and the cron worker, so the encryption envelope, the audit row, the optional S3 upload, and the inline verify all behave identically.
Tags the resulting backup_runs row with kind='manual' and actor_user_id=<admin's user id> so the dashboard can attribute the backup back to the operator who triggered it.
Refuses to run when the vault is nil. Silently producing a plaintext backup from a UI button would surprise operators; the flash points them at trimus-cli backup --no-encrypt if plaintext is intentional.
Is wrapped in a 5-minute context.WithTimeout. A timeout maps to a dedicated flash that recommends the CLI (where the operator's terminal owns the run lifetime past the HTTP server's write timeout). A non-timeout failure carries the error string and a pointer to /admin/audit?source=backup_runs.

Limitations:

Synchronous. A multi-GB dump that exceeds the 5-minute timeout surfaces as a timeout flash. The recommended fallback for that case is trimus-cli backup from the host, where the operator's terminal owns the run lifetime.
No streaming progress yet. A future x release could promote this to a background goroutine + SSE-driven progress pill, but for the dev-cluster dump sizes Trimus actually carries the synchronous path is fine.

Restore from a local backup¶

Restoring a Trimus instance to a working state from a complete backup. Assume you have nothing — empty Postgres, empty filesystem, no env vars.

1 · Provision the new host¶

Install Trimus at the same version that was running when the backup was taken. Version mismatch is the single most common restore failure — newer Trimus expects schema migrations that the backup hasn't applied yet, and older Trimus won't understand columns the backup has.

# Pin to the version recorded in the backup MANIFEST.
./install.sh v3.0.0

Provision a Postgres at a compatible major version (Trimus develops against Postgres 16; 14+ is supported).

2 · Restore the encryption key FIRST¶

Before touching Postgres. Write it into the new server's env (systemd unit, docker-compose env, whatever the deployment uses):

# Read the value out of the backup.
cat /backups/trimus/<stamp>/trimus_encryption_key.key

# Write it into the new server's env. Examples:
#   - systemd: edit /etc/trimus/trimus.env, then `systemctl daemon-reload`
#   - docker:  add to compose env or pass --env-file
#   - bare:    export TRIMUS_ENCRYPTION_KEY=... in the shell that runs trimus-server

Verify the new server's env will read it:

sudo -u trimus env | grep TRIMUS_ENCRYPTION_KEY

⚠️ If you mistype or skip this step, the restore "succeeds" but every encrypted column in Postgres is unreadable. The error surfaces the first time an agent tries to use a secret or a provider — and at that point you'll be debugging while the instance is "up."

3 · Restore Postgres¶

# Create an empty database first.
psql "postgres://postgres@localhost/postgres" \
    -c "CREATE DATABASE trimus OWNER trimus"

# Restore from the local dump (encrypted backups need trimus-cli
# restore, see the next subsection). For plain pg_dump output
# produced manually:
pg_restore -d "postgres://trimus@localhost/trimus" \
           --no-owner --no-privileges \
           --clean --if-exists \
           /backups/trimus/<stamp>/postgres.dump

For a Trimus-encrypted backup (.dump.enc):

DATABASE_URL=postgres://trimus@localhost/trimus \
TRIMUS_ENCRYPTION_KEY=...                       \
    trimus-cli restore \
        /backups/trimus/<stamp>/trimus-backup-<stamp>.dump.enc \
        --yes

Notes on the pg_restore flags:

--no-owner --no-privileges — drops the original ownership / GRANT clauses. The new instance may have a different DB user; let the connection role own everything.
--clean --if-exists — drop existing objects before recreating them, idempotent. Safe on an empty DB; required if you're restoring over an aborted attempt. Take the platform offline before restoring against a live instance — pg_restore --clean takes the destination through a "no tables exist" window.

Sanity-check:

psql "$DATABASE_URL" -c '\dt' | head
psql "$DATABASE_URL" -c "SELECT COUNT(*) FROM users"
psql "$DATABASE_URL" -c "SELECT COUNT(*) FROM companies"

The row counts should match the source instance.

4 · Restore the filesystem trees¶

# TRIMUS_HOME — adjust the target path to match the new host's $TRIMUS_HOME.
mkdir -p "$(dirname "$TRIMUS_HOME")"
tar -xpzf /backups/trimus/<stamp>/trimus_home.tar.gz \
    -C "$(dirname "$TRIMUS_HOME")"
chown -R trimus:trimus "$TRIMUS_HOME"

# TRIMUS_STORAGE_DIR — same shape.
mkdir -p "$(dirname "$TRIMUS_STORAGE_DIR")"
tar -xpzf /backups/trimus/<stamp>/trimus_storage.tar.gz \
    -C "$(dirname "$TRIMUS_STORAGE_DIR")"
chown -R trimus:trimus "$TRIMUS_STORAGE_DIR"

5 · Start trimus-server¶

systemctl start trimus-server
journalctl -u trimus-server -f

You're looking for:

database connection established
No TRIMUS_ENCRYPTION_KEY not set warning
No failed to decrypt errors

If you see decrypt errors, stop. Your encryption key doesn't match the backup. Recheck step 2. Do NOT proceed — restarting the server in this state risks the engine attempting write-paths against half-readable encrypted data.

6 · Verify the restore¶

Run through this checklist. Anything that fails means stop and investigate before declaring restore complete.

Check	How
Server is up	`curl http://localhost:3100/healthz` returns 200
Web UI loads	`curl -I http://localhost:3100/login` returns 200
Log in as a known user	UI login with credentials known from before the backup
Issues list renders	Open Issues page; the count matches expectation
One known company secret reads	Open Secrets page → click Reveal on a known secret → value matches
One known provider works	Open Providers page → click "Test connection" on a known provider
File attachment downloads	Open an issue with a known attachment, download it, verify bytes match
FS skills work	Trigger an agent that uses a company skill; observe the run completes

The "secret reveal" check is the single most important: if that passes, your encryption key is correct AND Postgres + key are mutually consistent. SEC-017 records every reveal in secret_audit_events, so the post-restore probe also leaves a clear audit trail.

If any check fails:

Login fails — could be session-related (sessions don't survive across restores in any meaningful way; users will need to log in fresh). If a known PASSWORD doesn't work, Postgres restore is incomplete or wrong.
Issues render but missing rows — partial Postgres restore. Re-run pg_restore.
Secrets fail to reveal — encryption key is wrong. Stop.
Provider test fails — provider api_key decrypt failed (also encryption key) OR the provider's upstream credential expired. Try a different known-good provider before concluding.
Files / attachments missing — TRIMUS_STORAGE_DIR was not restored (or chown was forgotten and the server can't read it).

Restore from S3¶

The symmetric counterpart to the S3 upload path: pull the encrypted dump straight out of S3 (or any S3-compatible bucket) and pipe it through the Vault into pg_restore — never touching local disk.

This closes the disaster-recovery loop. The operator's runbook is no longer "step 1: pull the object down with aws s3 cp; step 2: trimus-cli restore <local-path>". It is now a single command.

The same credential chain and endpoint configuration applies as the upload path (AWS SDK default chain for the credentials themselves; TRIMUS_BACKUP_S3_* env vars or --s3-* flag overrides for the addressing). The bucket itself comes from the s3://bucket/key positional argument — there is no --s3-prefix flag for the restore path because the operator supplies the full key.

The s3:// prefix is strict: an unsupported scheme (gs://, https://, file://), a missing key (s3://bucket/), or a malformed URI like s3:/bucket/key (single slash) all fall through to the local-file path, which then surfaces as a "backup file: no such file or directory" error. There is no fuzzy parsing — if the URI is mistyped, the operator notices immediately.

The S3 object is required to be encrypted (i.e. start with the TRIMUS_BACKUP_v1 magic). Plaintext custom-format dumps from --no-encrypt are not supported on the s3:// path because there is no operational reason to push plaintext dumps to S3 — they leak every operator secret in cleartext. If you need to restore a plaintext dump, pull it with aws s3 cp first and feed the local file to trimus-cli restore.

AWS S3¶

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export TRIMUS_BACKUP_S3_REGION=us-west-2
export TRIMUS_ENCRYPTION_KEY=...    # must match the backup-time key
export DATABASE_URL=postgres://...

trimus-cli restore \
    s3://ops-trimus-backups/prod/trimus/trimus-backup-20260529T140112.dump.enc \
    --yes

minio¶

export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export TRIMUS_BACKUP_S3_ENDPOINT=https://minio.internal:9000
export TRIMUS_BACKUP_S3_PATH_STYLE=true
export TRIMUS_BACKUP_S3_REGION=us-east-1
export TRIMUS_ENCRYPTION_KEY=...
export DATABASE_URL=postgres://...

trimus-cli restore \
    s3://trimus-backups/trimus-backup-20260529T140112.dump.enc \
    --yes

Cloudflare R2¶

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export TRIMUS_BACKUP_S3_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com
export TRIMUS_BACKUP_S3_REGION=auto
export TRIMUS_ENCRYPTION_KEY=...
export DATABASE_URL=postgres://...

trimus-cli restore \
    s3://trimus-backups/trimus-backup-20260529T140112.dump.enc \
    --yes

Notes¶

The same --clean --if-exists semantics apply. pg_restore --clean takes the destination through a "no tables exist" window. Take the platform offline before restoring against a live instance.
The download uses a single streaming GetObject, not a multi-part range download. Multi-GiB dumps are limited by single-connection throughput; if that's a real bottleneck, fall back to aws s3 cp --recursive --multipart-chunk-size-mb=... and then the local-file restore.

Verify a backup file¶

A backup you have never restored is a backup you do not have — but restoring is destructive, slow, and requires standing up a throwaway Postgres. trimus-cli backup-verify is the middle ground: stream the encrypted file end-to-end through the same NewBackupReader the restore path uses, discard the plaintext, and report whether the file is structurally readable.

DATABASE_URL=postgres://... TRIMUS_ENCRYPTION_KEY=... \
    trimus-cli backup-verify ./trimus-backup-20260529T140112.dump.enc
# → ✓ verified: ./trimus-backup-20260529T140112.dump.enc  dek=01HM...  chunks=0  decrypted=132.40 MiB in 4.21s

What backup-verify checks (and what it does NOT check):

Check	Verified?
Header magic is `TRIMUS_BACKUP_v1`	Yes
Header has dek_id, wrapped_fk, nonce_prefix	Yes
The named DEK exists in the Vault (KEK matches)	Yes
Every chunk's GCM auth tag is intact	Yes
The trailing zero-length EOF marker is present (no truncation)	Yes
The plaintext is a well-formed Postgres custom-format dump	No
The dump's row counts match the source instance	No

The first six get an operator "no surprises at restore time" signal that's almost as strong as a real restore. The last two require actually running pg_restore — for those, the Backup verification (test restore) procedure below is canonical.

Exit codes:

Exit	Meaning
`0`	Header parsed + every chunk decrypted + EOF marker seen.
`1`	Any failure: bad magic, malformed header, truncated stream, GCM auth failure, unknown dek_id, network failure on s3:// inputs. The first line on stderr is `✗ failed: <reason>`.

Cron-friendly pattern¶

--quiet suppresses the success summary so successful runs produce no output. Failures still write the ✗ failed: ... line to stderr, which is what monitoring systems grep for:

#!/usr/bin/env bash
set -euo pipefail
source /etc/trimus/trimus.env

LATEST=$(ls -t /var/backups/trimus/*.dump.enc | head -1)
if ! /usr/local/bin/trimus-cli backup-verify "$LATEST" --quiet 2> /tmp/trimus-verify.err; then
    cat /tmp/trimus-verify.err | mail -s "TRIMUS BACKUP VERIFY FAILED" ops@example.com
    exit 1
fi

Drop into /etc/cron.daily or a systemd timer. Verify takes about 4 s of wall-clock per 100 MiB on commodity hardware, so even multi-GB dumps fit comfortably in a 5-minute timer window.

The built-in backup_verify worker covers the same need for operators who don't want to manage their own cron — see Architecture.

Verifying an S3-uploaded backup¶

backup-verify takes the same s3://bucket/key URI form as restore:

DATABASE_URL=postgres://... TRIMUS_ENCRYPTION_KEY=... \
    trimus-cli backup-verify \
        s3://ops-trimus-backups/prod/trimus/trimus-backup-20260529T140112.dump.enc

The S3 path streams via the same io.Pipe plumbing the restore command uses — never lands on disk, no temp files left behind. There is one cosmetic difference: the success summary line drops the dek=... label for s3:// inputs because peeking at the header would mean a second GetObject. The dek id is recoverable via trimus-cli vault-status against the active DEK if needed.

Validate a backup file end-to-end with `pg_restore --list`¶

backup-verify confirms the file is a well-formed encrypted blob; trimus-cli restore-validate confirms the file's plaintext is a restorable Postgres archive. The deeper check decrypts (if needed) into a temp file in $TMPDIR and runs pg_restore --list <tempfile> — a read-only walk of the archive's table-of-contents that never executes a CREATE / INSERT / DROP.

DATABASE_URL=postgres://... TRIMUS_ENCRYPTION_KEY=... \
    trimus-cli restore-validate ./trimus-backup-20260529T140112.dump.enc
# → ✓ Archive valid: 47 tables, 92 indexes, 35 sequences, 4 views — Restore would succeed.
#     source=./trimus-backup-20260529T140112.dump.enc  toc-entries=193  decrypted=132.40 MiB  in 5.83s

What restore-validate adds over backup-verify:

Check	`backup-verify`	`restore-validate`
Header magic / DEK ID / wrapped_fk	Yes	Yes
Every chunk's GCM auth tag	Yes	Yes
Trailing EOF marker	Yes	Yes
Plaintext is a parseable `pg_restore` custom archive	No	Yes
Archive has at least one TOC entry	No	Yes
Per-kind object counts (tables / indexes / sequences / views)	No	Yes

A file that passes backup-verify but fails restore-validate typically means the underlying pg_dump output was truncated or corrupted before the encryption layer ran — e.g. disk filled mid-dump or pg_dump was killed. The two commands are complementary; run both on a daily cadence.

The combined trio reads:

Command	Cost	Catches
`backup-verify`	4 s / 100 MiB	encryption layer broken
`restore-validate`	~5 s / 100 MiB (decrypt + `pg_restore --list`)	encryption OK, archive malformed
`restore --yes`	minutes / GBs (full replay)	encryption + archive OK, FK violation at INSERT

Exit codes mirror backup-verify:

Exit	Meaning
`0`	Decryption (if needed) succeeded, `pg_restore --list` returned a non-empty TOC.
`1`	Any failure: missing file, decryption error, `pg_restore` non-zero exit, empty TOC. The first line on stderr is `✗ failed: <reason>`.

Cron-friendly pattern¶

--quiet suppresses the success summary, leaving only the canonical trimus-restore-validate ok <path> tables=N indexes=N sequences=N views=N in <duration> one-line summary plus exit code:

#!/usr/bin/env bash
set -euo pipefail
source /etc/trimus/trimus.env

LATEST=$(ls -t /var/backups/trimus/*.dump.enc | head -1)
if ! /usr/local/bin/trimus-cli restore-validate "$LATEST" --quiet 2> /tmp/trimus-validate.err; then
    cat /tmp/trimus-validate.err | mail -s "TRIMUS RESTORE VALIDATE FAILED" ops@example.com
    exit 1
fi

The decrypted temp file lives under $TMPDIR (typically /tmp) and is deleted before the command returns. A 64 GiB cap guards against a runaway decrypt filling /tmp — at 64 GiB the command exits 1 with an explicit "refusing to fill /tmp" error rather than truncating silently.

S3 inputs use the same s3://bucket/key form as backup-verify:

DATABASE_URL=postgres://... TRIMUS_ENCRYPTION_KEY=... \
    trimus-cli restore-validate \
        s3://ops-trimus-backups/prod/trimus/trimus-backup-20260529T140112.dump.enc

List backups¶

trimus-cli backup-list is the one-shot "what's in the pile?" answer. Walks the local backup directory (top-level only — same sweep semantic as the retention worker) and optionally lists the S3 mirror; emits a sorted-newest-first fixed-width table with size, age, encryption status, and expires-in:

$ trimus-cli backup-list
LOCATION  PATH/KEY                                             SIZE        AGE   ENCRYPTION  EXPIRES_IN
local     /var/backups/trimus/trimus-20260529T140112.dump.enc  132.40 MiB    2h  enc:v2      30d
local     /var/backups/trimus/trimus-20260528T140057.dump.enc  132.10 MiB   26h  enc:v2      29d
local     /var/backups/trimus/trimus-20260101T140042.dump.enc  122.00 MiB  148d  enc:v2      (expired)

--s3 enables the remote listing (uses the same TRIMUS_BACKUP_S3_* env vars as the upload/download/verify commands):

$ trimus-cli backup-list --s3
LOCATION  PATH/KEY                                                       SIZE        AGE   ENCRYPTION  EXPIRES_IN
local     /var/backups/trimus/trimus-20260529T140112.dump.enc            132.40 MiB    2h  enc:v2      30d
local     /var/backups/trimus/trimus-20260528T140057.dump.enc            132.10 MiB   26h  enc:v2      29d
s3        s3://ops-trimus-backups/prod/trimus-20260528T140057.dump.enc   126.30 MiB   26h  enc:v2
local     /var/backups/trimus/trimus-20260101T140042.dump.enc            122.00 MiB  148d  enc:v2      (expired)

The EXPIRES_IN column derives from TRIMUS_BACKUP_RETENTION_DAYS (default 30) — the same threshold the retention worker uses. So the answer to "what's the next thing to be deleted?" is "whatever's at the bottom of the table". S3 rows are blank — bucket lifecycle rules are the operator's responsibility, not the worker's.

JSON output¶

--json swaps the table for a JSON array, stable shape:

$ trimus-cli backup-list --dir=/var/backups/trimus --json | jq '.[0]'
{
  "location": "local",
  "path": "/var/backups/trimus/trimus-20260529T140112.dump.enc",
  "size": 138807424,
  "mtime": "2026-05-29T14:01:12Z",
  "encryption": "enc:v2",
  "expires_in_seconds": 2589120
}

expires_in_seconds is signed: negative values mean "already eligible for pruning". S3 rows omit the field entirely.

Common patterns¶

# Force a specific retention window (override the worker default)
# to see what would prune if we tightened to 7 days:
trimus-cli backup-list --retention-days=7

# Hide the EXPIRES_IN column entirely (worker disabled / external cron):
trimus-cli backup-list --retention-days=0

# Pipeline: count expired backups for monitoring.
trimus-cli backup-list --json | jq '[.[] | select(.expires_in_seconds != null and .expires_in_seconds <= 0)] | length'

Date range filter¶

Scope a listing to a specific incident window with --since and --until. Both flags accept YYYY-MM-DD (UTC midnight; --until is inclusive at end-of-day) or RFC3339 (2026-05-28T14:00:00Z):

# Everything during the May 28 incident.
trimus-cli backup-list --since=2026-05-28 --until=2026-05-28

# Recent half-day window.
trimus-cli backup-list --since=2026-05-28T14:00:00Z

# Compose with --s3 / --json.
trimus-cli backup-list --s3 --since=2026-05-28 --until=2026-05-29 --json

Filter applies to local mtime and S3 LastModified. Malformed dates or inverted bounds (--until before --since) reject with a clear error rather than silently matching nothing.

backup-list opens NO database connection — it's a pure filesystem + S3-API walker so it runs from a recovery shell with no DATABASE_URL set. Useful during disaster recovery when the operator wants to see what's available before deciding which restore invocation to issue.

Encryption status is sniffed from the first 17 bytes of each file (matching TRIMUS_BACKUP_v1). For local files this is a single os.Open + io.ReadFull. For S3 entries it's a Range-GET (Range: bytes=0-16) — about 17 bytes per object on the wire, acceptable even for 100-object listings.

Detect drift / orphans¶

trimus-cli backup-list --orphans reconciles the on-disk (and optionally S3) inventory against the backup_runs audit table. The reconciliation produces three statuses:

tracked — a file exists AND a backup_runs row points at it. The expected healthy state.
orphan-on-disk — a file exists with NO matching audit row. Either predates the backup audit table, or an operator-managed file the CLI never wrote (a hand-rolled pg_dump, a debug artifact, a file copied in from another instance).
orphan-in-db — a backup_runs row claims a file or S3 object that's no longer there. Either a hand-deletion, a disk swap, or an out-of-band S3 lifecycle rule the retention worker doesn't know about.

$ trimus-cli backup-list --orphans
LOCATION  PATH/KEY                                              SIZE        AGE   ENCRYPTION  EXPIRES_IN     STATUS
local     /var/backups/trimus/trimus-20260529T140112.dump.enc   132.40 MiB    2h  enc:v2      30d            tracked
local     /var/backups/trimus/trimus-20260528T140057.dump.enc   132.10 MiB   26h  enc:v2      29d            tracked
local     /var/backups/trimus/trimus-20260101T140042.dump.enc   122.00 MiB  148d  enc:v2      (expired)      orphan-on-disk
local     /var/backups/trimus/trimus-20240601T140000.dump.enc        9 KiB    1y  enc:v2      (in-db only)   orphan-in-db
4 orphan(s) detected

When --orphans is set AND any orphan is detected, the CLI exits 1 — cron-friendly for an alert pipeline:

0 8 * * * trimus-cli backup-list --orphans >/dev/null || \
  send-alert 'trimus backup orphans detected'

When no orphans are detected, exit is 0 and the summary line reads no orphans found. Combining --orphans with --s3 also reconciles the S3 surface; without --s3 the backup_runs.s3_* columns are skipped (we can't distinguish "S3 object missing" from "we just didn't look at S3"). The reconciliation considers the most recent 500 non-pruned backup_runs rows by created_at DESC — beyond that the historical data is usually long gone from disk anyway.

--orphans requires $DATABASE_URL (the audit table lives in Postgres); without it the command fails fast with a clear error so a recovery shell that intentionally lacks DB access doesn't silently report "no orphans".

The backup_reconcile worker auto-heals both directions of orphan drift on a 6-hour cadence so operators don't have to keep running the CLI. The CLI is the right tool for ad-hoc investigation; the worker is the right tool for routine maintenance.

Retroactively register legacy backups¶

trimus-cli backup-import is the one-shot operator companion to the backup_reconcile worker. It walks the local backup directory once, picks files matching the retention worker's extension list (.dump.enc, .dump, .sql.gz, .sql), and inserts one backup_runs row per file with kind='reconciled'. After running, those files surface on /admin/audit and are tracked by retention as normal.

Use cases:

Legacy dumps written before the backup audit table existed, so the rows were never created.
Files dropped into the backup dir by an out-of-band tool — a hand-rolled pg_dump, a recovery snapshot copied from another instance, an emergency dump produced before trimus-cli was installed on the host.
Files surfaced as orphan-on-disk by backup-list --orphans that the operator wants to claim without waiting for the next reconcile worker tick.

# Dry-run (default).
$ trimus-cli backup-import
Scanning /var/backups/trimus
  will-insert      /var/backups/trimus/old-2025-12-01.dump (1234567 bytes)
  will-insert      /var/backups/trimus/old-2025-12-02.dump.enc (1245678 bytes)
  skip-tracked     /var/backups/trimus/trimus-20260529T140112.dump.enc (138758144 bytes)
Found 3 files. Would import 2 (1 already tracked, 0 corrupted/unreadable)
Pass --yes to actually write the rows.

# Live run.
$ trimus-cli backup-import --yes
Found 3 files. Imported 2 (1 already tracked, 0 corrupted/unreadable)

Idempotent — re-running after a successful import is a no-op (every file lands in the skip-tracked bucket). Files that can't be opened for the encryption header sniff (chmod 000, racing delete, broken disk) are reported as skip-unreadable rather than aborting the whole import.

# Cron-friendly: one-line summary.
$ TRIMUS_QUIET=1 trimus-cli backup-import --yes
trimus-backup-import ok found=3 imported=2 tracked=1 unreadable=0

Requires $DATABASE_URL (the audit row lives in Postgres). The encryption class is sniffed from the first 17 bytes of each file — no decrypt happens, so the import is fast even with hundreds of encrypted files.

Aggregate stats — "how have my backups been doing?"¶

trimus-cli backup-stats aggregates the backup_runs audit table over a trailing window and emits a one-glance summary. Useful when an operator wants the daily / weekly answer to "are we still backing up reliably?" without opening the audit dashboard or writing SQL:

$ trimus-cli backup-stats
Backup statistics — last 30 days

  Total runs:      48
  Success:         44 (91.7%)
  Failure:         4 (8.3%)

  By kind:
    create:        12
    cron:          18
    cron_verify:   16
    manual:        2

  Latest success:  2026-05-30T02:00:00Z (12.0 MiB)
  Largest:         18.0 MiB (cron, 2026-05-25T02:00:00Z)
  Avg duration:    4.2s

--days=N overrides the default 30-day window (capped at 365). --json swaps the summary for a single JSON object the same shape as the documented keys (window_days, total_runs, success_count, failure_count, failure_rate_pct, by_kind, latest_success, largest_backup, avg_duration_ms).

--since=DATE and --until=DATE override --days with an absolute window. Useful for ad-hoc "what happened during the May 28 incident?" queries:

# Just the May 28 incident window.
trimus-cli backup-stats --since=2026-05-28 --until=2026-05-28

# Half-bounded — from May 28 to now.
trimus-cli backup-stats --since=2026-05-28

Dates accept YYYY-MM-DD (UTC midnight; --until is inclusive at end-of-day) or RFC3339. Malformed dates or inverted bounds reject before the DB pool is opened.

--quiet collapses to the canonical one-line cron summary:

$ trimus-cli backup-stats --quiet --days=7
trimus-backup-stats ok window=7d total=12 success=11 failure=1 failure_rate=8.3%

Pair --quiet with TRIMUS_QUIET=1 and a daily cron for a fleet-wide "weekly digest" line that's easy to grep out of syslog:

0 8 * * 1 trimus-cli backup-stats --days=7 --quiet >> /var/log/trimus-backups.log

The CLI requires $DATABASE_URL — same posture as backup-list --orphans — because the aggregation reads backup_runs. A recovery shell without DB access can't compute the stats; for that case the audit dashboard at /admin/audit?source=backup_runs is the alternative.

Composite health check — "is my backup story working?"¶

trimus-cli backup-health collapses four backup signals into a single OK / WARN / CRIT verdict. Designed for cron + alerting pipelines that want one command, one exit code, one canonical output line.

Signals aggregated (worst-of):

Signal	Source	Bands
Backup freshness	most recent successful `backup_runs` row with `kind IN ('create','cron')`	OK ≤ threshold, WARN > threshold, CRIT > 2× threshold; no row ever = CRIT
Verify freshness	most recent successful `cron_verify` OR `verify_outcome='verified'` row	OK ≤ 7d, WARN > 7d; never = WARN
Retention worker	most recent `pruned_at` on `backup_runs`	OK ≤ 2h (2× the worker interval), WARN > 2h; never = OK (treated as "idle / disabled")
Disk pressure	local `/` partition via the shared `health.CollectDiskSummary` probe	OK < 85%, WARN ≥ 85%, CRIT ≥ 95%; statfs failed = WARN

Default output:

$ trimus-cli backup-health
📋 Backup health check

Status:        WARN

  Backup fresh:    OK    (last 8h ago, threshold 36h)
  Verify fresh:    OK    (last 12h ago)
  Retention:       OK    (last prune 18m ago)
  Disk pressure:   WARN  (87% used, warn at 85%)

--max-age-hours=N (default 36, range 1..720) overrides the backup-freshness threshold. The verify / retention / disk thresholds are fixed in the binary — change those by editing backupHealthDefaultMaxAgeHours / verifyStalenessWarnHours / retentionStalenessWarnMinutes and rebuilding.

--json swaps the table for the structured envelope:

{
  "status": "WARN",
  "generated_at": "2026-05-30T12:45:18Z",
  "checks": [
    { "name": "backup",    "status": "OK",   "detail": "last 8h ago, threshold 36h", "value": 8.0 },
    { "name": "verify",    "status": "OK",   "detail": "last 12h ago", "value": 12.0 },
    { "name": "retention", "status": "OK",   "detail": "last prune 18m ago", "value": 18.0 },
    { "name": "disk",      "status": "WARN", "detail": "87% used, warn at 85%", "value": 87.0 }
  ]
}

--quiet collapses everything to the canonical one-line summary the spec defines:

$ trimus-cli backup-health --quiet
trimus-backup-health WARN backup=OK verify=OK retention=OK disk=WARN

Cron callers grep the WARN / CRIT / OK token and branch on the exit code:

Exit	Meaning
0	OK — all signals nominal
1	WARN — at least one signal warning, none critical
2	CRIT — at least one signal critical

A representative alerting setup:

*/15 * * * * /usr/local/bin/trimus-cli backup-health --quiet \
    | logger -t trimus-backup-health

Pair with a journalctl filter that pages on CRIT and emails on WARN. Exit code 2 also makes it trivial to wire into Healthchecks.io, BetterUptime, PagerDuty, etc.

The CLI requires $DATABASE_URL because three of the four signals come from backup_runs. A recovery shell without DB access cannot run this command; the disk-pressure signal alone is available via the /admin/operations panel in that case.

Partial restore (one table or one company)¶

When you only need to recover one piece — accidentally deleted a company, dropped a single table, etc.

One table¶

pg_restore -d "$DATABASE_URL" -t issues /backups/trimus/<stamp>/postgres.dump

Risk: foreign-key references from other tables may break. pg_restore -t doesn't follow FK dependencies. Prefer the single-company procedure below.

One company (the operator's most common ask)¶

There's no built-in single-company restore — Trimus's tenant isolation is at the application layer, not via schema-per-tenant. The procedure:

Restore the entire dump into a separate database (e.g. trimus_recovery).
Identify the company you want to bring back by id in trimus_recovery.companies.
Use the portability export feature: from the recovery server's web UI, Company → Portability → Export. This produces a bundle of issues, agents, projects, skills, etc.
Import the bundle into the live instance via the same UI.

This is more work than a direct SQL restore but it avoids the cross-tenant data-corruption risk of partial FK restoration.

If you genuinely cannot use the UI path (e.g. live instance is also down), pg_restore -d trimus_recovery /backups/.../postgres.dump and then write company-scoped INSERT-INTO-SELECTs from trimus_recovery to the live DB, table by table, in referential-integrity order. This is what enterprise support engagements look like — talk to a Trimus operator who has done it before rather than blind-flying it.

Backup verification (test restore)¶

A backup you have never restored is a backup you do not have. Verify quarterly at minimum.

The cleanest way to verify: restore into a throwaway instance and poke around. The make sandbox-validate-fresh target spins up exactly such an environment and is the same plumbing that the plugin sandbox runbook uses. Adapted procedure:

# 1. Spin up a fresh throwaway instance.
make sandbox-validate-fresh

# 2. Stop the auto-started server (it's running an empty DB).
make sandbox-validate-fresh-down-server-only   # see Makefile

# 3. Restore the backup into the throwaway postgres.
pg_restore -d "postgres://trimus:trimus@localhost:5434/trimus_sandbox?sslmode=disable" \
           --no-owner --no-privileges --clean --if-exists \
           /backups/trimus/<stamp>/postgres.dump

# 4. Untar trimus_home and trimus_storage into .sandbox-test/trimus-home
#    and .sandbox-test/storage respectively.

# 5. Put the right encryption key in the env, then restart the
#    throwaway server pointing at the throwaway DB.

# 6. Walk through the verification checklist from
#    `Restore from a local backup` → step 6.

# 7. Tear it all down.
make sandbox-validate-fresh-down

If the verification passes, mark the backup as known-good. Keep a log of last-verified dates so you don't get stuck assuming "we'd have noticed if backups broke."

6. Monitoring¶

Operations panel¶

/admin/operations — instance-admin gated. Carries the at-a-glance widgets every backup-related signal flows into:

Deploy card — version / commit / build date / hostname / uptime. Confirms which build is running before you touch anything else.
Environment summary card — presence checks for the operationally-important env vars (encryption, Slack, S3 bucket, OTLP, etc.) plus the safe-to-display tuning knobs (backup dir, cron, retention days, subagent caps). Never prints secret values.
Disk usage card — root filesystem usage (used / total / percentage with an inline bar), systemd journal size (/var/log/journal du-walk), and configured backup-dir size. Status bands track root usage: green/"healthy" below 85%, yellow/"warning" at 85–94%, red/"critical" at 95% and above. Warning + critical states surface inline cleanup hints (sudo journalctl --vacuum-size=300M, lowering TRIMUS_BACKUP_RETENTION_DAYS, go clean -cache). Probe errors on the journal or backup dir degrade silently to 0 bytes — permission-denied on a hardened host doesn't 500 the page. Root Statfs failure flips the card to unknown rather than reporting a misleading green band.
Backups widget — most recent successful backup summary (relative time, encryption flag, size, duration); only shown when more recent than the last success, the most recent failure with truncated error message. Left border color-tagged against last-successful age: green within 24h, yellow within 7 days, red older than 7 days or never run. Carries the 15 "Run backup now" button and a "View all backup runs" deep-link into the audit dashboard.
Retention preview tile — answers "what does the next sweep prune?" without waiting for the worker to fire. Badges: <N> eligible (with Oldest / Newest dates), disabled when TRIMUS_BACKUP_RETENTION_DAYS=0, dir missing (red border) when the configured path doesn't exist. The preview is a pure read of the filesystem — no DB writes, no worker side effects.
Backup workers tile — health summary for all four background workers (backup_cron, backup_verify, backup_retention, backup_reconcile). Per-worker last-tick relative time, ticks / errors counters, and a health badge. The tile's left-border color and overall badge aggregate across all four rows: green when every row is "ok"; yellow when any row is overdue (>1× its interval) or still in boot grace; red when any row is stale (>2× its interval) or when a row is missing entirely from the registry. The compact <n> of <m> registered affordance top-right flags the missing-worker case at a glance.

Audit dashboard¶

/admin/audit?source=backup_runs — filters the consolidated x audit feed to just backup activity. Each row carries a friendly summary:

backup success (encryption=enc:v2, size=132.4 MiB, duration=4.2s) for normal CLI / cron / manual create rows.
verify verification_failed: ... for verify-cron failures.
reconciled (size=<bytes>) for kind='reconciled' rows from the auto-reconcile worker.
Lifecycle marker — PRUNED <when> appended to rows whose file has since been deleted by the retention sweep or marked missing by the reconcile worker.

The Company filter does NOT apply — backups are instance-wide.

Prometheus metrics¶

Two gauges land on /metrics:

# HELP staple_backup_last_success_timestamp_seconds Unix epoch seconds of the most recent successful backup_runs row. 0 = none recorded.
# TYPE staple_backup_last_success_timestamp_seconds gauge
staple_backup_last_success_timestamp_seconds 1717003272

# HELP staple_backup_last_size_bytes Size in bytes of the most recent successful backup (S3 size when uploaded, else local). 0 = none recorded.
# TYPE staple_backup_last_size_bytes gauge
staple_backup_last_size_bytes 138807424

Both are populated from query.GetLastSuccessfulBackupRun and cached in-process for 5 minutes so a busy scraper doesn't hammer PostgreSQL on every poll. 0 on either gauge means "no successful backup has ever landed" — the cold-start signal operators can alert on.

Each background worker also registers in the worker tick table, so the standard staple_worker_ticks_total{name="..."} and staple_worker_errors_total{name="..."} counters carry per-worker health for backup_cron, backup_verify, backup_retention, and backup_reconcile.

Suggested alert recipes¶

# Most recent backup is stale.
- alert: TrimusBackupStale
  expr: time() - staple_backup_last_success_timestamp_seconds > 86400
  for: 5m
  labels: {severity: warning}
  annotations:
    summary: "Trimus backup is over 24 h old"
    description: |
      The last successful backup_runs row is more than 24 hours old.
      Check the backup cron + the trimus-cli backup exit status.

# Cold-start: no successful backup ever recorded.
- alert: TrimusBackupNeverRecorded
  expr: staple_backup_last_success_timestamp_seconds == 0
  for: 30m
  labels: {severity: warning}
  annotations:
    summary: "No trimus backup has ever been recorded"
    description: |
      A trimus instance is running but no backup_runs row exists.
      Either the cron has never run, or every invocation has failed.

# A verify-cron failure means a long-lived dump has rotted.
- alert: TrimusBackupVerifyFailing
  expr: increase(staple_worker_errors_total{name="backup_verify"}[24h]) > 0
  for: 5m
  labels: {severity: warning}
  annotations:
    summary: "Trimus backup-verify worker reported an error in the last 24h"
    description: |
      Look up the failing row at /admin/audit?source=backup_runs and
      reproduce locally with `trimus-cli backup-verify <path>`.

# A reconcile worker error usually means a permissions / DB issue.
- alert: TrimusBackupReconcileFailing
  expr: increase(staple_worker_errors_total{name="backup_reconcile"}[24h]) > 0
  for: 5m
  labels: {severity: warning}
  annotations:
    summary: "Trimus backup-reconcile worker is erroring"
    description: |
      The reconcile worker can't heal drift between disk and
      backup_runs. Most likely a permissions issue on
      TRIMUS_BACKUP_DIR or a DB outage.

The dashboard + the gauges combine to give an operator two different angles on the same signal: the dashboard is "what's been happening, with context"; the gauges are "are we still within the backup SLA?". Both read from the same backup_runs table — the only writers are trimus-cli backup, the /admin/operations manual trigger, and the four background workers.

7. Troubleshooting¶

"Restore worked but secrets don't decrypt"¶

Almost always: the wrong TRIMUS_ENCRYPTION_KEY. Less commonly: the key has been rotated since the backup and the rotation didn't re-encrypt everything. There's no automatic key-rotation pass — when you change the key, you must run trimus-cli encrypt-secrets and trimus-cli encrypt-providers against the new key OR keep the old key available for legacy decrypt. The SEC-019 key rotation milestone will formalise this; track that item for updates.

"I restored postgres but the UI is empty"¶

If issues / agents / companies count zero rows but pg_restore exited 0, the most likely cause is that the source dump was created against a database alias and the restore landed in a different one. Check:

psql "$DATABASE_URL" -c "\l" | grep -i trimus
psql "$DATABASE_URL" -c "SELECT current_database()"

Make sure $DATABASE_URL and the database pg_restore wrote to agree.

"File attachments 404 after restore"¶

Usually chown was forgotten — the server's service user can't read the restored TRIMUS_STORAGE_DIR. Check:

sudo -u trimus ls "$TRIMUS_STORAGE_DIR" | head
# If this returns "Permission denied", that's the bug.

The session-cookie HMAC isn't the same as the encryption key — but if you're running Trimus behind a reverse proxy on a different domain after the restore, the secure / samesite cookie attributes may be silently rejecting cookies. Check the browser network panel.

"I lost my encryption key"¶

There is no recovery. The encrypted columns (secrets, provider api_keys, custom_headers, webhook signing secrets) are permanently unreadable. The instance can still be brought up — just those features will fail. Procedure:

Generate a new key. Set TRIMUS_ENCRYPTION_KEY to the new value.
The server will start; non-secret data is unaffected.
Every existing row in company_secret_versions, company_providers, and routine_triggers is dead weight at that point. Either:
Delete the rows and have operators re-enter every secret / api_key / signing secret, OR

Truncate just those columns:

UPDATE company_secret_versions SET encrypted_value = '';
UPDATE company_providers SET api_key = '', custom_headers = '{}';
UPDATE routine_triggers SET signing_secret = NULL;

and have the operator re-enter values via the UI.

This is the single most important reason to take the Encryption key contract seriously.

"The cron worker isn't running"¶

Walk this list:

TRIMUS_ENCRYPTION_KEY set? The worker auto-disables without it (see the boot log for the disable reason).
pg_dump in $PATH? Install postgresql-client in the container or systemd unit's PATH.
TRIMUS_BACKUP_CRON=disable? The worker logs cron disabled at boot in that case — intentional.
The Operations panel "Backup workers" tile — does the backup_cron row show ticks? If not, the worker registered but isn't ticking; check staple_worker_ticks_total{name="backup_cron"} in /metrics against the configured TRIMUS_BACKUP_CRON_INTERVAL_HOURS.

"Retention sweep isn't pruning anything"¶

Walk this list:

TRIMUS_BACKUP_RETENTION_DAYS=0? The worker auto-disables.
TRIMUS_BACKUP_DIR exists? Missing-dir → Warn log + skip.
Files have the expected extension (.dump, .dump.enc, .sql, .sql.gz)? Files with other extensions are NEVER touched.
File mtimes actually older than retention_days * 24h? Use trimus-cli backup-list --retention-days=<N> to preview what would prune at a given cutoff.

"An orphan-in-db row I expected got auto-cleaned"¶

That's the reconcile worker doing its job. pruned_at is stamped with error_message appended (auto-reconciled: file missing at <ts>). Set TRIMUS_BACKUP_RECONCILE=disable if you want manual control via trimus-cli backup-list --orphans.

8. Reference¶

Environment variables¶

Variable	Default	Read by	Notes
`TRIMUS_ENCRYPTION_KEY`	(unset)	every component	Base64(32 bytes). Required for encrypted backups. Workers auto-disable when unset.
`DATABASE_URL`	(unset)	every component	Standard Postgres connection string.
`TRIMUS_BACKUP_DIR`	`/var/backups/trimus`	all 4 workers + CLI	Top-level only; no recursion. Must exist or workers skip.
`TRIMUS_BACKUP_NO_VERIFY`	(unset, false)	`trimus-cli backup`	Truthy → skip the inline auto-verify after a new backup.
`TRIMUS_BACKUP_S3_BUCKET`	(unset)	upload + restore + verify + list	Required to enable the S3 sink.
`TRIMUS_BACKUP_S3_PREFIX`	`trimus-backups/`	upload + list	Key prefix.
`TRIMUS_BACKUP_S3_REGION`	`us-east-1`	upload + restore + verify + list	Some providers expect `auto`.
`TRIMUS_BACKUP_S3_ENDPOINT`	(empty)	upload + restore + verify + list	Custom endpoint URL. Empty → AWS S3.
`TRIMUS_BACKUP_S3_PATH_STYLE`	`false`	upload + restore + verify + list	`true` for minio + most self-hosted gateways.
`TRIMUS_BACKUP_RETENTION_DAYS`	`30`	`backup_retention` worker + `backup-list`	Clamped to `[1, 3650]`. `0` disables the worker.
`TRIMUS_BACKUP_CRON`	(unset, enabled)	`backup_cron` worker	`disable` skips the worker entirely.
`TRIMUS_BACKUP_CRON_INTERVAL_HOURS`	`24`	`backup_cron` worker	Clamped to `[1, 720]`.
`TRIMUS_BACKUP_VERIFY_INTERVAL_HOURS`	`24`	`backup_verify` worker	Clamped to `[1, 720]`.
`TRIMUS_BACKUP_RECONCILE`	(unset, enabled)	`backup_reconcile` worker	`disable` skips the worker entirely.
`TRIMUS_BACKUP_RECONCILE_INTERVAL_HOURS`	`6`	`backup_reconcile` worker	Clamped to `[1, 168]`.
`TRIMUS_HOME`	`$HOME/.trimus/`	filesystem-side ops	Backed up by your own ops tooling (not the workers).
`TRIMUS_STORAGE_DIR`	`./storage`	filesystem-side ops	Backed up by your own ops tooling (not the workers).

CLI commands¶

Command	Purpose	Notes
`trimus-cli backup`	Take a backup. Optionally upload to S3, auto-verify on success.	Writes a `backup_runs` row (`kind='create'`). Exit 1 on failure → local file is deleted.
`trimus-cli backup --no-verify`	Skip the inline auto-verify for one invocation.	Use only when an external pipeline already validates.
`trimus-cli backup --no-s3`	Skip the S3 upload for one invocation.	Local file is still written + audited.
`trimus-cli backup --no-encrypt`	Plaintext dump (EMERGENCY-USE-ONLY).	Auto-verify is skipped. Never push to S3.
`trimus-cli restore <path>`	Restore from a local encrypted dump.	Use `--clean --if-exists` semantics by default.
`trimus-cli restore s3://bucket/key`	Restore from an S3 object.	Encrypted files only; strict `s3://` scheme parsing.
`trimus-cli backup-verify <path>`	Stream the encrypted dump through the vault decrypt pipeline.	Exit 0 = valid; 1 = any failure. `--quiet` for cron friendliness.
`trimus-cli backup-verify s3://bucket/key`	Same as above, against an S3 object.	Drops the `dek=...` label from the success line.
`trimus-cli restore-validate <path>`	Decrypt (if encrypted) into a temp file, then `pg_restore --list` the archive.	Stronger than `backup-verify`: passes iff the plaintext is a restorable archive. Exit 0 = valid TOC; 1 = any failure.
`trimus-cli backup-list`	List local dumps with size / age / encryption / expires-in.	No DB connection required.
`trimus-cli backup-list --s3`	Add the S3 mirror to the listing.	Range-GET sniffs encryption per object.
`trimus-cli backup-list --json`	Stable JSON shape for pipelines.	`expires_in_seconds` is signed (negative = already eligible).
`trimus-cli backup-list --orphans`	Reconcile disk inventory vs `backup_runs`.	Exit 1 when any orphan is found. Requires `$DATABASE_URL`.
`trimus-cli vault-status`	Inspect the active DEK / KEK / wrap counts.	Useful when debugging "which DEK encrypted this backup?".

`backup_runs` audit schema¶

One row per invocation. The CHECK constraint on kind pins the value space so a typo cannot silently pollute the table.

Column	Type	Meaning
`id`	uuid	Primary key.
`created_at`	timestamptz	When the row was inserted (i.e. backup or verify or reconcile start time).
`kind`	text	Source discriminator: `create`, `cron`, `manual`, `cron_verify`, `reconciled`.
`outcome`	text	`success` or `failure`.
`encryption`	text	`enc:v2` or `plaintext`.
`dek_id`	text	Active Vault DEK used to wrap chunk keys (NULL for plaintext).
`local_path`	text	Where the file landed locally; NULL after pruning.
`local_size_bytes`	bigint	Local file size; 0 / NULL when no local artifact.
`s3_bucket`	text	S3 bucket the upload landed in (NULL when no upload).
`s3_key`	text	Full S3 key (NULL when no upload).
`s3_size_bytes`	bigint	S3-side size (may differ from local after encryption).
`duration_ms`	bigint	Wall-clock duration from start to return.
`verify_outcome`	text	`verified` / `verification_failed` / `skipped`.
`error_message`	text	On failure, the first-line error message. On reconcile / prune, the lifecycle marker.
`actor_user_id`	uuid	The user who initiated the backup (NULL for cron / reconcile / verify-cron).
`hostname`	text	Which box ran the operation.
`staple_version`	text	Which Trimus build version produced the row.
`pruned_at`	timestamptz	When the matching file was deleted (by retention sweep or reconcile worker). NULL while the file still exists.

The kind value space is enforced by the CHECK constraint added across migrations 123–126. Adding a new value means adding a migration that DROPS the existing constraint and ADDs a wider one; see internal/db/migrations/126_backup_runs_manual_kind.sql for the most recent example.

Disaster recovery cadence¶

A reasonable starting point. Adjust per your tolerance for data loss.

Tier	Cadence	Storage
Production	Hourly Postgres WAL archive; daily full Postgres dump; daily filesystem tar	30 days local + offsite (S3 with versioning)
Production (small)	Daily full backups	14 days local + offsite weekly
Staging	Daily full backups	7 days local
Dev	Backup before destructive operations only	Local

Test the restore quarterly at every tier. A monthly cadence costs ~30 minutes of operator time and catches every kind of "silent" backup failure.

database.md — Postgres setup, migrations, maintenance.
storage.md — Filesystem layout for TRIMUS_HOME and TRIMUS_STORAGE_DIR.
secrets-and-encryption.md — What the encryption key encrypts; rotation guidance.
upgrades.md — Version-to-version upgrade procedure (which references this doc for the "back up first" step).
audit-dashboard.md — The consolidated audit feed at /admin/audit (filterable to ?source=backup_runs).

Backup and restore¶

Table of contents¶

1. Overview¶

What the backup system does¶

What the backup system does NOT do¶

What you must back up¶

Encryption key contract¶

2. Architecture¶

CLI: trimus-cli backup¶

Background workers¶

Audit table: backup_runs¶

3. First-time setup¶

0 · Confirm where each component lives¶

1 · Install pg_dump¶

2 · Provision the encryption key¶

3 · Configure the backup directory¶

4 · (Optional) Configure S3 for off-host copies¶

Configuration¶

AWS S3¶

minio (or any self-hosted S3 gateway)¶

Cloudflare R2¶

What gets uploaded¶

5 · Back up filesystem trees (outside the built-in workers)¶

6 · (Optional) One-shot full backup script¶

4. Daily operations¶

What runs automatically¶

What to glance at periodically¶

5. Operator actions¶

Manual backup (CLI)¶

Skipping auto-verify¶

Cron usage (--quiet)¶

Manual backup (UI)¶

Restore from a local backup¶

1 · Provision the new host¶

2 · Restore the encryption key FIRST¶

3 · Restore Postgres¶

4 · Restore the filesystem trees¶

5 · Start trimus-server¶

6 · Verify the restore¶

Restore from S3¶

AWS S3¶

minio¶

Cloudflare R2¶

Notes¶

Verify a backup file¶

Cron-friendly pattern¶

Verifying an S3-uploaded backup¶

Validate a backup file end-to-end with pg_restore --list¶

Cron-friendly pattern¶

List backups¶

JSON output¶

Common patterns¶

Date range filter¶

Detect drift / orphans¶

Retroactively register legacy backups¶

Aggregate stats — "how have my backups been doing?"¶

Composite health check — "is my backup story working?"¶

Partial restore (one table or one company)¶

One table¶

One company (the operator's most common ask)¶

Backup verification (test restore)¶

6. Monitoring¶

Operations panel¶

Audit dashboard¶

Prometheus metrics¶

Suggested alert recipes¶

7. Troubleshooting¶

"Restore worked but secrets don't decrypt"¶

"I restored postgres but the UI is empty"¶

"File attachments 404 after restore"¶

"Login works but the user gets logged out every request"¶

"I lost my encryption key"¶

"The cron worker isn't running"¶

"Retention sweep isn't pruning anything"¶

"An orphan-in-db row I expected got auto-cleaned"¶

8. Reference¶

Environment variables¶

CLI commands¶

backup_runs audit schema¶

Disaster recovery cadence¶

CLI: `trimus-cli backup`¶

Audit table: `backup_runs`¶

1 · Install `pg_dump`¶

Validate a backup file end-to-end with `pg_restore --list`¶

`backup_runs` audit schema¶