trstctl doctor — prove tenant isolation on your own deployment

Every vendor asks you to trust their CI. trstctl doctor runs the isolation checks against your running deployment instead, and prints a receipt you can keep. It is a read-only command by default, it exits non-zero when a check fails, so you can run it in your own pipeline.

This is a served command in the shipping trstctl binary — not a library, not a hosted service. It talks to the same PostgreSQL database your control plane serves from and, when one is configured, the same S3-compatible object store that holds its audit archives, report artifacts and backups.

What it actually does

The command opens one connection to your database and runs a fixed list of probes. Each probe answers one yes/no question about an architecture invariant, reports pass, fail, warn, or skip, and — this is the part that matters — states what it does not prove.

The probes are not a second copy of the test suite. Where a CI test contains the canonical query, that query was lifted into a shared helper in internal/store (TenantTableRLSStates, USINGOnlyTenantPolicies), and both the test and the probe call it. They cannot drift, because there is only one of each query.

A probe that cannot run reports skip with the reason. A skipped proof never looks like a passed proof. That rule is why the summary line reports skips separately, and why the honest answer below is "11 pass, 15 skipped" rather than a green tick.

Running it

trstctl doctor --prove-isolation --write-probe --json receipt.json --sign

The database connection comes from --postgres-dsn, defaulting to $TRSTCTL_POSTGRES_DSN. Without a DSN the command exits 2. The object store comes from the same settings the control plane reads: the object_store section of the file named by $TRSTCTL_CONFIG_FILE and the TRSTCTL_OBJECT_STORE_* variables (see configuration). Without them the object probes report skip.

Exit codes: 0 every probe passed · 1 at least one probe FAILED (or WARNed under --fail-on warn) · 2 a configuration or connectivity error. Gate your own CI on the exit code; that is the point at which this stops being a demo and becomes a control.

The safety model — read this before running it in production

Read-only is the default. The write probes are the only ones that touch data, and they do so like this:

• Reserved synthetic tenants. Probe tenants use the id prefix 00000000-d0c7 — deliberately not a real UUIDv4 shape, so they are unmistakable in an audit log.
• Cleanup is verified, not assumed. ISO-CLEAN deletes the probe rows and then re-queries for residue. It reports pass only after observing zero rows. The test suite induces a mid-probe failure and asserts cleanup still ran.
• A leaked probe tenant is a FAIL. ISO-LEAK runs on every invocation, including read-only ones. Rows left under the reserved prefix by an earlier run fail the current run.
• Probe activity is ordinary database activity and may surface in your operational monitoring. It is not hidden. If you see two short-lived tenants under the reserved prefix, that was doctor.
• Object probes stay under the same reserved prefix. The object lane writes only under tenant/00000000-d0c7…/ (beneath your deployment prefix). OBJ-CLEAN deletes those objects and then lists the reserved prefix, and reports pass only on an empty listing; OBJ-LEAK fails any run that finds an earlier run's objects there. Without --write-probe the lane only lists that prefix.

What each probe proves, and what it does not

Tenant isolation (AN-1)

Object-store isolation (AN-1)

These run when an object store is configured. They use the same tenant-prefixed client the audit archive, report artifact and backup targets use, so the proof and the product cannot drift.

The groups that mostly skip, and why

Doctor holds one seat: the database. That seat cannot honestly answer questions about the signer process, the event log, or the serving process's memory. Rather than guess, those probes skip with the reason stated:

• Signer isolation (AN-4) — SIG-1 skips because the signer's health RPC reports serving status, not a PID. SIG-3 skips because no dependency-closure fingerprint is compiled into the signer binary yet; until one is, AN-4's closure proof stays in CI (cmd/trstctl-signer/core_boundary_test.go). SIG-4 skips because key-custody proof needs signer API credentials this seat does not hold. SIG-2 runs only when you pass --signer-socket.

• Event integrity (AN-2) — EVT-1 (audit hash chain), EVT-2 (projection checkpoint lag), and EVT-3 (read-model rows resolving to source events) all need the NATS event stream, which this seat has no credentials for. Verify the audit chain through the served audit export instead.

• Durability and backpressure (AN-5/6/7) — DUR-2 (outbox lane age) and FABRIC-1 (agent job queue age) run from the database. DUR-1, DUR-3, and DUR-4 live in the serving process; query GET /api/v1/operations/bulkheads on the running control plane for pool depth.

  FABRIC-1 sweeps work that is reserved for an agent and that no agent has claimed, and warns when the oldest has waited longer than fifteen minutes. It names the ROLE the waiting work demands, because that is usually the answer: a job stamped for a network relay in a fleet of host-only agents waits forever and looks exactly like a busy queue. Age rather than depth is the signal, for the same reason as DUR-2 — a deep queue that is draining is healthy, and a shallow one that is not is a stopped fabric.

  It is deliberately separate from DUR-2 rather than folded into it. A stalled outbox lane means the control plane is not delivering; a stalled agent queue means the control plane is correctly not touching the work and the fleet is not taking it. Those need different runbooks, and one number reporting both would send an operator to the wrong one. Its limit: it is an age sweep at one instant, over rows an agent is meant to claim. It cannot tell you whether a claiming agent is making progress — that is the credential-redemption age in GET /api/v1/operations/jobs and the TrstctlCredentialRedemptionStuck alert.

One probe outside those groups always runs: POSTURE-1 reports the server's row_security setting and version string — the deployment posture that makes the ISO probes meaningful in the first place. It proves the server has not globally disabled row security, not that any individual policy is correct.

Do not read a skip as a weakness being present or absent. It means doctor did not look, and says so.

The receipt

Below is a real receipt from a probe run against a migrated test deployment, abridged to four probes. Values are as emitted; nothing here is illustrative.

{
  "schema": "trstctl.doctor.v1",
  "generated_at": "2026-10-25T10:47:04.71674Z",
  "deployment": {
    "version": "trstctl dev (commit none, built unknown, darwin/arm64, go1.26.10)",
    "write_probe": true
  },
  "probes": [
    { "id": "ISO-1", "group": "tenant-isolation (AN-1)", "status": "pass",
      "detail": "182/182 tenant tables ENABLE + FORCE row level security",
      "limits": "proves the catalog posture of every tenant table now, not the absence of RLS-bypassing SQL elsewhere",
      "evidence": { "source": "pg_class", "tables_checked": 182 } },
    { "id": "ISO-3", "group": "tenant-isolation (AN-1)", "status": "pass",
      "detail": "cross-tenant read returned 0 rows (expected 0)",
      "limits": "proves the policy held for this table and this path, not that no bug exists anywhere" },
    { "id": "ISO-CLEAN", "group": "tenant-isolation (AN-1)", "status": "pass",
      "detail": "probe tenants fully cleaned up (verified zero residue)" },
    { "id": "SIG-1", "group": "signer-isolation (AN-4)", "status": "skip",
      "detail": "the signer health RPC intentionally reports only serving status, not a PID; separate-process proof stays with the AN-4 dependency-closure gate in CI" }
  ],
  "summary": { "pass": 11, "fail": 0, "warn": 0, "skip": 15 }
}

With --sign, a signature object is appended carrying alg RS256, key_id audit-export, and a compact JWS over the receipt's canonical JSON — the receipt serialized with the signature field absent. This is the same key and the same signer-side artifact path the audit bundle export already uses. The protected JWS header binds trstctl.audit-evidence/doctor-receipt/v1; the signer rejects an unknown kind, wrong authority, or wrong handle. Doctor introduces no private-key parser, key file, or second signing path.

How this page stays honest

docs/doctor_doc_test.go asserts that every probe id the code emits appears in the tables above, that every flag the command parses is documented, and that the exit-code contract stated here matches the code. Add a probe without documenting it and the build fails.