Performance SLOs

This page is the committed performance contract for the served trstctl hot paths. It is intentionally concrete: every row has a PERF-SLO-* identifier, an owner, latency targets, a minimum smoke-throughput target, queue/projection limits, and a benchmark name. The local smoke gate writes the current measurement receipt:

scripts/perf/run-local.sh --profile smoke --out /tmp/trstctl-perf-smoke.json

The committed baseline receipt is scripts/perf/artifacts/smoke-baseline.json. The smoke profile is a fast CI guard over representative product code paths. It is not a substitute for the served live-load receipt, a multi-hour soak, or a customer-specific load test, but it turns the hot-path denominator into executable release evidence.

Served live-load gate

The served live-load profile boots the local eval perf stack, drives each HTTP-addressable PERF-SLO-* hot path through its production route family, and exercises the signer path through the generated signer gRPC service in a separate child process over a real Unix-domain socket. The stack uses an actual loopback HTTP listener, bundled PostgreSQL, and embedded file-backed JetStream. The projection row drives the event replay/apply loop directly because it is a worker surface, not an HTTP endpoint. In plain terms: the harness enters through the same doors and crosses the same process and storage boundaries as the product; it rejects test-only HTTP routes, httptest, bufconn, and in-process signer shortcuts. Customer load runs should repeat the profile with their production PostgreSQL, JetStream, and UDS or mTLS signer placement.

make perf-live
scripts/perf/run-local.sh --profile live --out /tmp/trstctl-perf-live.json

The committed live receipt is scripts/perf/artifacts/live-load-baseline.json. Each SLO row must have both realistic and peak phase measurements with p50, p95, p99, max latency, throughput, error count, queue saturation, projection lag, and resource metrics. The receipt also carries an event_spine_burst reference to the CAP-SMALL PostgreSQL/JetStream replay and bounded-outbox gate so live review does not lose the heavier spine evidence. The live profile is still a local eval-stack receipt, not a promise that one vendor SKU will satisfy every production tenant shape; customer capacity reviews should run the same profile against their chosen datastore, signer placement, and connector mix.

make test builds the isolated signer before timed live work. The serial live packages keep race detection and whole-repository atomic coverage for correctness; both the live-load and soak-capture stack startups use the explicit four-minute instrumented harness allowance. The exact live mutation SLO test then runs once without race or coverage instrumentation, with its original 90-second startup deadline and unchanged latency, throughput, error, queue, and lag limits. make test prints every measurement and reports SLO misses without failing the functional target; operation errors and stack failures still fail it. Direct execution of the test still enforces the SLO. Instrumented timings are never release performance evidence. Run the uncached make perf-live-wall release gate alone on a quiet machine against a stored baseline from that same machine and configuration:

PERF_LIVE_OUT=/path/on/this/host/live-baseline.json make perf-live
PERF_LIVE_BASELINE=/path/on/this/host/live-baseline.json \
  PERF_LIVE_OUT=/path/on/this/host/live-candidate.json make perf-live-wall

The first command saves the baseline only if the existing absolute SLOs pass. Both receipts carry a hashed machine fingerprint. The second command fails on a different host, incompatible phase/sample set, or an absolute SLO failure, and prints p99 and throughput side by side; it adds no new relative SLO budget. Record the two receipts and the host configuration with the release decision. CI's scheduled make perf-live receipt is supplemental when its runner differs from the baseline host.

The shipped Prometheus alert pack mirrors this table. The trstctl-slo-hot-paths rule group in deploy/observability/alerts.yml records p99 latency and 5m/1h error ratios for every PERF-SLO-* row, then fires a latency threshold alert and a 14.4x/6x multi-window burn-rate alert against the committed 0.10% error budget. internal/observ tests enumerate internal/perf.HotPaths() and fail if a new SLO row lacks matching PromQL coverage. PERF-SLO-007 and PERF-SLO-008 currently use served route-family alert denominators plus the direct signer/projection health alerts; first-class signer RPC and projection replay histograms remain the next precision upgrade.

Endurance / soak gate

Sustained-load behavior — memory/heap/goroutine/FD leak slopes, DB-pool saturation, projection/outbox lag, queue rejects, signer restarts, and storage growth — is held to a pass/fail threshold contract by an executable soak gate:

make soak                      # self-test: an induced leak series MUST fail, a healthy series MUST pass
scripts/perf/soak.sh --in <series.json> --out <report.json>   # analyze a captured sustained-load series
make soak-capture              # capture local eval-stack samples, then analyze them with --in
make spine-burst               # capture embedded Postgres/JetStream replay + outbox burst, then analyze it with --in

The threshold contract and the trend analyzer are shared by this gate, make soak, and these docs so they consume one denominator — the same pattern as the smoke gate. The gate exits non-zero on a leak slope or an SLO breach and emits a JSON trend report. The scheduled CI soak job captures a real local eval-stack series with scripts/perf/capture-soak-series.sh, feeds that series into scripts/perf/soak.sh --in, uploads the trend report, and separately proves an induced leak substitute fails the gate.

The spine-burst gate is the focused event-spine capacity receipt for SPINE-002:

scripts/perf/run-spine-burst.sh --profile cap-small --out scripts/perf/artifacts/spine-burst-cap-small.json
scripts/perf/soak.sh --in scripts/perf/artifacts/spine-burst-cap-small.json

It starts embedded PostgreSQL, applies migrations, seeds tenants and agents, starts embedded JetStream, appends the cap-small event workload, replay/decode-applies the event log with a bounded projection-lag target, injects a slow upstream destination through the outbox, and records projection lag, outbox backlog, queue rejects, DB pool utilization, p95/p99 latency, and resource counters. The committed receipt is scripts/perf/artifacts/spine-burst-cap-small.json. When that receipt is analyzed with scripts/perf/soak.sh --in, the trend report includes input_evidence with the source, workload, slow-upstream, appended/replayed event counts, and outbox backlog summary from the burst.

PERF/RUNOPS can run the same harness against external capacity profiles:

export TRSTCTL_POSTGRES_DSN='postgres://trstctl:...@perf-db:5432/trstctl?sslmode=require'
export TRSTCTL_NATS_URL='nats://perf-nats:4222'
export TRSTCTL_NATS_REPLICAS=3
SPINE_BURST_PROFILE=cap-medium make spine-burst
SPINE_BURST_PROFILE=cap-large make spine-burst

cap-medium and cap-large fail closed without external PostgreSQL and external JetStream, and write spine-burst-cap-medium.json or spine-burst-cap-large.json unless SPINE_BURST_OUT overrides the path.

Gates

The fast local gate:

scripts/perf/run-local.sh --profile smoke

The served local live-load gate:

make perf-live
scripts/perf/run-local.sh --profile live

The event-spine burst gate:

make spine-burst

The Go benchmark denominator (the Benchmark* targets named in the SLO table above), and the broader benchmark discovery command used for release review:

go test -run '^$' -bench=. ./...

CI runs the smoke profile and uploads the JSON receipt as a workflow artifact. Release review compares the smoke receipt, the served live-load receipt, the capacity calibration receipt at scripts/perf/artifacts/capacity-measurement-baseline.json, the spine-burst receipt at scripts/perf/artifacts/spine-burst-cap-small.json, and the capacity model in docs/performance-capacity.md.

CAP-SCALE-01 is also served as operator-facing posture: GET /api/v1/scale/orchestration and trstctl-cli scale orchestration return the 100k, 250k, and 1M credential bands, execution lanes, bulkhead/backpressure controls, release gates, and residuals tied to this performance contract.

CAP-SCALE-02 is a regional HA issuance reference architecture: GET /api/v1/scale/ha-issuance and trstctl-cli scale ha-issuance return example regional ingress lanes, tenant write fences, failover gates, and 5s/30s RPO/RTO targets. served: true means the contract route is available; evidence_scope: reference_architecture makes clear that this response does not discover deployed regions or measure recovery. Qualify each installation with regional issue/renew/revoke smoke and a multi-host failover drill before claiming HA or those objectives. The targets assume a healthy shared or promoted PostgreSQL writer endpoint, replicated JetStream event log, isolated signer placement, and green regional smoke. Independent split-brain writers remain unsupported.