Operations & resilience

This page is for the operator responsible for keeping a running trstctl deployment available. It explains the controls, signals, and recovery behavior for overload, slow external systems, shutdown, and signer failure. Before changing production, have a verified backup, alert access, the current configuration, and a tested rollback path. Start diagnostics in Trust Operations, then use /healthz, /readyz, and /metrics to separate service health from tenant work.

The serving control plane is built so one overloaded or failing part cannot take down the rest: each subsystem runs in its own bounded lane and rejects fast when full. This page covers the resilience controls in the live path: bulkheads, the per-tenant rate limiter, graceful drain, and the fail-closed signer timeout.

Notification delivery evidence

Open an alert in Alerts and delivery to see which channels accepted it and which routing policy selected each successful channel. These receipts describe transport acceptance; they do not prove that a person read the alert.

The dispatcher records each successful channel separately. A failed channel can retry without sending again to channels whose receipts already exist. The receipt keeps the policy ID, policy level and a SHA-256 digest of the routing settings used for that delivery. Later policy changes or deletion do not rewrite that history. Renewal failures follow the affected identity's route, even when the alert also carries the certificate from its last completed deployment.

GET /api/v1/notifications/{id} includes deliveries for the exact tenant, notification destination, idempotency-key digest and payload digest. A receipt's routing_source distinguishes an explicitly requested policy, an automatically selected policy, configured defaults, all-channel fallback and a direct channel test. Historical receipts without that field have unknown routing history; the console never fills it from today's configuration. Only successful effects have receipts. Pending or failed delivery still needs investigation. If the detail read fails, the console labels the displayed snapshot as potentially stale and offers a retry.

Investigating a failed renewal attempt

The Renewal reliability panel on Jobs and queues reads the tenant's GET /api/v1/operations/renewal-slo result. It shows the measurement window, successful and failed rotation counts, the denominator, and whether the measured rate meets the configured target. An empty window says no renewals completed; the server's empty-window percentage is not presented as a measured success rate. If the read fails, the panel reports that reliability is unavailable. Use the exact rotation run and its host-job evidence below to investigate a failure; an aggregate rate does not prove any one target was deployed.

A failed renewal warning records the host job and attempt that failed. Open Open exact run and current retry status to see whether that same run is still working, recovered, failed, or was canceled. The historical warning does not change when a later attempt succeeds.

The link uses /operations?run=<run-id> and reads the exact run, including runs outside the currently loaded history page. For a bound host job, the detail shows its current status, number of attempts started, and terminal completion time. An attempt failure can coexist with an ongoing retry; notification delivery attempts count a different operation.

GET /api/v1/notifications/{id} returns rotation_run_id, renewal_job_id, and renewal_attempt when retained. The server resolves these from the authenticated job claim and checks the tenant, identity, run key, and predecessor certificate. GET /api/v1/lifecycle/rotation-runs/{id} includes optional host_job metadata for that exact retained child command. Missing historical bindings remain unknown; the console does not choose the latest run as a substitute. The run detail refreshes every ten seconds while visible. A failed read displays an error and labels any previous result as the last successful read, rather than proof of current status.

Alerts opens with the newest 100 notifications. Use Older alerts and Newer alerts to page through history, or Latest alerts to return to the current page. Counts and filters describe the displayed page, not the entire notification history. List readers can request order=desc; retain that order with every cursor. The API default remains order=asc for existing clients. New notifications do not shift an older page's ID cursor.

Bulkheads (isolation + backpressure)

Each subsystem runs on its own bounded worker pool with a bounded queue: the API, projection workers, outbox dispatcher, signing path, heavy query path, policy engine, served issuance protocols, and the agent steady-state gRPC channel. When a pool is saturated it rejects fast rather than blocking — an API flood returns 503 with a Retry-After header, and an agent heartbeat/renewal flood returns gRPC ResourceExhausted with retry guidance, instead of consuming capacity another subsystem needs.

Because the pools are isolated, a saturated API cannot starve the things you rely on to observe and recover: /healthz, /readyz, and /metrics are served outside the API bulkhead and keep answering even while the API sheds load. The continuous outbox dispatcher runs on its own pool, so a backlog of external calls applies backpressure to itself (it sheds a sweep rather than piling up) without touching API capacity. The agent pool similarly isolates reconnect storms and certificate-renewal waves from the API/protocol/outbox pools, and the agent gRPC listener also caps streams per connection.

The pool sizes ship with conservative defaults and are tuned per deployment.

Datastore backpressure and idempotent retries

The PostgreSQL pool is bounded too. A request that cannot obtain a connection within the acquire window (10 s by default, TRSTCTL_POSTGRES_ACQUIRE_TIMEOUT), or whose statement is canceled by the server-side statement deadline, or that runs into its own request deadline while the datastore is busy, is refused with 503 and a Retry-After header; the problem detail says so ("datastore is busy ... retry"). It is safe to retry such a request with the same Idempotency-Key: the mutation either never claimed the key (the retry executes it) or completed (the retry replays the recorded response).

Two more answers are part of the idempotency contract under concurrency:

• 409 with Retry-After, "still in progress": an identical request with the same Idempotency-Key is executing at that moment (a client retry storm, or two replicas). The control plane waits briefly for it; if it has not finished, the retry is told to come back. The next retry replays the recorded response.
• 409, "effect is indeterminate": the original request's command ran but its result could not be confirmed or recorded (for example an immutable CA event was acknowledged but the SQL projection could not be read back). The key is walled rather than re-executed, because re-running could duplicate the effect. Inspect the resource and audit first; use a new key only after confirming the effect is missing and following the affected subsystem's recovery steps.

Connection budget

Each replica opens five PostgreSQL pools from the same DSN. The request pool carries client commands; the four small pools carry work that must keep moving while the request pool sheds load, and request handlers can never borrow from them:

The default budget is 32 connections per replica (each small pool keeps one connection warm, so an idle replica holds about five). Size PostgreSQL so that max_connections ≥ replicas × 32 + superuser_reserved_connections + every other client of the database (backup tooling, the doctor, dashboards). Every pool is opened and pinged at startup: a PostgreSQL that cannot honor the budget fails the process closed with a message naming the budget and the settings, instead of starving at runtime. The control plane logs store: connection budget at startup and exports trstctl_store_pool_connections{pool,state} (max, total, acquired, idle) on every /metrics scrape.

A tenant burst that holds every request-pool connection is shed with 503s as above, but it cannot make /readyz time out (probe pool), starve the tail into "projection tail worker stopped; retrying" (reserved pool), leave a completed command's result unrecorded and wall its key as "indeterminate" (bookkeeping pool), or convoy commands behind the projection lock in acquire-window steps (lock pool).

Readiness after a restart

ssh_krl brings the served SSH revocation list up to the event log — the same step a KRL request takes — and fails while it cannot: behind by more than one request's replay page, or on a revocation generation this issuer has not reconciled (trstctl ssh krl adopt). Each probe advances one page, so a lagging replica catches up and turns ready by itself. audit_read takes the history read an audit query takes and fails while history reads are refused, such as while a restored history awaits cutover. Both read ok on a deployment without the SSH CA or audit service. Before them, /healthz and the other checks could report a restarted replica ready while its KRL answered 503 and its audit reads failed.

Readiness under load

The db check pings the probe pool and fails readiness when PostgreSQL is unreachable. The datastore-reading checks (nats samples, projection, secret_sync_recovery_authority, application_secret_reconciliation) read through the request pool under a one-second budget. When one of them is shed by load (the store's busy signal, or the budget runs out) /readyz still answers 200 so the replica stays in rotation, but the check reads degraded: …, the body carries a degraded list naming the checks, and trstctl_readiness_probe_shed{check} is 1 for as long as it lasts. Shedding that outlasts the two-minute grace window fails the check: a datastore that has been too slow for that long is a stall, not a burst. A check that fails for a real reason (a stalled projection, a missing recovery authority) fails readiness immediately, as before.

For 503 with sqlstate 40001 or 40P01 and retryable: false, PostgreSQL rolled back one transaction after a concurrent write. That does not prove the whole command had no effect: it may already have appended an event. Inspect the resource and audit before retrying with the same Idempotency-Key. Event-aware CA creation reconciles its acknowledged event and exact projected authority before returning success; if it cannot confirm them, it answers the walled 409 above. Anything that still answers 500 is logged by the control plane (api: internal error answered as 500) with the response's traceparent, the error's type chain, the SQLSTATE and constraint when it is a PostgreSQL error, and a message with every quoted literal masked; row values never reach the log.

Approval decisions (/api/v1/approval-requests/{id}/approvals and /denials, /api/v1/identities/{id}/approvals) run on this claim path too: identical in-flight decisions execute once and replay; a retry that arrives after the idempotency retention window re-executes the decision and finds it already recorded for that approver, so the answer is the recorded decision and never a second one.

Every mutation claims its Idempotency-Key in a short transaction, runs the command with no pooled connection held, and records the result in a second short transaction (the same discipline the outbox uses for external calls), so parallel writes never deadlock on the pool waiting for each other. A claim abandoned by a crashed process is taken over after ten minutes.

Outbox delivery fairness

The outbox worker does not keep a PostgreSQL transaction open while it calls an external CA, connector, webhook, or notification target. It first leases one due row in a short transaction (processing, worker_id, lease_until), commits, does the external call, then records success or retry state in a second short transaction. If a worker dies after claiming a row, the lease expires and another worker returns the row to pending.

Dispatch is also fair by tenant and destination. Each sweep rotates across tenants and destinations, with explicit in-flight caps per tenant and per destination, so one down connector or one noisy tenant cannot occupy every outbox worker while unrelated tenants wait behind it.

Each delivery also has a per-message deadline. If a connector, plugin, webhook, or notification target does not return before that deadline, the row is marked pending again through the normal retry/backoff path, and the served binary increments trstctl_outbox_delivery_timeouts_total{destination}. That counter is the operator's direct signal that one destination is timing out without starving the rest of the outbox.

Rate limiting (per tenant, PostgreSQL-backed)

A per-tenant token bucket, persisted in PostgreSQL (no Redis — the limit holds across every replica), sheds load on the guarded routes: each tenant may make requests calls per window, admitting a burst of requests and refilling steadily. Over-budget requests get 429 Too Many Requests with a Retry-After header. The check runs after authentication and authorization, so one noisy tenant cannot exhaust the control plane while others are unaffected.

Graceful drain on shutdown

On SIGTERM the control plane drains without losing in-flight work: it stops accepting new connections, stops the outbox dispatcher, drains the per-subsystem worker pools (finishing queued and running tasks), runs a final outbox sweep so no enqueued external effect is lost (the journaled outbox guarantees at-least-once delivery), then closes the event log and datastore in order.

Fail-closed signing

Issuance is bounded by a per-operation timeout. If the out-of-process signer is slow, unreachable, or stopped, IssueLeaf fails closed — it returns an error within the timeout and never falls back to an in-process signature. This is exercised by fault injection (a deliberately slow signer) in the test suite.

What an operator should watch

Pair this with Observability: the trstctl_http_requests_total counter shows 429/503 shedding as it happens, and the alert rules fire on sustained error rate or latency. A rising 503 rate points at a saturated subsystem; a rising 429 rate points at a tenant over budget. A rising trstctl_outbox_delivery_timeouts_total series points at a slow destination and includes the affected tenant and destination labels.

For metadata-only SIEM pipelines, enable OTLP export to your OpenTelemetry Collector. Served HTTP spans arrive as OTLP traces, and event-sourced audit records arrive as OTLP logs with trstctl.audit.sequence and trstctl.tenant.id attributes. Use the sequence attribute to dedupe restarted streams and alert on gaps.

For native evidence delivery, configure a Splunk HEC or Microsoft Sentinel feed and watch GET /api/v1/audit/feeds. A non-zero lag_records means one exact batch is still owed; retrying is recoverable and carries next_attempt_at; failed is terminal and keeps its safe error code and collector request ID. Re-saving a corrected configuration emits a new immutable configuration version. Never repair the cursor, outbox row, or feed projection with SQL: startup reconciliation checks the queued event against the retained audit range and either recreates the exact intent or fails closed.

After upgrading a deployment that issued managed-CA leaves with version-2 ca.endentity.issued events, allow the first control-plane start to complete its retained-history check. It atomically rebuilds the read model if it finds a responder-only completion without a certificate inventory receipt. The recovery cursor checks only newly projected events on later starts, including events written after a binary rollback. If startup reports a public leaf or issuer signature mismatch, inspect the retained event and CA authority; do not insert an inventory row directly or reissue merely to clear the error.

Dynamic PKI secrets issued before ca.certificate.issued version 3 kept only a responder serial. On the first lifecycle sweep after an upgrade, the control plane recovers each version-2 leaf the issuing CA signed into inventory, once, through a ca.certificate.inventory_recovered event, and logs how many it recovered and how many it could not verify. An unverifiable leaf stays serial evidence: the issuing CA may have changed since it was minted. A version-1 mint kept only the serial; submit the leaf's public certificate with trstctl certificates inventory-pki-secret -f leaf.pem to give it the same inventory row once the issuing CA's signature and the recorded serial match. The same sweep recovers a managed-CA leaf recorded only in the version-2 responder form of ca.endentity.issued when no other event inventoried it, checking it against that managed CA's stored certificate. A managed-CA leaf a version-1 ca.endentity.issued mint recorded by serial only joins inventory the same way once its public certificate is submitted with trstctl certificates inventory-managed-ca -f leaf.pem: the managed CA's stored certificate must have signed it and that CA's version-1 mint must have recorded its serial.