Configuration

trstctl resolves its configuration from, in increasing precedence: built-in defaults, an optional JSON config file (TRSTCTL_CONFIG_FILE), and environment variables. The configuration is validated on boot — a bad combination fails fast rather than starting half-configured.

Inspect the effective configuration at any time (credentials are redacted):

trstctl -check-config

Server

Transport encryption (TLS)

The control plane serves over TLS by default so no credential, token, or session ever travels in cleartext.

• internal (default) — the control plane creates a self-signed certificate once and reloads it from server.tls.internal_state_file on later starts. The certificate covers localhost, 127.0.0.1, the first-boot hostname, and the Compose service name trstctl. Existing malformed, expired, or group/world- readable state fails closed instead of rotating trust silently. The server atomically publishes the matching certificate-only PEM at server.tls.internal_trust_file; clients verify with that file instead of disabling TLS checks. This is suitable for evaluation and tightly controlled internal or air-gapped use. Public deployments must use server.tls.mode=file with an operator-provided certificate chain from your CA; do not expose the eval self-signed certificate to public clients.
• file — the control plane presents an operator-provided certificate and key. Use this in production with a certificate from your CA. A missing or malformed file fails fast at startup rather than falling back to plaintext.
• disabled — plaintext HTTP. Local development only and mechanically bounded: startup fails unless TRSTCTL_DEV_ALLOW_PLAINTEXT=true and TRSTCTL_SERVER_ADDR is loopback-only (localhost, 127.0.0.1, or ::1). Production TLS termination should use server.tls.mode=file at trstctl or a TLS-terminating proxy in front of a TLS-enabled trstctl listener; disabled mode is not the production proxy pattern.

The control-plane↔signer channel (the private keys live in a separate, isolated process) is independent of this setting. The default (single-binary child mode, and external mode with signer.socket) is a peer-authenticated Unix domain socket — a 0600 socket in a 0700 directory, restricted to the signer's own uid via SO_PEERCRED on Linux — not a TLS channel. For a separately-hosted signer across nodes, set signer.mtls_address (with the signer.mtls_* certificate material): the control plane then reaches the signer over mTLS — TLS 1.3, AEAD-only, with the control plane and the signer each pinning the other's certificate (an untrusted or merely CA-signed-but-unpinned peer is rejected, fail-closed). Exactly one of signer.socket or signer.mtls_address is used in external mode; a partial mTLS block fails closed at startup.

Datastores

trstctl stores its read state in PostgreSQL (the source-of-truth event log lives in NATS JetStream). PostgreSQL is the datastore in every deployment mode — there is no SQLite path.

!!! important "Datastores: bundled single-node for eval, external for production" The serving binary (trstctl, via server.Run) can run a single-node eval stack: bundled PostgreSQL (TRSTCTL_POSTGRES_MODE=bundled, the default — the binary starts and supervises an embedded single-node Postgres with data under TRSTCTL_POSTGRES_DATA_DIR on TRSTCTL_POSTGRES_PORT, default 5432) and embedded NATS (TRSTCTL_NATS_MODE=embedded, the default — in-process file-backed JetStream). Bundled PostgreSQL is available only for host archives with committed runtime pins in deploy/supply-chain/embedded-postgres.json (summarized in Supply chain): currently linux-amd64, linux-arm64v8, and darwin-arm64v8. It downloads that pinned PostgreSQL runtime once on first use, verifies the cached archive before execution, and fails closed if the host archive is unsupported, unpinned, or hash-mismatched. For production, use external for both: TRSTCTL_POSTGRES_MODE=external with TRSTCTL_POSTGRES_DSN and TRSTCTL_NATS_MODE=external with TRSTCTL_NATS_URL, which the Compose stack and Helm chart wire up. There is no silently-failing default: an invalid mode — or external without a DSN — fails fast at startup. (External mode never downloads anything. --migrate / --backup target a managed datastore and require external.)

External datastores

To point trstctl at managed PostgreSQL and NATS, switch both to external mode and supply their connection strings:

export TRSTCTL_POSTGRES_MODE=external
export TRSTCTL_POSTGRES_DSN='postgres://user:pass@db.internal:5432/trstctl?sslmode=require'
export TRSTCTL_NATS_MODE=external
export TRSTCTL_NATS_URL='nats://nats.internal:4222'
export TRSTCTL_NATS_REPLICAS=3

When a mode is external, its connection string is mandatory; trstctl refuses to start without it. External NATS also refuses to serve under-replicated: the event stream defaults to three replicas, startup fails on a non-clustered single NATS server, and /readyz reports degraded if the observed stream later has fewer replicas than configured. The Docker Compose eval stack uses the same external code path but explicitly sets TRSTCTL_NATS_REPLICAS=1 and TRSTCTL_NATS_ALLOW_SINGLE_REPLICA=true; keep that opt-in out of production.

The control plane also creates a file-backed TRSTCTL_EVENT_IDS_<epoch> JetStream stream for exact event-ID recovery. It uses the same replica count as the source event stream. Give the control-plane NATS account permission to create, list, read, publish to, and delete these streams and their trstctl_event_ids.> subjects. Startup rebuilds a missing index from the retained event log before serving; privacy rewrites, physical deletion, and authenticated restore discard old generations. An index that has not covered the current event head refuses to report an ID absent; each request advances at most 1,024 events and a retry resumes from its durable cursor. Retain the source event log for rebuilds. The index is derived state, not a backup of the source.

Schema migrations

trstctl embeds its PostgreSQL schema as versioned SQL migrations and applies them itself; there is no separate migration binary.

trstctl --migrate applies pending migrations under the advisory lock and exits; trstctl --migrate-status lists the pending plan (dry run) and exits. See Database migrations & upgrades for the full upgrade/rollback runbook.

High availability (leader election and snapshots)

trstctl is safe to run as more than one control-plane replica against one shared external PostgreSQL and NATS (RESIL-002 / RESIL-004). With the defaults, a single replica behaves exactly as before; adding replicas needs no extra configuration beyond pointing them at the same datastores.

Cross-cluster federation

Federation is disabled by default. When enabled on a passive cluster, the leader worker imports a peer's event log into the local event log, advances a durable peer cursor, and projects the imported events locally. The passive region therefore serves from its own PostgreSQL and NATS after failover; it does not read the primary region's PostgreSQL tables.

JSON config can declare multiple peers under federation.peers; the environment overlay above configures one common peer. Keep ha.leader_election on so one replica owns imports while all replicas serve the replicated read state.

Lifecycle

How far ahead of expiry trstctl renews and alerts. Values are Go durations.

Native connector and external-CA assembly

The 24 native deployment connectors and 14 external-CA drivers are compiled into trstctl, but start deny-by-default: the operator chooses the exact integrations to construct. The connector allowlist and its shared network policy — which native connectors may be constructed, the shared HTTP timeout, private-CIDR grants, and the loopback-only insecure-HTTP escape hatch — are flattened into the environment variables below.

The tenant-bound structured bindings — connectors.local_profiles (local execution profiles), connectors.right_size (tenant-to-endpoint bindings), and every external_cas entry — remain JSON/YAML config-file only, because they carry tenant/provider associations where a parallel-list typo could cross a security boundary.

A local connector needs both an enabled driver and an operator-owned execution profile. Tenant target JSON can then select that profile, but cannot invent a command or escape its allowed roots:

{
  "connectors": {
    "enabled": ["nginx", "f5"],
    "http_timeout": "15s",
    "allow_private_cidrs": ["10.40.0.0/16"],
    "local_profiles": {
      "nginx-prod": {
        "allowed_roots": ["/etc/nginx/tls"],
        "actions": [{
          "logical_name": "nginx",
          "command": "/usr/sbin/nginx",
          "pass_args": true,
          "timeout": "15s"
        }]
      }
    },
    "right_size": [{
      "tenant_id": "11111111-1111-4111-8111-111111111111",
      "connector": "least-privilege",
      "endpoint": "https://entitlements.internal",
      "token_ref": "secret://connectors/right-size-token",
      "executor": "network"
    }]
  }
}

Local action commands must be absolute, executable, regular non-symlink files. Unix shell interpreters are rejected. The one native-administration exception is IIS PowerShell: its profile must pin the complete connector logical_args, set a separate complete operator-owned args list, and leave pass_args false. The runtime compares the connector request with that fixed logical argv and executes only the fixed operator argv, so target data never becomes PowerShell source text.

connectors.right_size binds one tenant plus the playbook's connector name to an operator endpoint and a same-tenant encrypted-secret reference. The durable connector.right_size worker sends an idempotent PATCH, verifies the vendor's mutation receipt, performs an authenticated GET readback, and only then records a delivered connector receipt. Store the referenced token through the tenant secrets surface (secrets.enable_api=true) before enabling the binding. An unconfigured tenant/connector pair fails closed. executor is empty or control_plane by default; network hands each operation to an enrolled network relay in that tenant, which redeems the token for one attempt, runs the same mutation and readback from inside its segment, and signs the evidence. For that, enable connector.right_size in agent_channel.claimable_job_kinds; until a relay claims it the operation stays queued rather than running on the control plane.

External-CA credentials use file:/absolute/path references. Files are loaded into locked memory for one outbox attempt and wiped afterward. Network policy, private CIDRs, custom roots, and mTLS identities are operator-owned:

{
  "external_cas": [{
    "id": "aws-pca-prod",
    "type": "awspca",
    "name": "Production AWS Private CA",
    "endpoint": "https://acm-pca.us-east-1.amazonaws.com",
    "region": "us-east-1",
    "certificate_authority_arn": "arn:aws:acm-pca:us-east-1:123456789012:certificate-authority/UUID",
    "access_key_id": "AKIA...",
    "secret_access_key_ref": "file:/run/secrets/aws-pca-secret",
    "network": {"timeout": "15s"}
  }, {
    "id": "azure-managed-hsm-ca",
    "type": "azurekv",
    "name": "Azure Managed HSM Issuing CA",
    "tenant_id": "11111111-1111-4111-8111-111111111111",
    "endpoint": "https://production.managedhsm.azure.net",
    "managed_key_ref": "https://production.managedhsm.azure.net/keys/issuing-ca/0123456789abcdef",
    "ca_cert_file": "/etc/trstctl/azure-issuing-ca-chain.pem",
    "network": {"timeout": "15s"}
  }, {
    "id": "entrust-prod",
    "type": "entrust",
    "name": "Entrust CA Gateway",
    "endpoint": "https://entrust-ca.internal",
    "ca_id": "production-ca",
    "network": {
      "root_ca_file": "/etc/trstctl/entrust-server-ca.pem",
      "client_cert_file": "/etc/trstctl/entrust-client.pem",
      "client_key_file": "/run/secrets/entrust-client-key.pem",
      "server_name": "entrust-ca.internal",
      "timeout": "15s"
    }
  }, {
    "id": "letsencrypt-prod",
    "type": "letsencrypt",
    "name": "Let's Encrypt",
    "directory_url": "https://acme-v02.api.letsencrypt.org/directory"
  }, {
    "id": "harica-tcs",
    "type": "letsencrypt",
    "name": "HARICA (GÉANT TCS)",
    "directory_url": "https://acme.harica.gr/acme/directory",
    "eab_key_id": "kid-issued-by-the-ca",
    "eab_hmac_key_ref": "file:/run/secrets/harica-eab-hmac",
    "acme_profile": "tcs-ov",
    "ari_issuer_key_ids": ["<hex subject key identifier of the issuing intermediate>"]
  }, {
    "id": "shell-ca-prod",
    "type": "shellca",
    "name": "Isolated Shell CA",
    "command": "/usr/local/libexec/trstctl-shell-ca",
    "args": ["--profile", "production"],
    "env_refs": {"CA_TOKEN": "file:/run/secrets/shell-ca-token"},
    "network": {"timeout": "15s"}
  }]
}

External-CA endpoints require HTTPS. The separate network.allow_insecure_http escape hatch is accepted only for an HTTP localhost, 127/8, or ::1 development emulator; the runtime client also refuses any non-loopback resolution or cross-origin/scheme-changing redirect. allow_private_endpoint and private CIDR grants never authorize plaintext.

The Azure CA block carries only routing and public trust material. tenant_id binds the authority to one tenant, managed_key_ref is the opaque HTTPS key id returned by that tenant's managed-key lifecycle, and ca_cert_file is the public issuer chain whose leaf public key must match the managed key. Do not put an Azure bearer token, key_name, or key_version in external_cas: Azure credentials and private-key operations live in the isolated signer's managed_keys backend.

Let's Encrypt account JWS signatures also cross the signer transport. The control plane derives a stable account handle from the external-CA id and directory URL, but it receives only the public key and signatures; the ECDSA account private key never enters the HTTP process.

Entrust always requires the complete network mTLS identity shown above. The client pins root_ca_file, verifies server_name, and presents the paired client certificate/key. Missing, partial, or untrusted client material fails before an Entrust enrollment request is sent; plaintext HTTP is not an Entrust mode.

For shellca, each env_refs key is a logical credential name, not a secret-valued environment variable. The child sees CA_TOKEN_FD=<number> and reads the exact credential bytes from that inherited anonymous pipe. It never receives CA_TOKEN, and trstctl consumes and wipes the byte buffer after the one child attempt. No credential pathname or filesystem-backed secret copy is created for the child.

Supported type values are adcs, awspca, azurekv, digicert, ejbca, entrust, gcpcas, globalsign, letsencrypt, sectigo, shellca, smallstep, vaultpki, and venafi. trstctl -check-config validates every provider's required fields before the server accepts traffic. The exact connector target schemas are listed in Deployment connectors.

Vault PKI: keep the existing signing authority

Start with the complete Vault configuration example. Merge its external_cas row into the server JSON selected by TRSTCTL_CONFIG_FILE; keep your other configured authorities and settings. Replace these example values:

• id: the stable registry ID that tenant lifecycle plans will select.
• endpoint: the HTTPS address of the real Vault server, without /v1 or the PKI path.
• mount and role: the existing PKI mount and signing role. For the example, trstctl sends CSRs to /v1/pki/sign/web-certs.
• bearer_token_ref: an operator-owned file containing only the Vault token. Give that token update permission on the selected signing path; add update on pki/revoke only if this integration should revoke certificates. Keep the file readable only by the service account and mount it read-only into the control plane.
• network.root_ca_file: the public CA certificate that verifies Vault's HTTPS server. This is transport trust; it need not be the CA that signs workload CSRs.
• network.private_egress_cidrs: the exact private address ranges Vault resolves to. The example grants one host, 10.40.0.15/32; replace it with your inspected address. Keep allow_private_endpoint: true for a private Vault endpoint. Neither setting permits plaintext or disables TLS verification.

Run trstctl -check-config with the same environment and mounted files used by the service. This checks configuration, not live connectivity. Restart the control plane, then confirm trstctl-cli external-cas list returns your exact ID and type: vaultpki. With the evaluation Compose stack, recreate trstctl and oidc-loopback together so the sign-in companion joins the replacement network namespace; keep the existing data and TLS volumes.

The console's Configure Vault PKI action explains these operator steps. It does not create a metadata-only issuer row as a substitute for the connection. Never paste the Vault token into a tenant form.

Continue with Keep your existing CA: use the configured registry ID in the lifecycle preview, and verify actual issuance, product deployment and the listener separately. A registry row alone proves none of those outcomes.

Authority agreement

Authority agreement compares the certificate inventory with the internal CA's issuance ledger. It helps identify certificates present on only one side or with different revocation states. The Posture panel and GET /api/v1/reconcile/agreement require certs:read and return only the authenticated tenant's authorities, counts, resolution metrics and schedule status.

Add a schedule to the control plane's configuration file, using the tenant's actual ID, then restart the control plane:

{
  "reconcile": {
    "schedules": [{
      "tenant_id": "YOUR_TENANT_ID",
      "cadence": "1h",
      "jitter": "5m",
      "liveness": "24h",
      "authorities": [
        {"authority_id": "trstctl-self"},
        {"authority_id": "trstctl-ca"}
      ]
    }]
  }
}

trstctl-self reads captured certificates in the inventory; trstctl-ca reads the internal CA ledger. Both compare certificates within that tenant's internal CA jurisdiction. External-CA certificates and inventory entries without captured certificate bytes do not participate in this comparison. This schedule does not configure a connection to an external CA.

An external HashiCorp Vault or OpenBao server joins a round as its own authority when the authority entry carries a source block:

{"authority_id": "vault-prod", "source": {
  "kind": "vault",
  "address": "https://vault.internal:8200",
  "token_file": "/run/secrets/vault-observe-token",
  "namespace": "",
  "kv_mounts": ["secret"],
  "pki_mounts": ["pki"],
  "transit_mounts": ["transit"],
  "private_endpoint_cidrs": ["10.40.0.0/16"]
}}

The token is read from token_file at each observation and needs only list and read rights on the metadata paths. The source observes KV v2 secret metadata (path, current version, created and deleted times; the data path is never requested), the certificates a PKI mount issued (issuer and serial, key algorithm, validity, subject, revocation standing) and Transit key metadata. List only the mounts whose universe the paired authority also observes: a round compares whatever each authority asserts, so a KV-only Vault authority paired with the certificate inventory witnesses presence divergence by construction. Plain http:// addresses and loopback dialing need "allow_insecure_loopback": true; private_endpoint_cidrs widens the SSRF-safe dialer to operator-owned ranges. Any authority id may carry a vault source (vault-prod, vault-dr are separate authorities).

A KMIP server joins a round the same way, with a kmip source:

{"authority_id": "kmip-hsm", "source": {
  "kind": "kmip",
  "address": "kmip.internal:5696",
  "server_name": "kmip.internal",
  "ca_file": "/run/secrets/kmip/ca.pem",
  "client_cert_file": "/run/secrets/kmip/observer.pem",
  "client_key_file": "/run/secrets/kmip/observer-key.pem",
  "max_objects": 10000
}}

The source speaks KMIP 1.4 over TLS with that client certificate and trusts only ca_file; server_name defaults to the address host. Each observation opens one connection, lists the objects the server shows that client with Locate (paged with Maximum Items and Offset Items), reads each object's type, algorithm, length, state and dates with Get Attributes, and reads a certificate's issuer and serial from its value. Keys and secret objects are never fetched, so give the certificate an account that can only locate objects and read their attributes. An observation that cannot be complete fails rather than report part of the server: more objects than max_objects (default 10000), or a server that does not page with Offset Items. Templates and other object types that have no reconciliation record are skipped. The TLS floor is 1.3; "allow_tls12": true accepts a server whose TLS stops at 1.2, with AEAD suites only. allow_insecure_loopback permits a loopback address (TLS is still used), and private_endpoint_cidrs widens the SSRF-safe dialer as it does for Vault. An authority id with no source block, such as cloud-kms, fails observation closed.

An external CA's issuance ledger joins a round as its own authority. Pair it with an inventory authority: the certificate inventory scoped to the same CAs. Both read only the CAs whose certificates are in issuer_certificates_file (a PEM bundle) and only certificates unexpired at the observation, and both assert each certificate's issuer and serial, read from its DER, and whether it is revoked:

{"tenant_id": "YOUR_TENANT_ID", "cadence": "1h", "authorities": [
  {"authority_id": "inventory-ejbca", "source": {
    "kind": "inventory",
    "issuer_certificates_file": "/etc/trstctl/ejbca-issuing-ca.pem"
  }},
  {"authority_id": "ejbca-prod", "source": {
    "kind": "ejbca",
    "address": "https://ejbca.internal:8443",
    "server_name": "ejbca.internal",
    "ca_file": "/run/secrets/ejbca/server-ca.pem",
    "client_cert_file": "/run/secrets/ejbca/observer.pem",
    "client_key_file": "/run/secrets/ejbca/observer-key.pem",
    "issuer_certificates_file": "/etc/trstctl/ejbca-issuing-ca.pem",
    "ca_names": ["IssuingCA"],
    "private_endpoint_cidrs": ["10.40.0.0/16"]
  }}
]}

An ejbca source uses EJBCA's REST v2 certificate search: enable the REST Certificate Management V2 protocol and put the client certificate in an EJBCA role that may view certificates. address is EJBCA's base URL; the source adds /ejbca/ejbca-rest-api unless the URL already has a path. ca_names narrows the search to those EJBCA CA names. The search covers active and revoked certificates issued before the observation and expiring after it; on hold counts as revoked. The source reads the count first, then pages in expiry order (page_size, default 500; EJBCA allows up to 1000 once its Maximum Query Count is raised). If the pages do not add up to the count, or there are more than max_objects certificates (default 20000), the observation fails and the next round tries again; a partial ledger never reads as drift. The TLS floor is 1.3 unless "allow_tls12": true.

A stepca source reads step-ca's PostgreSQL database, because step-ca has no API that lists what it issued:

{"authority_id": "stepca-prod", "source": {
  "kind": "stepca",
  "dsn_file": "/run/secrets/stepca/observer.dsn",
  "issuer_certificates_file": "/etc/trstctl/stepca-intermediate.pem",
  "private_endpoint_cidrs": ["10.40.0.0/16"]
}}

dsn_file holds a PostgreSQL connection string. It is read at each observation and is not kept. Use a role that has SELECT on step-ca's x509_certs and revoked_x509_certs tables only. The source reads both tables in one read-only, repeatable-read transaction, and fails past max_objects rows (default 20000). step-ca's file backends (Badger, BoltDB) and its MySQL backend are not read. For both kinds, allow_insecure_loopback permits a loopback address, and private_endpoint_cidrs widens the SSRF-safe dialer as it does for Vault. The store-backed ids trstctl-self and trstctl-ca take no source block.

Workload-held PCAS co-signers

An identity whose predecessor key lives in a workload agent (not in the isolated signer) is registered with predecessor_custody: "workload" and the agent key's public half, and the agent serves the co-sign RPC with trstctl-agent --workload-cosign-listen. The control plane reaches it through one entry per identity:

{
  "pcas": {
    "workload_cosigners": [{
      "tenant_id": "YOUR_TENANT_ID",
      "identity_id": "spiffe://example.org/payments/api",
      "address": "payments-api.internal:9447",
      "tls_cert_file": "/etc/trstctl/cosign-client.crt",
      "tls_key_file": "/etc/trstctl/cosign-client.key",
      "peer_ca_file": "/etc/trstctl/cosign-agent-ca.pem",
      "peer_pin_hex": "<hex SHA-256 of the agent listener's SPKI>",
      "server_name": "payments-api.internal"
    }]
  }
}

A host:port endpoint needs all five TLS fields: the same pinned mutual TLS the agent requires of its caller. "address": "unix:/run/trstctl/cosign.sock" reaches a co-located agent over its owner-only socket and needs none. The identity's first succession request runs the co-sign flow through this endpoint; after it the successor key is in the signer and later successions need no agent.

An empty schedule list compares nothing. Each schedule needs a tenant ID and at least two authority entries with nonempty IDs; use the two distinct authorities above. Missing, invalid or nonpositive duration values use defaults: one hour for cadence, five minutes for jitter and 24 hours for liveness. An authority's optional liveness value inherits the schedule value when omitted.

collecting indicates that this tenant has a configured schedule. It does not prove that collection succeeded. Check control-plane logs for xrec round failed when investigating a failed comparison. replay_watermark is the last event sequence consumed for this tenant; it is not the time of the last comparison. Zero open witnesses alone do not establish agreement or current authority health.

Notifications

Notification channels are off until an operator configures them. When enabled, lifecycle expiry alerts are written to the notification.* outbox first, then delivered by the registered channel workers. Tenant-scoped routing policy authoring and channel-test delivery are served from the console; channel secrets remain in operator-managed secret references or files.

A tenant can deliver to a private HTTPS receiver after the deployment operator adds a grant for that tenant. This grants network reachability; the receiver's certificate must still chain to a trusted CA. Tenants can save their endpoint and credential reference, but cannot create or widen the grant.

Set the grant in deployment configuration, then restart the control plane:

{
  "notifications": {
    "tenant_private_egress": [
      {
        "tenant_id": "11111111-1111-1111-1111-111111111111",
        "cidrs": ["10.40.0.9/32"],
        "reason": "Tenant alert receiver"
      }
    ]
  }
}

The default is no grant. Each tenant has at most one entry, with one or more exact private network CIDRs; prefer a /32 or /128 for one receiver. Startup rejects an invalid or empty tenant UUID, duplicate tenant entries, wildcards, host bits, ranges extending into public space, loopback, link-local and metadata ranges. Neither a grant nor a tenant channel enables HTTP. Removing a grant blocks the next delivery after restart, including an endpoint saved earlier. DNS answers are checked at each socket dial, redirects keep the same origin, and air-gap destination rules also apply. A delivery receipt proves receiver acceptance, not downstream incident handling.

The Alerts and delivery console lets an operator edit or delete saved tenant channels and routing policies from their cards. A channel update replaces the entire channel. The API hides configured endpoint URLs and credential references on readback, so the edit form requires the operator to re-enter those write-only values when present; leaving them blank must not silently erase a working destination. Deletion shows the exact channel or policy ID, explains the routing consequence, and requires its display name before the API call. Read the channel and policy lists again after a change; the console cache alone is not delivery proof. When editing a routing policy, the console carries through existing severity mappings that its form does not display and the saved digest timezone in both the preview and replacement request.

Renewal-failure alerts link to the exact identity and retain its owner name and contact at the failure. When a completed deployment or rollback receipt exists, the alert also captures that certificate's fingerprint, serial, expiry, receipt ID and receipt time. The console labels this as the last recorded deployment: it does not prove what a listener serves now. An issued but uninstalled successor cannot supply that deadline. Missing receipt or certificate metadata remains unknown, and older alerts are not backfilled from today's inventory.

Retries after a rolled-back database transaction retain the original alert snapshot. Automatic routing is resolved when delivery runs; an absent policy ID in the alert is not proof that no routing policy was used. Owner contact metadata does not change the configured channel's recipient list.

Native SMTP delivery bounds the entire connection and protocol exchange to ten seconds, including the greeting, STARTTLS, authentication, message acceptance and QUIT. A shorter caller deadline takes precedence; normal notification fanout gives each channel five seconds. Cancellation closes the connection. If the relay offers STARTTLS, its certificate must pass system-root and exact-hostname verification. An acknowledgment lost after message submission can cause a duplicate on retry; email delivery is at least once.

Email subjects name the alert condition and affected subject, capped at 160 Unicode characters. The full alert, including certificate, owner, deadline and diagnostic detail, remains in the plain-text UTF-8 body. Long or non-ASCII subjects use folded MIME encoded words; the body uses quoted-printable encoding.

Each message includes a UTC Date recording when that attempt was rendered and a Message-ID bound to the exact outbox command, payload, sender and recipients. Retries retain that ID across process restarts; separate commands with identical text receive different IDs. A stable ID helps receivers recognize duplicate copies but does not guarantee that they suppress them. Direct notification SDK calls without an outbox envelope use the full alert as their identity, so callers must provide distinct operation IDs to distinguish otherwise identical alerts.

PagerDuty and OpsGenie credentials are copied into locked, non-dumpable memory at startup and wiped on shutdown. Both integrations send deterministic vendor idempotency identifiers, require the vendor's exact acceptance receipt, and redact remote error bodies. *_ALLOW_INSECURE_HTTP exists only for a loopback development emulator; even when set, it cannot enable cleartext delivery to a non-loopback host.

The same private-egress policy applies to the Slack, Teams, SMS, and SIEM channels: *_ALLOW_PRIVATE_CIDRS names the exact private ranges that one channel may resolve into, entries are validated at config load (no wildcard, no host bits), and the channel then uses the same bounded SSRF-safe client as the incident channels, with the air-gap guard wrapped when air-gap mode is on. A channel with no policy keeps the public-HTTPS-only default. Tenant-authored channels never inherit any of these allowances; they remain public-HTTPS-only.

Code signing

The shipped code-signing routes stay fail-closed until code_signing.enabled is true. Tenant-to-key and tenant-to-OIDC associations are structured JSON only: putting these parallel lists in environment variables increase the risk of attaching one tenant's trust to another tenant. A minimal key-backed plus GitHub Actions keyless configuration looks like this:

{
  "code_signing": {
    "enabled": true,
    "keys": [{
      "tenant_id": "tenant-acme",
      "id": "release-key",
      "handle": "acme-release-code-signing",
      "algorithm": "ecdsa-p256",
      "create_if_missing": true
    }],
    "github_oidc_tenants": [{
      "tenant_id": "tenant-acme",
      "issuer": "https://token.actions.githubusercontent.com",
      "audience": "sigstore",
      "jwks_file": "/run/trstctl/github-actions-jwks.json",
      "allowed_owners": ["acme"]
    }],
    "ephemeral_algorithm": "ecdsa-p256",
    "rekor": {
      "endpoint": "https://rekor.example.com/api/v1/log/entries",
      "timeout": "10s",
      "log_public_key_file": "/run/trstctl/rekor-log-public-key.pem"
    }
  }
}

create_if_missing provisions a code-sign-purpose-constrained handle inside the separate signer process; the control plane never receives the private key. Keyless requests also create and destroy their one-use key inside that signer. Pin the Rekor log public key from the Rekor operator's authenticated distribution channel. Startup fails if this trust anchor is absent or malformed, and every outbox delivery verifies the returned signed-entry timestamp against it before acknowledging the row.

Code signing uses the shared live policy controls rather than a decorative, code-only allow list. With ca.policy.enabled=true, OPA receives action code_sign, the authenticated actor, the configured key ID (or keyless:<verified-san>), and attrs.digest_sha256; evaluation errors and policy-pool saturation fail closed. With ca.policy.require_approval=true, a denied response returns approval_required:codesign:<sha256>. A distinct certs:issue approver posts {"action":"sign"} to /api/v1/identities/{that-resource}/approvals, after which the requester retries the same signing tuple with a new Idempotency-Key. The resource hash binds tenant, authenticated requester, key/keyless identity, and exact digest.

TRSTCTL_CODE_SIGNING_REKOR_ALLOW_INSECURE_HTTP is accepted only for a loopback development emulator. It cannot relax transport security for a remote Rekor service.

Telemetry

Telemetry is off by default and never sends anything unless you opt in. When enabled, it sends only coarse, anonymized, non-PII data to a collector you name. The project operates no public telemetry collector, so there is no default endpoint: enabling telemetry without TRSTCTL_TELEMETRY_ENDPOINT is a configuration error and trstctl refuses to start.

See Telemetry for exactly what is and is not collected.

Air-gap and outbound egress

Air-gap mode arms the no-phone-home outbound HTTP(S) guard. It is off by default; when enabled, public destinations are blocked unless you explicitly allow a host or CIDR. It is intended for disconnected networks and is paired with the Helm values-airgap.yaml overlay described in Air-gapped install.

When TRSTCTL_AIRGAP_ENABLED=true, trstctl rejects TRSTCTL_TELEMETRY_ENABLED=true and TRSTCTL_AI_MODEL_MODE=cloud at startup. Local OTLP collectors and local AI runtimes are still allowed when their hosts or CIDRs are private or explicitly allowlisted.

Scheduled Splunk HEC and Sentinel audit feeds

Audit-feed schedules are tenant configuration, not process-global environment settings. Review the exact candidate first with POST /api/v1/audit/feeds/{id}/preview, trstctl-cli audit feeds preview, or the Audit console. Preview runs the same server validator as save but creates no event, feed row, idempotency record, outbox row, or network call. Save only after the review shows the expected collector host, non-secret credential reference, permission, durable writes, later external effect, proof, and recovery steps. Saving uses PUT /api/v1/audit/feeds/{id}, trstctl-cli audit feeds set, or the Audit console. Each request chooses splunk-hec or sentinel, an absolute endpoint URL, an interval of at least 60 seconds, and an exact maximum batch size from 1 through 500.

The request stores only an env:NAME pointer. Put that pointer in TRSTCTL_OUTBOUND_ENV_CREDENTIAL_REFS, then inject the named environment variable into the control-plane process. The secret value is read only by the delivery worker; it never enters the configuration event, projection, API response, log, or collector error receipt. Public HTTPS is the default. A private endpoint also requires the caller's private-egress permission plus exact private_egress_cidrs; the SSRF-safe client rejects addresses outside that tenant configuration.

The scheduler does no network I/O. It commits audit.feed.batch.queued, the exact sequence range and record IDs, the destination projection, and one deterministic outbox intent before a worker can call the collector. A retry or restart therefore uses the same batch ID and bytes. GET /api/v1/audit/feeds exposes the delivered cursor, persistent record-count lag, attempts, next retry, terminal error code, and collector request ID without exposing response bodies or credentials. Failed delivery retains the exact batch and delivered cursor. Automatic retry reuses the same durable batch ID and idempotency key; the cursor advances only after the collector accepts that exact batch. Disable the feed to stop scheduling new batches while retaining failure and delivery evidence.

ITSM and ServiceNow bindings

ServiceNow ticket creation is fail-closed until an operator pre-registers the destination URL and credential reference. Requests to /api/v1/itsm/servicenow/tickets and ServiceNow response-dispatch destinations must match that binding exactly; callers cannot send the ServiceNow token to an arbitrary instance_url or opt into a private endpoint unless the binding permits it.

JSON config supports the same policy as itsm.servicenow.bindings[] with instance_url, token_ref, relay_only, allow_private_endpoint, and private_egress_cidrs. Ticket and response-dispatch bindings use env:NAME and relay_only: false; private endpoint requests also require the caller to hold the dedicated egress:private permission.

Ownership CMDB reconciliation uses the same approved binding but a different credential path: its token_ref must be a secret:// reference redeemable by the enrolled network relay, and its schedule sets execution: "relay" and allow_private_endpoint: false. For this path, register the exact URL and secret:// reference with relay_only: true and allow_private_endpoint: false in itsm.servicenow.bindings[] before configuring the tenant schedule through the Ownership console, API or CLI. That binding cannot authorize a control-plane ticket or response dispatch. The control plane never polls the CMDB directly. The legacy env: default above is for ticket dispatch and cannot activate CMDB reconciliation.

OpenTelemetry export

OTLP export is off by default and sends data only to the collector endpoint you configure. It is separate from product telemetry: use it to feed your own OpenTelemetry Collector, Splunk, Datadog, or SIEM pipeline with served HTTP traces and audit-event log records.

In air-gap mode, point OTLP at an operator-owned collector on a private address or add the collector host to TRSTCTL_AIRGAP_ALLOW_HOSTS; public collector SaaS endpoints are blocked by the egress guard.

Audit

The audit trail is a projection of the event log; these settings govern its evidence export and retention policy. Plain signed export stays core; timestamp anchoring, retention and compliance packaging require Enterprise audit compliance (also inherited by Provider). Without that feature, retention settings do not start a worker or advance existing checkpoints. See Audit trail & compliance for the trust model and what trstctl enables vs. what you must operate.

The audit query (/api/v1/audit/events) and signed export (/api/v1/audit/export) endpoints are wired into the serving binary, so they return real data — not an error — out of the box. The evidence private key lives only in the signer's sealed key store; back up that store and the KEK, and distribute the public JWKS to auditors out of band.

Privacy Retention

Non-audit personal data is retained by class, then pseudonymized after the operational need ends. This is separate from audit retention: the immutable event trail remains the source of truth, while tenant read surfaces stop carrying stale names, emails, subjects, SANs, comments, profile authors, approval actors, and free-form evidence. The worker emits privacy.retention.enforced and projects the anonymization from that event, so rebuilds replay the same result.

Subject erasure also pseudonymizes matching raw subject bytes in the hot JetStream event store. The control plane records privacy.subject.erased, builds a sequence-preserving replacement generation with the erased subject changed to its tenant-bound erased:<subject_ref> placeholder, and signs the old/new continuity evidence before switching authority. The old generation remains authoritative if the operation stops before that switch; only after the signed replacement is active are its superseded subject bytes securely scrubbed. Replay, projection tailing, retention, and backups share a history barrier with that cutover. This keeps future hot-log replay, audit queries, and full backups taken after the erasure from carrying the raw subject without ever purging and rebuilding the sole authoritative stream. Backups and signed audit archives created before the erasure are not rewritten automatically. Record the outcome of your legal-hold, WORM, backup-deletion, or cryptographic-shredding procedure with POST /api/v1/privacy/archive-erasure-attestations; inspect the tenant evidence ledger with GET /api/v1/privacy/archive-erasure-attestations or trstctl privacy archives attest/list. The attestation event stores subject_ref and redacted evidence refs, not the raw subject.

Operators can trigger and inspect the same served path with POST /api/v1/privacy/retention-runs, GET /api/v1/privacy/retention-runs, or the matching trstctl privacy retention run/list CLI commands.

Provider workforce identity and customer access

The Provider plane is a different privilege domain from tenant login. OIDC or SAML proves which Provider employee is calling. SCIM is the live joiner/leaver directory. A customer delegation then narrows that active employee to one customer and one operation. Think of the chain as three locks: a request opens only when identity, employment state, and exact customer authority all agree.

OIDC verifies an offline-pinned JWKS plus issuer, audience, time, mapped role, and MFA. SAML serves login, ACS, and metadata at /provider/v1/auth/saml/{login,acs,metadata} and verifies signed assertions inside internal/crypto. Its HttpOnly cookie is Provider-only and mutations use double-submit CSRF. When Provider SCIM is enabled, both methods resolve the SCIM-projected operator on every request, so deprovisioning immediately refuses an otherwise valid token/session and revokes the operator's live delegations. Without Provider SCIM, a signed role and MFA claim may bootstrap a subject absent from the operator directory. An existing inactive subject or a directory lookup error still refuses the request; an outage is never treated as absence.

Multi-token JSON configuration and a complete SAML block:

{
  "provider": {
    "oidc": {
      "issuer": "https://idp.provider.example",
      "audience": "trstctl-provider",
      "jwks_file": "/etc/trstctl/provider-idp.jwks",
      "role_claim": "groups",
      "admin_values": ["provider-admin"],
      "operator_values": ["provider-operator"],
      "login": {
        "authorization_endpoint": "https://idp.provider.example/protocol/openid-connect/auth",
        "token_endpoint": "https://idp.provider.example/protocol/openid-connect/token",
        "client_id": "trstctl-provider-console",
        "client_secret_file": "/etc/trstctl/provider-console-client.secret",
        "redirect_url": "https://trstctl.provider.example/provider/v1/auth/oidc/callback",
        "session_secret_file": "/var/lib/trstctl/provider-oidc-session.secret"
      }
    },
    "saml": {
      "enabled": true,
      "entity_id": "https://trstctl.provider.example/provider/v1/auth/saml/metadata",
      "metadata_url": "https://trstctl.provider.example/provider/v1/auth/saml/metadata",
      "acs_url": "https://trstctl.provider.example/provider/v1/auth/saml/acs",
      "idp_metadata_file": "/etc/trstctl/provider-idp-metadata.xml",
      "response_binding": "post",
      "sp_key_file": "/var/lib/trstctl/provider-saml-sp-encryption.sealed",
      "session_secret_file": "/var/lib/trstctl/provider-saml-session.secret",
      "role_attribute": "groups",
      "admin_values": ["provider-admin"],
      "operator_values": ["provider-operator"],
      "mfa_attribute": "amr",
      "mfa_values": ["mfa"]
    },
    "scim": {
      "enabled": true,
      "tokens": [
        {"name": "entra", "token_file": "/etc/trstctl/provider-scim-entra.token"}
      ]
    }
  }
}

provider.oidc.login turns on Provider console sign-in through the IdP: the console's "Sign in with your identity provider" runs the authorization code flow with state, nonce and PKCE S256 (GET /provider/v1/auth/oidc/login and /provider/v1/auth/oidc/callback). The endpoints are configured, never discovered at request time; the code is redeemed with the client secret from client_secret_file (client_secret_basic) and the ID token is verified against the pinned keys above, with client_id as its audience, the same role and MFA claims, and the nonce. The browser receives an HttpOnly, host-only, SameSite=Strict Provider session (session_ttl, default 12h; the secret in session_secret_file is created on first start) plus the double-submit CSRF cookie; no token reaches the browser or browser storage, and when this is set the console no longer offers the bearer-token box. Every request re-reads the SCIM directory row, so a leaver or a revoked role is refused at once, and sign-out revokes the session.

The operator console lists SCIM lifecycle and every current/historical delegation. Grant/revoke/role mutations require a Provider admin, current MFA, and Idempotency-Key; exact retries return the same event result.

Browser SSO

Browser sign-on is optional. Scoped API tokens still work when browser sign-on is off. OIDC is available in the Free core. Tenant SAML, LDAP and SCIM require the Enterprise SSO feature, included in Enterprise and inherited by Provider. Enabling any of those three methods without the licensed implementation refuses startup with SAML, LDAP and SCIM login require an Enterprise licence. A core-only binary cannot load those implementations, even with a license. Provider operator-plane authentication uses its separate provider.* settings. When it is on, each verified OIDC, SAML, or LDAP / Active Directory user must map to exactly one trstctl tenant by subject, tenant claim, directory group, or an explicit single-tenant fallback. Missing mappings fail the login closed instead of silently dropping a user into the wrong tenant.

An OIDC login started from a protected console page returns to that page after sign-in, including its query filters and fragment. This also works when opening a bookmarked page after the browser session has expired.

The console passes a local return_to path to /auth/login. The server validates it and binds it to the same one-use pre-login record as state, nonce, and PKCE; a callback parameter cannot replace it. A tenant-mapping recovery page retains that validated destination for the next sign-in attempt.

The path is limited to 4096 bytes. External URLs, protocol-relative URLs, backslashes in the path, control characters, and login or authentication routes are ignored. With an absent or invalid destination, OIDC uses its configured auth.oidc.login_redirect, or / when unset. The LDAP console form resumes its validated local destination after confirming the new session.

A console-started SAML login passes the same local return_to input to /auth/saml/login. The server binds its raw value to the random state and SAML request ID in a signed, ten-minute browser context. The destination is never sent to the identity provider, and ACS query or form parameters cannot replace it. Two HttpOnly cookies carry that context, each limited to 3500 value bytes so the serialized cookie fits browser limits. The 4096-byte input limit applies before URL encoding, including for Unicode paths. Instances using the same browser-session secret can validate the context without shared process memory.

Absent or invalid initial destinations, older in-flight logins without either context cookie, and IdP-initiated logins use the configured post-login default. A partial, duplicate, tampered or expired context in an SP-initiated callback returns HTTP 400; restart sign-in. Successful login clears both context cookies. This destination binding supplements the existing SAML assertion verification; it is not an assertion-replay cache or an authenticated session.

API tokens

A token minted through the API for a different subject is delegated: it works with its subject's scopes, but approval and denial routes refuse it (see Policy & governance).

Session roles

A browser session starts with the roles of the tenant mapping that matched (tenant_mappings[].roles), or with default_roles when that mapping names none. When the signed-in person has a tenant member record that names roles, set in Operations → People and roles, through PUT /api/v1/access/members/{subject} or by SCIM, those roles replace the sign-in roles on every request. A member record that names no roles keeps the sign-in roles. Offboarding immediately invalidates that subject's browser sessions and API tokens, including while the event projector is recovering; subsequent requests return 401. An explicit rehire creates a new member event and requires new sessions and tokens. Old credentials stay revoked. Role changes commit a subject-bound admission fence before their event is appended. While that exact event is pending projection, existing browser sessions keep their identity but have no grants; an access check returns 403. A failed member/fence read returns an error rather than restoring roles from the old cookie. Recovery replays the exact event; a later offboard supersedes one that never appended. Subject erasure also revokes existing browser sessions and leaves an offboarded pseudonymous member tombstone, even when the member update had not projected yet. Recovery compares source event sequence with the erasure event: an older pending role change cannot reactivate the erased subject. A later explicit rehire requires a new member event and new browser login; the old session remains revoked. GET /auth/me returns the roles the session holds. The evaluation compose files set default_roles to admin so the first operator can sign in; in production, leave it unset or use a read-only role and grant more through member records or mappings.

Custom roles

A tenant can define its own roles as named subsets of the permissions the built-in roles grant: PUT /api/v1/access/roles/{name} with permissions and an optional description (trstctl-cli access roles define NAME -f role.json, or People and roles → Define custom role). Defining needs access:write and every permission the role grants, so nobody can define a role broader than themselves; assigning one to a member also needs access:role.assign and every permission it grants. Names are 2 to 63 lowercase letters, digits or hyphens; built-in names are taken, and the wildcard and the agent channel's machine permissions are never grantable. A role resolves on every request, so redefining it changes what its holders can do at once on every replica. DELETE /api/v1/access/roles/{name} (access roles delete) deletes a role no member holds; a held role is refused with 409. GET /api/v1/access/roles lists the built-in roles and the tenant's custom roles (custom: true). A directory group grants a custom role through the SCIM token's group_roles, below.

SCIM provisioning

SCIM 2.0 provisioning requires Enterprise SSO and is optional and separate from browser sign-on. Enable it when your identity provider should push users and groups into trstctl instead of an operator maintaining role membership by hand. The served endpoint is /scim/v2: IdPs use POST /scim/v2/Users, PATCH /scim/v2/Users/{id}, POST /scim/v2/Groups, and PATCH /scim/v2/Groups/{id}.

Each SCIM bearer token is bound to exactly one tenant in config. trstctl reads the raw token from auth.scim.tokens[].token_file, hashes it at startup, wipes the raw bytes, and keeps only the hash. When the file changes (its size, modification time, or the file itself), trstctl reads it again within two seconds, so rotating a token is rewriting its file, and the old token stops working then; no restart is needed. A rewrite it cannot use (an empty or unreadable file, or a token another entry already uses) is logged and the last good token stays in effect. Adding a token entry is a configuration change and needs a restart. The token selects the tenant for every SCIM request; tenant ids in SCIM payloads are ignored. A group named for a built-in role grants that role, so an IdP group with display name viewer grants the built-in viewer role to its members. Any other group needs an entry in auth.scim.tokens[].group_roles, mapping the group's display name to a built-in role or a custom role; a group with no entry is refused and grants nothing, and a group mapped to a custom role the tenant has not defined is refused with that role's name. Each role takes at most one group, and a mapped role is listed under its group's name.

Choose the login-subject mapping before provisioning. By default, userName must equal the exact authenticated principal (for OIDC, its sub claim), not merely its email address. If your IdP sends a readable userName and an opaque login subject separately, set auth.scim.tokens[].subject_attribute: externalId and configure that IdP's SCIM attribute mapping to put the exact login subject in externalId. SCIM defines externalId as a client-issued identifier; it is not automatically an OIDC subject. trstctl never infers this binding from an email address. Only a trusted deployment configuration selects the attribute. The console's People and roles → SSO groups and role bindings panel shows the current session's exact sign-in subject. Use it to check the IdP's SCIM attribute mapping before the first provision. For LDAP, that subject can be the full directory DN rather than the short login name.

Both userName and externalId are retained and returned. The selected attribute becomes the stable SCIM resource id and tenant-member subject. It cannot change through PUT or PATCH. With the explicit externalId mapping, the readable userName can change without moving permissions to a new login. User names are unique ignoring case within the tenant, including inactive resources. Provision the user first, then send the returned resource id as each group's member value. An inactive user must be explicitly activated before a group grant; a delayed group removal never activates an offboarded user. If event projection is interrupted during deprovisioning, the subject remains denied and the exact first offboard command resumes after restart. A group delta cannot reactivate the subject; send an explicit active user update after recovery.

Changing a token's mapping does not relink existing accounts. Legacy members without a stored binding are hidden from the externalId-mapped user list until explicitly provisioned with the correct subject. Review existing email-named members, create the correctly bound resource, reapply intended groups, and retire mistaken memberships. Existing bound resources reject a different mapping. Verify a real existing browser session and API token lose access after active:false or DELETE; a successful provisioning response alone is insufficient evidence. Use a new idempotency key for each deliberate lifecycle change. Retry the same operation with its original key and body.

Directory writeback (A73): a token may also name the directory's own SCIM 2.0 endpoint with writeback_endpoint and writeback_token_file (both or neither; https, or http only to a loopback host). A role change or offboard an operator makes in trstctl to a member that token provisioned is then written to the directory's User (active and roles), read back, and recorded as scim.writeback.recorded. The outbound bearer is loaded at startup into locked memory and read only at startup: rotate it by writing the file and restarting.

The environment variables replace the token list as one complete single-token configuration. A mapping-only environment override fails validation when enabled; use the per-token config field for a multi-token deployment. Example:

Example multi-token SCIM config:

auth:
  scim:
    enabled: true
    tokens:
      - name: okta-payments
        tenant_id: 22222222-2222-2222-2222-222222222222
        token_file: /etc/trstctl/scim/okta-payments.token
        subject_attribute: externalId # IdP mapping must supply the exact login subject
        group_roles: # directory group display name -> built-in or tenant custom role
          payments-ops: payments-operator
          Platform Admins: admin
      - name: entra-platform
        tenant_id: 33333333-3333-3333-3333-333333333333
        token_file: /etc/trstctl/scim/entra-platform.token
        writeback_endpoint: https://directory.example.internal/scim/v2 # the directory's own SCIM server
        writeback_token_file: /etc/trstctl/scim/entra-platform.writeback.token

Example SAML config:

Use HTTPS for browser SAML login. The identity provider posts its response back to /auth/saml/acs; the two ten-minute correlation cookies and the two return context cookies (when a valid destination was requested) use SameSite=None; Secure; HttpOnly so the browser includes them on that cross-site POST. The ACS still validates the state, request ID and signed assertion. OIDC correlation cookies remain SameSite=Lax; authenticated session and CSRF cookies remain SameSite=Strict. Plaintext development retains Lax correlation cookies and does not support cross-site SAML POST login.

auth:
  saml:
    enabled: true
    entity_id: https://trstctl.example.com/auth/saml/metadata
    metadata_url: https://trstctl.example.com/auth/saml/metadata
    acs_url: https://trstctl.example.com/auth/saml/acs
    idp_metadata_file: /etc/trstctl/idp-metadata.xml
    # post (default) or artifact: with artifact the IdP's metadata must declare a
    # SOAP ArtifactResolutionService, which the SP calls over the back-channel.
    response_binding: post
    # Optional: sealed RSA identity advertised for assertion encryption; created on
    # first start (needs secrets.kek_file). Omit to accept plaintext assertions.
    sp_key_file: /var/lib/trstctl/saml-sp-encryption.sealed
    # Optional single logout: the IdP metadata must declare a SingleLogoutService
    # (HTTP-Redirect); register slo_url with the IdP as this SP's logout service.
    single_logout: true
    slo_url: https://trstctl.example.com/auth/saml/slo
    session_secret_file: /var/lib/trstctl/saml-session.secret
    tenant_claim: tenant
    groups_claim: groups
    tenant_mappings:
      - subject: alice@example.com
        tenant_id: 11111111-1111-1111-1111-111111111111
        roles: [admin]

Example Kerberos / GSSAPI config (the browser presents its domain ticket; the keytab holds the HTTP service principal's keys exported from the KDC):

auth:
  kerberos:
    enabled: true
    keytab_file: /etc/trstctl/http.keytab
    service_principal: HTTP/trstctl.example.com
    realm: EXAMPLE.COM
    # true only with an Active Directory KDC: map PAC group SIDs through tenant_mappings[].group
    pac_groups: false
    session_secret_file: /var/lib/trstctl/kerberos-session.secret
    tenant_mappings:
      - subject: alice@EXAMPLE.COM
        tenant_id: 11111111-1111-1111-1111-111111111111
        roles: [admin]

Example NTLM config (the handshake is relayed to Samba's ntlm_auth on a domain-joined host; the helper, not trstctl, validates the NTLM response):

auth:
  ntlm:
    enabled: true
    helper_command: /usr/bin/ntlm_auth        # or helper_address: tcp://ntlm-helper.internal:3101
    domain: CORP
    session_secret_file: /var/lib/trstctl/ntlm-session.secret
    tenant_mappings:
      - subject: alice@CORP
        tenant_id: 11111111-1111-1111-1111-111111111111
        roles: [admin]

Example LDAP / Active Directory config (password_change serves the self-service password change; mode exop for OpenLDAP-style directories, active_directory for AD over ldaps://):

auth:
  ldap:
    enabled: true
    password_change:
      enabled: true
      mode: active_directory
    nested_groups:
      enabled: true
      mode: in_chain          # recursive (default) walks parents up to max_depth
    writeback:                # role changes made in trstctl flow back to these groups
      enabled: true
      base_dn: dc=example,dc=com
      role_groups:
        admin: cn=trstctl-admins,ou=groups,dc=example,dc=com
        viewer: cn=trstctl-viewers,ou=groups,dc=example,dc=com
    url: ldaps://ad.example.com:636
    bind_dn: cn=trstctl-reader,ou=service-accounts,dc=example,dc=com
    bind_password_file: /etc/trstctl/ldap-bind.secret
    user_search_base_dn: ou=people,dc=example,dc=com
    user_filter: "(sAMAccountName={username})"
    group_search_base_dn: ou=groups,dc=example,dc=com
    group_filter: "(member={user_dn})"
    group_name_attribute: cn
    email_attribute: mail
    session_secret_file: /var/lib/trstctl/ldap-session.secret
    tenant_mappings:
      - group: payments-trstctl-admins
        tenant_id: 22222222-2222-2222-2222-222222222222
        roles: [admin]

ABAC deny overlay

ABAC is an optional deny-only overlay on top of RBAC. RBAC must grant the permission first; the ABAC Rego module can then block the request using request attributes, identity tags, operator-provided environment state, and time. This is useful for rules such as "prod certs may issue only during change windows" or "break-glass actions may run only from the platform project."

The module must declare package trstctl.abac, define boolean deny, and may define string reason. Bad Rego fails startup. Runtime evaluation errors deny the request, and policy-worker saturation returns 503 instead of allowing.

In a config file, the keys are auth.abac.enabled, auth.abac.module, and auth.abac.environment.

Example ABAC config:

auth:
  abac:
    enabled: true
    environment:
      change_window: "false"
    module: |
      package trstctl.abac
      default deny := false
      default reason := ""

      deny if {
        input.permission == "certs:issue"
        input.resource.env == "prod"
        input.env.change_window != "true"
      }

      reason := "prod certificates may issue only during a change window" if {
        deny
      }

Break-glass lifecycle and reconciliation

Break-glass emergency issuance has two served paths. POST /api/v1/breakglass/issue performs online m-of-n emergency issuance when a signer-backed break-glass issuer is configured. First open the exact CSR/reason/TTL ceremony at POST /api/v1/breakglass/issue-ceremonies; then distinct configured operators approve that ceremony through the shared authenticated CA-approval route. The execution request carries ceremony_id and no approver names. The server derives quorum only from immutable ca.ceremony.approved event actors, consumes the ceremony once, returns a self-verifying bundle, and records breakglass.issued before responding. The same signer-backed lifecycle exposes purpose-bound rotation and target-CA cross-sign ceremony/execution pairs under /api/v1/breakglass/*-ceremonies, /api/v1/breakglass/rotate, and /api/v1/breakglass/cross-sign. POST /api/v1/breakglass/reconcile handles recovery after an offline ceremony: it accepts signed offline bundles, verifies them against trust anchors pinned in process config, and records verified facts as breakglass.issued events in the hash-chained audit log. Requests cannot supply their own verifier keys. For a genuinely disconnected v2 ceremony, preprovision the isolated signer and its two-custodian public kit while the control plane is healthy. The signer can issue with the API and PostgreSQL stopped. The deployment pins the exact kit SHA-256 in config; recovery checks the signed custodian evidence and stores the public proof in the v3 audit event. See the disconnected break-glass runbook. The server verifies every signature in a batch before recording any bundle. It uses the tenant and signed request_id as a durable event identity, so retrying the same bundle with a new Idempotency-Key, after a restart, or on another replica adds no second issuance event. The response's reconciled count is the number of newly recorded bundles; an exact retry returns 0. A different valid bundle with an already recorded request ID returns 409. If an event-log failure interrupts a batch after a prefix was recorded, resubmit the original batch: the recorded prefix is skipped and the remaining bundles are processed.

breakglass.enabled pins reconciliation verifier material. The files may be DER, or PEM with CERTIFICATE and PUBLIC KEY blocks, and startup fails closed when either is missing or unreadable. breakglass.online_enabled additionally requires one tenant, an already-persisted dual-control signer handle whose public key matches both files, a distinct operator roster, and a threshold of at least two. Reconciliation- only deployments do not need the online fields.

Startup also checks the keys themselves. The configured public key and the offline kit's public key must pass the structural weak-key checks (small factors, close primes, ROCA, the operator blocklist, a small EC scalar, a bad RSA exponent); offline keys stay with the recovery side, so that is their whole check. With online_enabled, the signer handle that signs after any recorded rotation must also prove it holds its key by signing a fresh challenge, which goes through the dual-control authorizer like any other signature. A failure stops startup with startup binding <name>: weak-key check failed: ... or ... possession proof failed: ..., naming the binding (breakglass.public_key_file, breakglass.offline_kit_file, breakglass.signer_handle "<handle>", or the rotated handle).

Example break-glass reconciliation config:

breakglass:
  enabled: true
  ca_cert_file: /etc/trstctl/breakglass-ca.pem
  public_key_file: /etc/trstctl/breakglass-public-key.pem
  offline_kit_file: /etc/trstctl/offline-kit.json
  offline_kit_sha256: <sha256-printed-by-offline-init>
  tenant_id: 10000000-0000-4000-8000-000000000001
  deployment_id: my-control-plane

Online lifecycle adds the custody and roster fields:

breakglass:
  enabled: true
  online_enabled: true
  ca_cert_file: /etc/trstctl/breakglass-ca.pem
  public_key_file: /etc/trstctl/breakglass-public-key.pem
  tenant_id: 10000000-0000-4000-8000-000000000001
  signer_handle: recovery-breakglass-ca
  operators: [recovery-operator-a, recovery-operator-b, recovery-operator-c]
  threshold: 2

To create that handle without a test-only or library-only path, leave online_enabled=false, use trstctl-cli ca ceremonies start plus distinct ca ceremonies approve calls, and finish with ca authorities create-root. Save the returned certificate_pem and signer_handle; derive the public-key file from the certificate (openssl x509 -pubkey -noout), set the online fields above, and restart. Startup proves the certificate, derived public key, and persisted signer handle all name the same key before it exposes any online break-glass route.

Secrets (credentials at rest)

Upstream CA and connector credentials — API keys, passwords, client secrets — are stored encrypted at rest using envelope encryption (R3.1): a fresh random data-encryption key (DEK) encrypts each credential with AES-256-GCM, and the key-encryption key (KEK) wraps the DEK. Only ciphertext is ever persisted; the plaintext never appears in the database, in config dumps, or in logs. The cryptography lives behind the platform's single crypto boundary.

Every default-binary idempotency result is outer-sealed through the same tenant-aware crypto resolver before PostgreSQL retains it. A legacy tenant uses the deployment KEK explicitly. An opted-in tenant names exactly one separately provisioned wrapper and never trial-decrypts or falls back to the deployment KEK. The local wrapper file is opened only inside a tenant-scoped shared database fence; its domain key lives in locked memory for that callback and is then wiped. Before the HTTP mutation surface becomes ready, startup drains historical raw-v0 and sealed-dynamic-lease-v1 cache rows in key-ordered, RLS-scoped batches into the authenticated sealed-row-v1 envelope. Each replacement is a compare-and-swap over tenant, key, binding, codec, and prior bytes. A crash therefore resumes safely, while a changed row or unavailable tenant wrapper fails readiness without logging or embedding the result bytes in the error. The runtime reader rejects both legacy codecs after this drain. When the fleet-ready assertion is set, startup also locks the table, proves every completed row is a CSL sealed container, changes the database default, and validates a permanent sealed-only constraint before readiness. This is a one-way compatibility decision: drain or stop all old writers first.

Schema-v1 scheduled-rotation terminal events from older releases could retain an arbitrary provider error string. A current binary scans the complete retained history before projection catch-up, audit search/retention, or backup export. If unsafe bytes exist and the fleet assertion above is false, startup fails with one fixed sanitation-required error. After the old fleet is stopped and the assertion is true, startup switches to a signed replacement generation that changes only the JSON error token to the status-specific closed vocabulary. Every other stored envelope byte and sequence stays fixed; normal append/import then rejects v1 scheduler runs permanently. The operation does not and cannot rewrite backup, export, or WORM/archive copies made before the switch.

{
  "secrets": {
    "kek_file": "/run/secrets/deployment-kek",
    "tenant_seal_local_wrappers": [
      {"id": "operator-a", "file": "/run/custody/operator-a-wrapper"},
      {"id": "operator-b", "file": "/run/custody/operator-b-wrapper"}
    ]
  }
}

Wrapper files are operator authority, not ordinary application data. Provision them separately with restrictive ownership/mode before configuring a tenant. Missing, corrupt, or mismatched custody fails that tenant closed while other tenants continue. This local mode makes no outbound call and works air-gapped.

Tenant domain keys under an HSM

A tenant can instead keep its domain KEK under an HSM key the isolated signer holds (wrapper_kind: "pkcs11"). The signer creates the tenant's domain KEK, keeps it wrapped by an AES-256 key the operator provisions on the HSM and reaches over PKCS#11, and wraps and unwraps the tenant's data keys itself. The database stores only the wrapped KEK; the control plane never holds the KEK and asks the signer for every data-key wrap and unwrap, and the HSM key never leaves the HSM. This mode needs the cgo signer built by deploy/docker/Dockerfile.signer-hsm and a license with the BYOK feature.

1. Provision a sensitive, non-extractable AES-256 key on the token, for example with OpenSC (--login prompts for the user PIN):

   pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so --token-label trstctl-prod --login --keygen --key-type AES:32 --label trstctl-tenant-kek-a --sensitive
   

   The signer refuses a key that is extractable, not sensitive, not AES-256, or not allowed to both encrypt and decrypt.

2. Give the signer its tenant KEK file with --tenant-kek-config (a child signer receives signer.tenant_kek_config, or TRSTCTL_SIGNER_TENANT_KEK_CONFIG). The PIN is a file reference only:

   {
     "pkcs11": {
       "module_path": "/usr/lib/softhsm/libsofthsm2.so",
       "token_label": "trstctl-prod",
       "user_pin_file": "/etc/trstctl/pkcs11-user-pin"
     },
     "wrappers": [
       {"id": "hsm-a", "key_label": "trstctl-tenant-kek-a"}
     ]
   }
   

   At startup the signer seals and opens a probe under every listed key and refuses to start if one fails, so a missing or mislabeled key is found before any tenant depends on it.

3. List the same wrapper IDs on the control plane:

   {"secrets": {"tenant_seal_pkcs11_wrappers": [{"id": "hsm-a"}]}}
   

4. Migrate the tenant from the console's tenant custody panel, or with trstctl-cli platform tenant-key-domain migrate -f migrate.json where migrate.json is {"wrapper_kind":"pkcs11","wrapper_id":"hsm-a"}. Seal and unseal work as they do for a local wrapper.

A wrapper ID the control plane does not list, or one the signer does not hold, is refused as not configured. If the signer loses the HSM key or the token, the tenant fails closed as wrapper_unavailable; nothing falls back to the deployment KEK. The status reports remote_wrapper_state: "signer_hsm" and local_wrapper_zero_egress: false for such a tenant.

Limits: the data keys return to the control plane, which decrypts tenant data with them; every tenant-crypto callback for such a tenant makes signer calls, an HSM decrypt each; and the PKCS#11 binding hands the decrypted KEK back through a C buffer it frees without zeroing (the signer is non-dumpable and refuses ptrace). It is proven against SoftHSM; proof against a hardware HSM needs hardware. What comes next: one non-extractable HSM key per tenant, so the tenant's KEK is itself an HSM object and data-key wraps run inside the HSM.

Rotating a tenant domain key

A migrated tenant rotates its domain key from the console's tenant custody panel, or with trstctl-cli platform tenant-key-domain rotate (no body keeps the current wrapper; -f rotate.json with {"wrapper_kind":"pkcs11","wrapper_id":"hsm-b"} moves the key onto another configured wrapper, local or HSM). The rotation makes a new domain key, the next generation, then re-wraps every data key the tenant holds in PostgreSQL and in its event history from the old generation to the new one without decrypting a payload, and promotes the new generation. While it runs, new data is sealed under the new generation and both generations stay readable; a failure leaves the tenant rotating with the error shown, and a retry resumes with the same target wrapper. Keep both wrappers configured until the rotation completes. A seal cannot start during a rotation.

A rotation re-wraps data keys; it does not re-encrypt payloads. Backups, archives, and exports taken before the rotation still open with the previous generation through its wrapper, and that generation's wrapped key stays in the tenant's event history. To retire a generation everywhere, rotate onto another wrapper and then retire the old wrapper's key or file.

Rotating the deployment KEK

The deployment KEK (secrets.kek_file) protects every stored credential of a tenant that has not moved to its own key domain, and the deployment's sealed files. A rotation moves it to a new generation: a key created under an operator wrapper, either a local wrapper file (secrets.tenant_seal_local_wrappers) or an HSM key the isolated signer holds (secrets.tenant_seal_pkcs11_wrappers, set up as in Tenant domain keys under an HSM). Only the wrapped key is stored; for an HSM key the signer wraps and unwraps every data key, so stored credentials decrypt only through the signer and the HSM key.

1. Name the operator tenant: secrets.kek_operator_tenant_id (TRSTCTL_SECRETS_KEK_OPERATOR_TENANT_ID). Only its keys:write principals may rotate.
2. Configure the target wrapper on the control plane (and, for pkcs11, in the signer's tenant KEK file).
3. Rotate from the console's deployment key panel (Admin, System), or with trstctl-cli platform deployment-kek rotate -f rotate.json where rotate.json is {"wrapper_kind":"pkcs11","wrapper_id":"hsm-a"}. The request returns 202 once the next generation is recorded.

From that moment every new data key is wrapped under the next generation. After secrets.kek_rotation_settle (default two minutes), the elected leader re-wraps every stored data key in bounded steps: each tenant's stored ciphertext, the event history of every tenant that has not completed its own key-domain migration, and the deployment's sealed files (the SAML encryption identity, the SCEP/CMP RA identity, a transit keyring checkpoint, and the agent reachability verdict key). Payloads are never decrypted. A verify pass then reads every stored data key back and must find none under the previous generation before the next generation is promoted and the previous one retired; the key file then opens none of them. Requests are served throughout, and a new leader resumes a rotation from its recorded step. GET /api/v1/platform/deployment-kek (trstctl-cli platform deployment-kek status) reports the step, progress, and the counts re-wrapped and read back.

A rotation that stops (for example on a history record that needs an explicit migration) keeps writing new data under the next generation, so it resumes only with the same wrapper. A later rotation moves the deployment KEK onto another wrapper the same way.

The key file stays in use after a rotation: it is the key for request evidence MACs and the backup HMAC, and it seals the signer's key store unless Helm externalKMS is set. Backups, archives, and exports taken before a rotation still open with the previous generation (the key file for the first rotation), so keep it, and any retired wrapper, as long as such backups. The deployment's sealed files must be on storage every replica shares; a replica's private copy left under a retired generation fails to open.

Dynamic-secret providers and outbound sync targets use structured JSON/YAML because each entry is bound to exactly one tenant. Authority-bearing values are references, not inline strings: file:/absolute/path loads an operator-owned 0600 file, while secret://path opens that tenant's encrypted secret-store row for one outbox attempt. Both are copied into locked, non-dumpable memory and destroyed after the provider call. PostgreSQL, MySQL, and MongoDB admin connection files may end in shell-added CR/LF line endings; the provider strips only those terminal bytes before parsing the connection value and creating a workload credential.

{
  "secrets": {"enable_api": true},
  "secret_integrations": {
    "dynamic_providers": [{
      "tenant_id": "11111111-1111-4111-8111-111111111111",
      "id": "orders-postgres",
      "type": "postgresql",
      "admin_dsn_ref": "file:/run/secrets/orders-postgres-admin-dsn",
      "database": "orders",
      "schema": "public",
      "allowed_roles": ["reader"],
      "max_ttl": "1h",
      "username_prefix": "trstctl_orders",
      "allow_static_rotation": false
    }],
    "sync_targets": [{
      "tenant_id": "11111111-1111-4111-8111-111111111111",
      "id": "orders-github",
      "type": "github-actions",
      "endpoint": "https://api.github.com",
      "owner": "example",
      "repo": "orders",
      "token_ref": "secret://integrations/github-actions-token"
    }]
  }
}

Dynamic type values are postgresql, mysql, mongodb, aws-iam, gcp-iam, azure-entra, kubernetes, and redis. Sync type values are aws-secrets-manager, aws-parameter-store, gcp-secret-manager, azure-key-vault, github-actions, gitlab-ci, vercel, netlify, generic-ci-json, webhook, kubernetes-secrets, terraform-cloud-opentofu, and vault-kv-v2. Startup validates each provider's required native fields, role bindings, endpoint scheme, private-egress CIDRs, and credential-reference form before accepting traffic. An absent tenant target or provider fails its served mutation closed; it never falls back to another tenant or to a test registry.

Signed webhook, Parameter Store and Netlify targets

A webhook target POSTs JSON to endpoint and signs every delivery with the HMAC key signing_key_ref names (an optional token_ref adds a bearer token):

X-Trstctl-Timestamp: <unix seconds>
X-Trstctl-Signature: sha256=<hex HMAC-SHA256(key, timestamp + "." + body)>
X-Trstctl-Delivery:  <delivery id, the same on every retry of one delivery>

A receiver recomputes the MAC over the timestamp header, a ., and the exact body bytes it received, compares it in constant time, and refuses a timestamp more than a few minutes from its clock (trstctl's own check, secretsync.VerifyWebhookSignature, allows five), so a changed body fails and a captured delivery cannot be replayed later to roll a secret back. body_template shapes the body: JSON with ${key} (the remote key), ${value} (the secret, which must then be UTF-8), ${value_base64}, ${delivery_id} and ${timestamp} placeholders; string placeholders go inside quotes, and startup refuses a template that does not render valid JSON or carries no value. Without one the body is {"key":"…","value_base64":"…","delivery_id":"…","timestamp":…}. generic-ci-json stays the bearer-token-only JSON push with no payload signature.

An aws-parameter-store target writes each secret as a SecureString parameter at /<path_prefix>/<remote key> with PutParameter (Overwrite, so every delivery is the parameter's next version), SigV4-signed for ssm in region with access_key_id, secret_access_key_ref and an optional session_token_ref; kms_key_id picks the KMS key (empty is the account's key for SSM), and endpoint is the regional SSM endpoint (https://ssm.<region>.amazonaws.com). Parameter values are text, so a non-UTF-8 secret is refused.

A netlify target sets a secret environment variable through Netlify's API (endpoint https://api.netlify.com/api/v1, token_ref a personal access token): account_id is the team that owns the variables and the optional site_id scopes them to one site. Each delivery reads the variable first, then sets its value for deploy_context (all by default; also dev, dev-server, branch-deploy, deploy-preview, production) or creates it as a secret variable with scopes (builds, functions, runtime by default; also post-processing). Netlify applies the value on the site's next deploy.

Privileged access broker targets

The pam block enables the broker and can seed operator-owned targets. A tenant registrar with access:targets.write can add targets through the console, POST /api/v1/access/targets, or trstctl-cli access targets register -f target.json. The matching list/get/disable API and CLI commands provide readback and retirement. access:write alone can request a session but cannot register a destination. Registered destinations are immutable; disable one with a reason and register a new ID to change its provider, host, roles, or principals. Disabled targets reject new requests; active sessions retain their native expiry and revocation handles. GET /api/v1/access/targets includes tenant and operator targets with their source. Operator targets remain managed by deployment configuration.

Targets carry references and allowlists, not administrator credentials. Create the PostgreSQL dynamic-secret provider first; that provider loads its administrator DSN from a mode-0600 file: or tenant secret:// reference. Enable the tenant's attester trust source before requesting a session.

{
  "pam": {
    "enabled": true,
    "default_ttl": "15m",
    "max_ttl": "1h",
    "approval_ttl": "15m",
    "required_approvals": 2,
    "postgres_targets": [{
      "tenant_id": "22222222-2222-4222-8222-222222222222",
      "id": "incident-db",
      "provider_id": "tenant-postgres-provider",
      "allowed_roles": ["readonly"]
    }],
    "ssh_targets": [{
      "tenant_id": "22222222-2222-4222-8222-222222222222",
      "id": "incident-host",
      "host": "host.internal.example",
      "port": 22,
      "principals": ["incident-operator"]
    }]
  }
}

An enabled broker with no target starts so a tenant can register its first one. Startup rejects an unknown PostgreSQL provider or role in an operator target, an invalid SSH host/port/principal allowlist, and malformed lifetimes. The tenant registration endpoint applies the same checks before appending an event. The defaults are 15 minutes for session TTL and approval TTL, one hour for the session cap, and two distinct reviewers. required_approvals sets the quorum; zero uses two. An operator must separately configure the SSH server to trust the trstctl SSH CA. The request and activation commands are documented in Incident response and JIT access.

For Kubernetes, create a dedicated namespace and an administrator ServiceAccount whose Role can create, get, and delete ServiceAccounts, Opaque Secrets, and RoleBindings, create serviceaccounts/token, and bind only the named target Roles or ClusterRoles in that namespace. Kubernetes rejects RoleBinding creation when the issuer lacks the bound role's permissions and lacks bind; a generic rolebindings:create grant alone is insufficient. Give each allowed trstctl role an existing, narrowly scoped Kubernetes Role or ClusterRole; trstctl creates a namespace RoleBinding for each lease. Configure one kubernetes_audience that the target API server accepts. Do not use a token meant for a different relying party. The TokenRequest API rejects durations below 600 seconds, so Kubernetes leases and max_ttl must be at least 10 minutes. trstctl requests the exact lease duration, checks the response expiration, and binds the token to an Opaque Secret with the same lease name and the returned Secret UID. Each created object carries a lease ownership label, and the provider records its UID. Retries inspect the label and delete with UID preconditions; an object with the same name but another owner stops the operation instead of being removed. Revocation and expiry delete that Secret first with its UID precondition, invalidating the token at the API server, then remove the RoleBinding and ServiceAccount. A Kubernetes renewal creates a new bound token and lease, revokes the predecessor, and returns the replacement credential once. Update the workload with the replacement token and new lease ID. The renewal response is withheld until the predecessor's provider removal is confirmed. The requested replacement lifetime must be at least 10 minutes and fit within the original lease's renewal limit. Example:

{
  "type": "kubernetes",
  "endpoint": "https://kubernetes.internal.example:6443",
  "namespace": "trstctl-credential-leases",
  "bearer_token_ref": "file:/run/secrets/kubernetes-issuer-token",
  "kubernetes_audience": "https://kubernetes.default.svc.cluster.local",
  "kubernetes_tls_ca_ref": "file:/run/secrets/kubernetes-api-ca.pem",
  "allowed_roles": ["reader"],
  "role_bindings": {"reader": "Role/lease-reader"},
  "max_ttl": "30m",
  "allow_private_endpoint": true,
  "private_egress_cidrs": ["10.45.0.7/32"]
}

Both file references must be mode 0600 and readable by the control-plane user. If the endpoint is an internal alias that is absent from the API certificate, set kubernetes_tls_server_name to a certificate SAN; this setting requires kubernetes_tls_ca_ref. Keep the private CIDR list to the API server's actual resolved address. Confirm the audience, CA, native permissions, token use, and post-revocation rejection with kubectl against the target cluster.

For MySQL, set admin_dsn_ref, database, addr, and account_host explicitly. addr is the server address returned to clients; account_host is the MySQL account's allowed client host or network matcher. A missing account_host fails startup instead of creating user@'%'. Patterns containing % or _ require allow_wildcard_account_host: true. Prefer a specific client IP, hostname, or MySQL-supported IP netmask; the opt-in exists for deployments with changing client addresses and gives every matching client a chance to authenticate with the leased password. Review the network boundary before enabling it.

For Redis, password_ref is required and each allowed_roles name needs one redis_acl_roles policy. Give a literal key namespace prefix, then list only the commands that workload needs. trstctl appends the sole wildcard, so a prefix of payments: grants payments:*; raw ACL syntax, global ~*, command categories, and administrative commands are refused. ping is added for client connections. Redis SELECT is never granted: ACLs cannot restrict its database argument, so a lease could otherwise access matching key prefixes in other logical databases. Only db: 0 is supported; use separate Redis instances or distinct prefixes for separate authority domains. Configure redis_tls_server_name to match the Redis server certificate. redis_tls_ca_ref pins a private CA from a mode-0600 file: or tenant secret:// reference; if omitted, system roots are used. The administrator connection verifies TLS before sending its password, and issued credentials use rediss://. Workload clients must trust the same CA. For example:

{
  "allowed_roles": ["reader", "writer"],
  "redis_tls_server_name": "redis.internal.example",
  "redis_tls_ca_ref": "file:/run/secrets/redis-ca.pem",
  "redis_acl_roles": {
    "reader": {"key_prefixes": ["payments:"], "commands": ["get", "mget"]},
    "writer": {"key_prefixes": ["payments:"], "commands": ["get", "set", "del"]}
  }
}

Redis must use a writable ACL file, with the task's administrator allowed to run ACL SETUSER, ACL DELUSER, and ACL SAVE. trstctl saves each issue and revoke before reporting provider completion, so an ordinary Redis restart cannot resurrect a revoked user or discard a still-active lease. The administrator password belongs in a mode-0600 file: reference or tenant secret:// reference; never inline it in JSON. allow_plaintext_redis: true is an explicit local-lab exception for a literal loopback or private IP address. It cannot be combined with TLS settings and returns a redis:// lease. Do not use it across an untrusted network.

AWS, GCP, and Azure sync targets can explicitly replace their static target credential with tenant-owned workload identity. Set exactly the provider switch (aws_workload_identity, gcp_workload_identity, or azure_workload_identity) on the matching target. The switch is false by default and forbids the corresponding static credential fields when true. The workload proof itself stays behind a file: or tenant-scoped secret:// reference. Only the bounded secret-sync outbox worker opens and validates that proof, POSTs the provider exchange, caches the locked short-lived token until its refresh boundary, and feeds it into the existing hand-written target client. The request handler never performs that egress. Air-gapped mode records offline_disabled, fails that queued delivery once, and makes no token or target request.

Azure uses an Entra federated credential that accepts the already validated OIDC proof directly as the OAuth client_assertion. This is simpler than a certificate-signed assertion: it adds no private key or certificate custody. Create the matching federated-credential binding on the Entra application, then configure the Key Vault target:

secret_integrations:
  sync_targets:
    - tenant_id: 11111111-1111-4111-8111-111111111111
      id: payments-azure-key-vault
      type: azure-key-vault
      endpoint: https://payments.vault.azure.net
      azure_workload_identity: true
      # Optional. When omitted, the source's azure_tenant_id selects:
      # https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
      # workload_identity_endpoint: https://login.microsoftonline.com/.../oauth2/v2.0/token

Create the tenant-scoped source through the console or trstctl-cli secrets syncs workload-identities create --body-file source.json. For Azure, the request sets provider: "azure", azure_tenant_id, the Entra application client_id, and an allowed Key Vault target_scope such as https://vault.azure.net/.default. It also binds the exact OIDC audience and subject, target_id, allowed remote-key prefixes, workload_proof_ref, and a JWT/JWKS trust_source_id. Those routing fields are stored under PostgreSQL RLS; proof bytes and minted bearer tokens are never stored in the source row, event, job, or API response.

Every dynamic-provider and sync-target HTTP endpoint must use HTTPS in production. allow_private_endpoint grants a named private destination; it never grants plaintext. Local emulators may set allow_insecure_loopback: true, but only beside an http:// endpoint whose host is exactly localhost, an address in 127.0.0.0/8, or ::1. The runtime client resolves and dials loopback only, so changing DNS or adding a private CIDR cannot stretch this development switch to a LAN, VPC, metadata service, or public host.

Machine-auth methods beyond the HMAC token are configured in the JSON/YAML config file under secrets.machine_auth. Each entry names one method: kubernetes, aws-iam, gcp, azure, oidc, or jwt. JWT-family methods require audience plus jwks_file or jwks_json, and must set either tenant_claim (credential-bound tenancy) or tenant_id (tenant-pinned config). AWS IAM must set tenant_id and allowed_accounts or allowed_arns because STS does not carry a trstctl tenant claim. Every method must also choose exactly one permission source: non-empty static scopes, or—for OIDC/generic JWT only—one signed scopes_claim. Static scopes cannot be widened by a credential's generic scope or scopes claim. Empty, duplicate, missing, or ambiguous scope configuration fails startup instead of producing an inert or over-privileged session. For a given tenant, each method name must resolve to one configuration. Two entries with the same name are valid only when both have different fixed tenant_id values. A tenant-claim entry applies across tenants and cannot share its method name with another entry, so the login preview and exchange always use the same verifier and permission source.

Treat the credential-store KEK as the root that also protects the signer's sealed audit-evidence handle: protect it and back it up (a lost KEK means sealed credentials and software-backed signer keys cannot be opened) with the same care described in the disaster-recovery runbook. The credential-store KEK is a local key file until a rotation moves it under an operator wrapper; the file still seals the signer's key store afterwards. Do not confuse it with Helm externalKMS, which applies to the signer's CA key-store DEK wrapping described in Signer topology and CA custody. On reload, local KEK, auth-secret, and session-secret files are accepted only if they are regular files, not symlinks, owned by the process user with 0600-or-stricter permissions or mounted as root-owned Kubernetes Secret files readable by the pod's fsGroup, and all parent directories reject group/world writes. Unsafe restored files fail startup instead of silently weakening key custody.

Backup

Event-log backups (trstctl --backup) are always integrity-protected (a SHA-256 trailer, plus an HMAC domain-derived from TRSTCTL_SECRETS_KEK_FILE when the deployment KEK exists). They require the live deployment's external PostgreSQL DSN as well as external NATS: PostgreSQL holds a shared history-generation barrier for the complete export, preventing a concurrent privacy rewrite from changing which JetStream generation is authoritative mid-backup. A full backup (trstctl --full-backup-dir) additionally captures operational secrets — the signer authorization secret and the sealed signer key store (including the audit-evidence key) — so production full backups require an operator-held encryption key.

See the disaster-recovery runbook for the full backup/restore procedure, including what each artifact covers and how to restore a signer host. Every scheduled result is signed by the isolated audit-export signer handle, appended to tenant-isolated history, and available through trstctl platform dr-posture. Failed, skipped, and objective-breaching results create an idempotent notification.restore_drill intent in the same PostgreSQL transaction as that history row.

Object store

An S3-compatible object store (AWS S3, MinIO, or another service that speaks the S3 API) for durable artifacts: signed audit archive segments and scheduled report and evidence-pack artifacts when audit.archive_object_store is set, and full backup sets copied with trstctl --full-backup-upload and fetched back with --full-backup-download (disaster recovery). The control plane opens it at startup without a network call; trstctl doctor proves its tenant-prefix confinement with the OBJ probes (doctor runbook). Objects are created, never replaced: every write is conditional (If-None-Match: *) and carries its SHA-256 for the store to verify, and every read is checked against that digest. Each tenant's objects live under its own prefix (tenant/<tenant-id>/, or silo/<tenant-id>/ for a tenant the Provider isolates), beneath the optional deployment prefix.

The client sends requests only to the configured origin and follows no redirects. The endpoint may be on a private network; it does not pass through the SSRF guard that tenant-supplied URLs do. Credentials are the static access key above; instance roles and other SDK credential chains are not read. A single object holds at most 5 GiB (no multipart upload).

License

trstctl ships as a single binary. Enterprise-tier features (managed keys, HA support, FIPS artifact posture, and governance — the full Enterprise row in Editions) unlock through an offline, no-phone-home license check: the file is verified locally against public keys baked into the binary at release time. No configured file means Community edition; a corrupt or untrusted file fails startup loudly; an expired file still loads and walks a grace ladder so licensed read paths stay observable.

The ID is not a secret and does not phone home. It is a local name bound by the vendor's Ed25519 signature. Copying a non-production license to another control plane without giving that deployment its own signed ID fails startup. See Editions for the claim shape and Editions for what is included.

The related TRSTCTL_PCAS_* variable surface (settings covering delegation, recovery, federation, KEM, checkpoints, monitors, and retirement for patent-covered credential algorithm succession) is off by default. PCAS is part of the core build and does not require an Enterprise or Provider license. The PCAS operations runbook documents first use and the checkpoint worker's exact startup settings. Other optional PCAS workers require configured trust and custody inputs before a positive operator journey. Delegation scope commands use the isolated signer transport and its persistent keystore. The old pcas.delegation block and TRSTCTL_PCAS_DELEGATION_* control-plane settings are rejected with a migration hint because they never configured the separate signer. The PCAS operations runbook shows queued commands, signer-confirmed readback, and how to handle older unconfirmed rows.

Conditional managed-key adapters (six served providers)

The managed-key lifecycle is off by default and requires an Enterprise license plus startup configuration. AWS KMS, Azure Key Vault / Managed HSM, GCP Cloud KMS, PKCS#11, TPM 2.0, and YubiHSM 2 are all required/census-served through the shipped cgo HSM signer artifact. “Conditional” means the route is inert until the operator selects and provisions a provider; it no longer means library-only. The control plane records a tenant event and PostgreSQL outbox command, while the isolated signer constructs the provider and performs the private operation. The control-plane process never receives provider credentials.

The Certificate authorities → Key custody console reads GET /api/v1/managed-keys/custody to show this startup configuration as a secret-free plan. It names environment variables and signer-only file-reference requirements for all six providers; it does not accept their values. After startup configuration is applied and the control plane and isolated signer are restarted, the console calls POST /api/v1/managed-keys/preview. That request is effect-free: it writes no event or projection and makes no provider call. Generation stays locked until the preview confirms the selected provider is one the deployment runs (and, for a pinned tenant, its pin), the lifecycle is attached, and the exact plan is ready.

Several providers at once

A deployment can run several providers together, for example a PKCS#11 HSM for one tenant and AWS KMS for another. List them in managed_keys.providers (TRSTCTL_MANAGED_KEYS_PROVIDERS=pkcs11,aws) and configure each one's block; managed_keys.provider stays the default. The isolated signer builds every listed provider at startup and records each key's provider in its journal. A key is created under the provider its generate request names. managed_keys.tenant_providers pins a tenant to one provider (TRSTCTL_MANAGED_KEYS_TENANT_PROVIDERS=<tenant-id>:pkcs11,...): that tenant's preview and generate accept only its pin, and the console starts on it. Every key reports its provider in the response and the inventory. Rotate, revoke, zeroize and verify-custody find the key's provider from its handle, or take provider in the request when the same handle exists under two providers. An external signer's --managed-keys-config lists the same providers.

managed_keys:
  enabled: true
  provider: pkcs11
  providers: [pkcs11, aws]
  tenant_providers:
    - {tenant_id: 11111111-1111-4111-8111-111111111111, provider: pkcs11}
    - {tenant_id: 22222222-2222-4222-8222-222222222222, provider: aws}
  pkcs11: {module_path: /usr/lib/softhsm/libsofthsm2.so, token_label: trstctl, user_pin_file: /run/secrets/pkcs11-pin}
  aws: {region: us-east-1, access_key_id: AKIA..., secret_access_key_file: /run/secrets/aws-kms-secret}

The providers are read at startup; adding or removing one needs a restart of the control plane and the signer. A key under a provider the deployment no longer runs stays in the inventory but cannot be changed until that provider returns.

Development-only LocalStack shape (not conformance or shipped-runtime proof):

export TRSTCTL_MANAGED_KEYS_ENABLED=true
export TRSTCTL_MANAGED_KEYS_PROVIDER=aws
export TRSTCTL_MANAGED_KEYS_AWS_REGION=us-east-1
export TRSTCTL_MANAGED_KEYS_AWS_ENDPOINT=http://127.0.0.1:4566
export TRSTCTL_MANAGED_KEYS_AWS_ALLOW_INSECURE_LOOPBACK=true
export TRSTCTL_MANAGED_KEYS_AWS_ACCESS_KEY_ID=test
printf '%s' test > /tmp/localstack-kms-secret
chmod 0600 /tmp/localstack-kms-secret
export TRSTCTL_MANAGED_KEYS_AWS_SECRET_ACCESS_KEY_FILE=/tmp/localstack-kms-secret

Private endpoint CIDR grants control where HTTPS may go. They never weaken TLS. The three ALLOW_INSECURE_LOOPBACK switches are deliberately separate and accept only loopback HTTP for a same-host emulator.

Example production shape:

managed_keys:
  enabled: true
  provider: aws
  aws:
    region: us-east-1
    access_key_id: AKIA...
    secret_access_key_file: /etc/trstctl/aws-kms-secret-access-key

Example Azure Key Vault HSM shape:

managed_keys:
  enabled: true
  provider: azure-key-vault
  azure:
    vault_url: https://trstctl-prod.managedhsm.azure.net
    bearer_token_file: /etc/trstctl/azure-kv-token

Example GCP Cloud KMS shape:

managed_keys:
  enabled: true
  provider: gcp-kms
  gcp:
    parent: projects/prod/locations/us/keyRings/trstctl
    bearer_token_file: /etc/trstctl/gcp-kms-token

Example PKCS#11 HSM shape:

managed_keys:
  enabled: true
  provider: pkcs11
  pkcs11:
    module_path: /usr/lib/softhsm/libsofthsm2.so
    token_label: trstctl-prod
    user_pin_file: /etc/trstctl/pkcs11-user-pin
    key_label_prefix: trstctl-ca

Example TPM 2.0 shape:

managed_keys:
  enabled: true
  provider: tpm2
  tpm2:
    path: /dev/tpmrm0
    owner_auth_file: /etc/trstctl/tpm-owner-auth
    key_auth_file: /etc/trstctl/tpm-key-auth
    persistent_handle_base: 2164326400 # 0x81010000

Example YubiHSM 2 shape:

managed_keys:
  enabled: true
  provider: yubihsm2
  yubihsm2:
    module_path: /usr/lib/yubihsm_pkcs11.so
    token_label: trstctl-prod
    user_pin_file: /etc/trstctl/yubihsm-auth
    key_label_prefix: trstctl-ca

Static no-cgo signer builds fail closed if pkcs11 or yubihsm2 is selected. Use the published artifact built by deploy/docker/Dockerfile.signer-hsm; its release profile enables cgo and installs /usr/local/bin/trstctl-signer-hsm.

When the licensed attach seam succeeds, operators with keys:write can exercise POST /api/v1/managed-keys and the rotate/revoke/zeroize API and CLI shapes. Before each destructive action, two different principals with keys:approve record the exact opaque key_id and rotate, revoke, or zeroize action through POST /api/v1/managed-keys/approvals; the requester cannot self-approve. Requests require Idempotency-Key; the requester must replay that same key after approval, because a different key creates and supersedes a different immutable request. The console retains its key in first-party storage for the signed-in tenant, requester, key, and action until the provider effect is terminal. Clearing site storage or switching browser profiles requires a new request and two new approvals. Lifecycle events omit private bytes; tenant projections use PostgreSQL RLS; and provider work is delivered by the durable outbox to the separate signer. The signer writes an fsync-backed operation intent before provider I/O. Every shipped provider also carries that identity into provider state: an atomic AWS tag, deterministic Azure/GCP resource identity, deterministic PKCS#11 CKA_ID (including YubiHSM), or a full-width TPM public tag plus deterministic handle probing. If a provider commits and the signer dies before journaling the response, restart finds the same effect; revoke/zeroize likewise confirm the provider accepted the effect rather than blindly repeating a mutation. For cloud KMS, zeroize returns deletion_pending: signing is blocked, while provider-held material may remain until its retention period ends. Check the provider's deletion deadline with its stock client. zeroized is reserved for confirmed local device destruction; older cloud events remain immutable and are presented conservatively as deletion_pending by the API and console. Managed-key outbox circuits are partitioned by provider and current key (or operation ID before a key exists), so one unavailable key does not stall every managed-key command. All six exact backend census rows are SERVED + REQUIRED. For an externally deployed signer, mount the same file-backed provider descriptor and pass it with --managed-keys-config plus the signed --license; never copy a credential into argv or an inline JSON field.

Signer topology and CA custody

The private-key operations run in a separate, sacred process, so the CA keys never live in the API process. Its issuing CA key is persisted, sealed at rest (R3.2) so a restart preserves the CA instead of silently rotating it. The signer can run two ways:

In Helm deployments, the signer key store uses local KEK custody by default: kek.existingSecret or eval-only kek.generate=true mounts /etc/trstctl/kek/kek.bin and the chart passes --kek to trstctl-signer. Regulated deployments can instead wrap the key store's data keys under a key held by AWS KMS, Google Cloud KMS, Azure Key Vault (or Managed HSM), or a PKCS#11 HSM:

externalKMS:
  enabled: true
  provider: awskms        # awskms | gcpkms | azurekv | pkcs11
  keyRef: arn:aws:kms:us-east-1:111122223333:key/0b1c2d3e-4f5a-6b7c-8d9e-0f1a2b3c4d5e
  wrapCommand: /usr/local/bin/trstctl-kms-wrap   # the shipped adapter (the default)
  adapterConfigSecret: trstctl-kms-wrap           # config.json plus credential files
  timeout: 10s

With externalKMS.enabled=true, the chart passes --kms-provider, --kms-key-ref, --kms-wrap-command, and --kms-timeout to the signer and does not mount the local KEK Secret. The signer invokes the adapter without a shell as <wrapCommand> wrap|unwrap <provider> <keyRef>, with data-key bytes only on stdin/stdout; a non-zero exit or a timeout fails the operation closed. Missing provider/keyRef/command values, unsupported provider names, a relative command path, or the shipped adapter without adapterConfigSecret fail at template time. Before it serves, the signer wraps and unwraps a random value through the adapter, so a missing, disabled or unreachable key stops startup with the provider's own reason (for example AWS KMS's DisabledException) rather than a generic seal error at the first key load.

The images ship that adapter as /usr/local/bin/trstctl-kms-wrap. The chart mounts adapterConfigSecret at /etc/trstctl/kms-wrap and points the adapter at its config.json through TRSTCTL_KMS_WRAP_CONFIG (outside Helm, set that variable for the signer, or put --config <file> in the wrap command). The file holds one block per provider, with credentials only as file paths:

{
  "aws":    {"access_key_id": "AKIA…", "secret_access_key_file": "/etc/trstctl/kms-wrap/aws-secret",
             "session_token_file": "", "region": "", "endpoint": "", "private_egress_cidrs": []},
  "gcp":    {"bearer_token_file": "/etc/trstctl/kms-wrap/gcp-token", "endpoint": ""},
  "azure":  {"bearer_token_file": "/etc/trstctl/kms-wrap/azure-token", "algorithm": "RSA-OAEP-256"},
  "pkcs11": {"module_path": "/usr/lib/softhsm/libsofthsm2.so", "token_label": "trstctl",
             "user_pin_file": "/etc/trstctl/kms-wrap/pin"}
}

pkcs11 needs the adapter's cgo build, which ships in the HSM signer image (trstctl-signer-hsm) next to the signer; the standard image's static adapter refuses pkcs11 with that pointer. Endpoints follow the managed-key rules: HTTPS, or HTTP to loopback only with allow_insecure_loopback, and private addresses only inside private_egress_cidrs. You may still name your own adapter in wrapCommand; it must speak the same contract.

Externally wrapped custody applies to a key store from its first key: moving a key store the local KEK already sealed onto a KMS or HSM key (re-wrapping each key's data key) is what comes next. Today, start the signer with --kms-* on an empty key store.

Back up the sealed key store, the signer custody input, and the CA cert together (the CA-key recovery set) per the disaster-recovery runbook. For local-KEK mode that custody input is the signer KEK Secret; for externalKMS mode it is access to the same provider keyRef plus the adapter, its configuration and the provider credentials. The deploy/docker/docker-compose.yml in the source checkout runs the signer as its own service in external mode.

Regulated CA governance mode

The individual issuance controls — the OPA/Rego policy gate, four-eyes dual control, a bound default certificate profile, revocation publication, and FIPS — can each be enabled on their own. For a compliance deployment that is error-prone: a single missing control silently weakens the posture. ca.governance_mode=regulated is the one coherent switch that closes that gap. In regulated mode the binary fails startup unless all of these are present together, each with an actionable error naming the field to set:

• the OPA policy gate is on (ca.policy.enabled=true);
• four-eyes dual control is on (ca.policy.require_approval=true) with at least two distinct approvers (ca.policy.required_approvals unset, defaulting to 2, or >= 2) — a single approver is rejected;
• a default certificate profile is bound (ca.default_profile);
• revocation publication is configured — at least one of ca.crl_distribution_points or ca.ocsp_servers — so issued leaves carry a status pointer for the built-in CA (composing with the served-leaf profile);
• and, when ca.require_fips=true is declared, the FIPS 140-3 module is active (the binary was built with GOFIPS140=v1.0.0 / make fips-build, or run with GODEBUG=fips140=on).

In regulated mode, CA key ceremonies also need at least two custodian approvals besides the opener, whatever the request asks for (see ca.ceremony_min_approvals). Managed CA leaves require the separate ca.hierarchy_revocation_base_url before issuance in this mode. It must be the externally reachable HTTP(S) origin of this control plane. The issuer-specific CRL and OCSP paths are added to each leaf; the built-in CA's global status pointers are never copied onto managed CA leaves.

A complete regulated config boots normally. The default posture (ca.governance_mode unset, or standard) imposes no coupling, so existing single-node deployments are unaffected.

Served AI surface and model adapter

The AI/RCA/MCP surface is off by default. MCP investigation tools are read-only when enabled; MCP write tools require the separate TRSTCTL_AI_MCP_WRITE_TOOLS=true operator opt-in. The control plane serves the tools as authenticated REST routes (GET /api/v1/mcp/tools, POST /api/v1/mcp/tools/{tool}); a standard Model Context Protocol client connects through trstctl-cli mcp serve, which speaks the MCP stdio transport (JSON-RPC 2.0: initialize, tools/list, tools/call) and forwards every call to those routes with the caller's own token, so tenant scope, RBAC, rate limits and audit apply unchanged. Verified with the official MCP SDK client. The model adapter is separately off by default: with TRSTCTL_AI_MODEL_MODE=off (or unset), query/RCA still return grounded citations, but no prompt leaves the process. GET /api/v1/ai/status reports the live enabled state, model mode, endpoint host, egress class, and redaction/refusal posture without echoing the full endpoint URL.

Ollama local mode sends the native generate shape to the endpoint. vLLM and cloud mode send an OpenAI-compatible chat-completions shape. Every model path goes through the boundary redactor and residual-secret refusal gate before the HTTP request is made; if no model is configured, the answer remains citation-grounded and air-gapped.

Served protocol listeners

ACME, EST, SCEP, CMP, SPIFFE, SSH, and KMIP protocol surfaces are opt-in until they are explicitly bound to a tenant. That startup check is intentional: a public protocol endpoint must know the tenant it acts for before it is exposed. KMIP is a raw mTLS TCP listener, not an HTTP route, so it additionally requires server certificate/key files and a client CA trust anchor, unless client registrations serve each tenant's own clients.

For the blank evaluation stack, TRSTCTL_PROTOCOLS_PROFILE=eval is a bounded shortcut: it assembles ACME, EST, SCEP, CMP, SSH, TSA, and SPIFFE for exactly TRSTCTL_PROTOCOLS_EVAL_TENANT_ID. Startup registers that explicitly configured evaluation tenant through the event log before first issuance; a later startup verifies its retained registration. It cannot automatically recreate a previously registered tenant after erasure or missing projection state. Production settings and arbitrary SSO tenant claims do not create tenants. Those responders stay behind a closed runtime gate until a tenant-authenticated operator activates them in the first-run wizard or with POST /api/v1/setup/protocols/activate. Activation appends an immutable tenant event before the HTTP routes and SPIFFE socket become reachable, and replay restores the gate after restart. The shortcut never enables KMIP. Production can omit PROFILE and use the individual toggles below. To try a full certificate journey against an endpoint of your own, from discovery to a signed audit export, run trstctl evaluate.

Each entry in protocols.acme_eab.keys[] also carries the scope that credential authorizes — allowed_identifiers (exact names or *.suffix, which covers the apex too), max_orders, an RFC 3339 not_after, and disabled. All are optional and a key with none behaves as it did before. An order outside a credential's scope, past its quota, or after its window is refused fail-closed and recorded. Scope has no env shortcut: it is per-key structured config. See ACME external account bindings.

For an internal ACME profile with acme_auth_mode: trust_authenticated, select it with ca.default_profile and require EAB. Put each HMAC key in a 0600 file and reference it as protocols.acme_eab.keys[].hmac_key_file; the credential's allowed_identifiers must cover only the intended internal names. The server checks the active tenant profile at each order and refuses a trusted order from an account without EAB, even if the account predates the configuration change.

| TRSTCTL_PROTOCOLS_ACME_MAX_NONCES | 4096 | Maximum outstanding ACME replay nonces retained by the tenant-bound ACME mount. | | TRSTCTL_PROTOCOLS_ACME_MAX_ACCOUNTS | 2048 | Maximum ACME accounts retained by the tenant-bound ACME mount. | | TRSTCTL_PROTOCOLS_ACME_MAX_PENDING_ORDERS | 4096 | Maximum pending ACME orders retained before the server returns ACME rateLimited (429). | | TRSTCTL_PROTOCOLS_ACME_MAX_PENDING_AUTHORIZATIONS | 8192 | Maximum pending ACME authorizations retained across pending orders. | | TRSTCTL_PROTOCOLS_ACME_MAX_PENDING_CHALLENGES | 24576 | Maximum pending ACME challenge records retained across pending authorizations. | | TRSTCTL_PROTOCOLS_ACME_MAX_PENDING_ORDERS_PER_ACCOUNT | 128 | Per-account pending-order cap, independent from source-IP budgets. | | TRSTCTL_PROTOCOLS_ACME_MAX_NEW_NONCES_PER_SOURCE | 120 | Per-source newNonce budget per source window. | | TRSTCTL_PROTOCOLS_ACME_MAX_NEW_ACCOUNTS_PER_SOURCE | 20 | Per-source account-creation budget per source window. | | TRSTCTL_PROTOCOLS_ACME_MAX_NEW_ORDERS_PER_SOURCE | 60 | Per-source order-creation budget per source window. | | TRSTCTL_PROTOCOLS_ACME_SOURCE_WINDOW_SECONDS | 600 | Source-budget window for ACME nonce/account/order creation. | | TRSTCTL_PROTOCOLS_ACME_NONCE_TTL_SECONDS | 600 | TTL for unused ACME replay nonces before the request-time janitor drops them. | | TRSTCTL_PROTOCOLS_ACME_STATE_TTL_SECONDS | 86400 | TTL for pending ACME order/authorization/challenge state before the request-time janitor drops it. | | TRSTCTL_PROTOCOLS_EST_ENABLED / …_TENANT_ID | false / — | Serve EST at /.well-known/est/... for the named tenant. | | TRSTCTL_PROTOCOLS_EST_MTLS_ADDR | — (off) | Also serve EST on a second HTTPS listener at this address that requires a client certificate issued by the enrollment CA. On it, /.well-known/est/simplereenroll authenticates a device by the certificate it presents and binds the renewal CSR to that certificate's identifiers; no bootstrap token is needed for renewals. The main listener never requests a client certificate, so browsers are not prompted. Uses the same server identity and TLS floor as the main listener; refused under server.tls.mode=disabled. | | TRSTCTL_PROTOCOLS_SCEP_ENABLED / …_TENANT_ID | false / — | Serve SCEP at /scep for the named tenant. | | TRSTCTL_PROTOCOLS_CMP_ENABLED / …_TENANT_ID | false / — | Serve CMP at /cmp for the named tenant. | | TRSTCTL_PROTOCOLS_TSA_ENABLED / …_TENANT_ID | false / — | Serve RFC 3161 timestamp evidence for the named tenant through the signer-held timestamping key, instead of minting inventory certificates. | | TRSTCTL_PROTOCOLS_RA_KEY_FILE | data/protocols/ra-transport.key | Sealed SCEP/CMP RSA transport identity. Put this on shared persistent storage in HA so replicas use the same cached-client RA material. | | TRSTCTL_PROTOCOLS_TSA_CERT_FILE | data/protocols/tsa.crt | Where the TSA's timestamping certificate is persisted, so it stays stable across restarts and is shared across HA replicas. Required when TSA is enabled. | | TRSTCTL_PROTOCOLS_KMIP_ENABLED / …_TENANT_ID | false / — | Serve the KMIP mTLS listener for the named tenant: AES-256 symmetric keys, and RSA/ECDSA key pairs that sign and verify on the server (see Secrets). | | TRSTCTL_PROTOCOLS_KMIP_ADDR | :5696 | TCP listen address for KMIP. | | TRSTCTL_PROTOCOLS_KMIP_CERT_FILE | — | PEM server certificate chain for the KMIP listener. Required when KMIP is enabled. | | TRSTCTL_PROTOCOLS_KMIP_KEY_FILE | — | PEM private key for the KMIP listener certificate. Required when KMIP is enabled. | | TRSTCTL_PROTOCOLS_KMIP_CLIENT_CA_FILE | — | PEM CA bundle used to verify KMIP client certificates. Required when KMIP is enabled, unless client registrations are on; then it serves the CA-issued clients of …_TENANT_ID. | | TRSTCTL_PROTOCOLS_KMIP_CLIENT_REGISTRATIONS | false | Serve every tenant's registered KMIP clients on this listener: a client's first request binds its own certificate with its tenant's one-time enrollment (/api/v1/kmip/clients). The tenant ID and client CA become optional. | | TRSTCTL_PROTOCOLS_SPIFFE_ENABLED / …_TENANT_ID | false / — | Serve the SPIFFE Workload API UDS for the named tenant. Requires TRSTCTL_PROTOCOLS_SPIFFE_TRUST_DOMAIN. | | TRSTCTL_PROTOCOLS_SPIFFE_SOCKET_PATH | /tmp/trstctl-spiffe-workload.sock | UDS path for the SPIFFE Workload API when enabled. | | TRSTCTL_PROTOCOLS_SPIFFE_TRUST_DOMAIN | — | SPIFFE trust domain, for example example.org. Required when SPIFFE is enabled. | | TRSTCTL_PROTOCOLS_SPIFFE_SERVE_LOCAL_WORKLOADS | false | Keep the control plane's own Workload API socket issuing SVIDs to workloads on the control-plane machine (protocols.spiffe.serve_local_workloads). Off, the socket serves trust bundles and JWT validation only and refuses FetchX509SVID/FetchJWTSVID, naming the host agent's --workload-api-socket; every refusal and, when on, every issuance is recorded as spiffe.workload_api.local_socket_used. The eval profile turns it on. | | TRSTCTL_PROTOCOLS_SSH_ENABLED / …_TENANT_ID | false / — | Serve the SSH CA JSON endpoints and KRL for the named tenant. |

Agent mTLS channel

The served agent gRPC channel is a real listener the control plane exposes for steady-state agent heartbeat, renewal, and command fan-out (WIRE-004 / OPS-005) — separate from the one-shot bootstrap enrollment path, which always works. It is off by default; enabling it without a configured signer (the agent CA is custodied there) is a startup error. At startup the agent CA's key must pass the weak-key checks and the signer must prove it holds that key by signing a fresh challenge; either failure stops startup with startup binding agent CA (signer handle "agent-ca"): ... naming the check.

Both the gRPC and HTTPS renewal listeners request a 24-hour server certificate from the existing agent CA. Each listener renews when half of its current leaf's remaining validity has elapsed. Certificate renewal retains the same local transport key and CA trust; connected agents do not need new trust bundles. The transport key is held in locked memory and destroyed when the listener stops.

A failed signer request keeps the previous certificate until its actual expiry. One worker per listener retries between once a second and once every 30 seconds; TLS handshakes do not wait for those requests. Expiry still blocks fresh verified connections. Restoring the signer lets the worker recover without a control-plane restart. This renews the listener leaf, not the agent CA itself. Monitor the listener expiry and issuance metrics before expiry interrupts agent claims and application-certificate delivery.

Which work agents may claim. agent_channel.claimable_job_kinds, or its comma-separated TRSTCTL_AGENT_CHANNEL_CLAIMABLE_JOB_KINDS environment form, lists the estate-touching job kinds enrolled agents may lease and execute: connector.deploy, connector.test, connector.rollback, endpoint.renew, endpoint.contain, endpoint.purge, connector.takedown, endpoint.verify, discovery.run, revocation.probe, adcs.inventory, trust.distribute, cmdb.sync, mdm.sync, ticket.sync, and agent.upgrade. Enabling endpoint.contain also enables endpoint.recovery_check, the read-only check of whether a stopped listener can come back that a replacement reads after a containment.

Empty is the default and means the job ledger is served but hands nothing out. Enable a kind when its agent-side executor ships; enabling one earlier fills the queue with work nothing can perform while the control plane's own worker stops doing it. Anything outside that list is dropped even if you write it here, so ca.issue and notification.expiry cannot be moved onto a host — those are the control plane's own effects. GET /api/v1/operations/jobs and the Operations console show what is waiting, what an agent holds, and how long the oldest job has waited.

Who published the upgrade. agent_channel.upgrade_publisher_key_file, or TRSTCTL_AGENT_CHANNEL_UPGRADE_PUBLISHER_KEY_FILE, pins the Ed25519 public key of whoever publishes agent builds. With it set, opening a staged upgrade campaign refuses any artifact that does not carry that publisher's signature over its sha256 (artifacts[].signature, base64). Each agent can pin the same key with --upgrade-publisher-key <pem> and then refuses to stage an artifact that is the pinned bytes but not the pinned publisher's. Neither key set keeps the operator-supplied sha256 as the only integrity control.

Revocation checks for scanned certificates. agent_channel.probe_scanned_certificates, or TRSTCTL_AGENT_CHANNEL_PROBE_SCANNED_CERTIFICATES, is off by default. When on, a relay's TLS discovery sweep returns the public bytes of each leaf it finds and of up to four issuers served after it, the control plane keeps only bytes that hash to what the relay reported, and the revocation health scheduler checks each such certificate's own serial at its CRL distribution points and OCSP responders against the served issuer that signed it (up to 256 scanned certificates per tenant per sweep). The answers appear on the certificate's detail. Relays older than the setting return no bytes.

How much one agent may hold. agent_channel.max_live_claims_per_agent, or TRSTCTL_AGENT_CHANNEL_MAX_LIVE_CLAIMS_PER_AGENT, caps the unexpired leases a single agent may hold at once. At the cap the channel withholds new claims for that agent until one of its leases completes, fails or lapses; below it a claim batch is trimmed to the slots left. Zero, the default, sets no per-agent quota and leaves the shared agent bulkhead as the only cap. The Agents page shows each agent's live claims against the quota, so a withheld claim is visible there rather than inferred from a queue that stops moving.

The evaluation Compose stack enables only discovery.run, endpoint.verify, and connector.test, matching the safe actions its blank-stack console offers. It does not enable connector.deploy; an evaluation install must not gain appliance-write authority merely because it started successfully. A host-family connector.test uses the enrolled host agent's local exec profile to validate roots and commands and performs only a read-only TLS handshake; it cannot write or reload. Production remains empty and fail-closed until an operator chooses each kind.

See Getting started for the blank Compose stack's published agent-channel port and the local CA-pinning steps to reach it from an agent CLI.

Private HTTPS targets reached by a network relay. The agent's --ca-bundle authenticates the control plane only. If an appliance, CMDB, or other relay target uses an internal CA that the relay host does not already trust, start the network agent with --relay-claim --relay-ca-bundle /path/to/appliance-ca.pem. The relay adds the PEM certificates to the host's normal HTTPS trust roots; it still checks the URL hostname, requires TLS 1.2 or newer, and refuses credential-bearing redirects. An unreadable or invalid PEM stops the agent at startup. After startup the relay re-reads the file when it changes (checked at most every five seconds, before the next request): a valid replacement is trusted from the next connection with no restart, and an unusable replacement keeps the bundle that worked and is reported, with the digest and load time of the bundle in force, in the agent's census on the Agents page (reported_relay_ca_bundle on GET /api/v1/agents). Keep the control-plane CA and appliance CA in separate files so adding appliance trust does not silently expand bootstrap trust. Windows service installation retains the relay CA path across service restarts.

What a relay may reach. A relay does not run the control plane's SSRF guard, because the private devices in its segment are exactly what that guard refuses. It runs its own egress policy on every connection it opens to run a job: appliance and API calls, endpoint checks, discovery scans, directory (LDAP) reads and its TLS probes. Link-local addresses, cloud instance metadata (169.254.169.254, 100.100.100.200, fd00:ec2::254), unspecified and multicast addresses are always refused. To refuse everything else as well, list what the relay may reach:

trstctl-agent --relay-claim --relay-egress-allow 10.20.0.0/16,192.0.2.7,198.51.100.9:8443 ...

Entries are CIDRs, addresses and address:port pairs ([fd00::5]:443 for IPv6). Host names are refused, since a connection sees only the address a name resolved to. Each request's named destination is checked before it is sent, so a request carried by HTTPS_PROXY is checked too, and each dial's resolved address when it connects. A refusal is reported with the job, naming the address and the reason. With an allowlist, list the local listeners a host job reloads (127.0.0.1) as well. The agent prints the policy in force at startup, and Windows service installation retains it across restarts.

Why a deploy failed. The control plane receives a closed phrase for a failed deploy, because an appliance can echo the credential it received back in an error. The agent keeps the rest on its own host, in a failure log of its newest 200 failed deploys: when, the job and attempt, the connector and target, the step that failed (intent, executor, redeem, material, connector, sandbox, predecessor or verify) and why. Redeemed values, PEM blocks and the values of authorization headers and credential-shaped fields are removed before anything is written. The file is mode 0600, beside --key as deploy-failures.jsonl unless --deploy-failure-log names it, and Windows service installation retains that flag. Read it on the host:

trstctl-agent --show-deploy-failures --key /var/lib/trstctl-agent/agent.key

Relay-local CRL and OCSP cache

A certificate enrolled with the network role may serve revocation data to an isolated segment. Start it with --revocation-cache-config <file>. The JSON file is limited to 1 MiB, rejects unknown fields, and contains public issuer certificates and routes—not private keys, tokens, or bind credentials:

{
  "listen": "10.42.0.12:8088",
  "segment": "plant-7",
  "refresh_interval": "5m",
  "issuers": [
    {
      "id": "manufacturing-root",
      "issuer_file": "/etc/trstctl/manufacturing-root.pem",
      "crl": {
        "upstream_url": "https://pki.internal/crl/manufacturing.crl",
        "local_path": "/crl/manufacturing"
      },
      "ocsp": {
        "upstream_url": "https://ocsp.internal/manufacturing",
        "local_path": "/ocsp/manufacturing"
      }
    }
  ]
}

Issuer and segment IDs are bounded tokens, every local path must be unique, and the refresh interval must be between one minute and 24 hours. RFC 1918 and IPv6 ULA upstreams are allowed because an internal CA is the normal case; loopback, link-local, metadata-service, multicast, CGNAT, credentials-in-URL, and DNS-rebind targets remain refused. CRLs are fetched on the refresh interval. OCSP is fetched on demand and only nonce-free responses are reused. The listener serves no stale or unverified bytes.

The first mTLS heartbeat already contains the relay's signed, metadata-only cache status. Read it at GET /api/v1/revocation/caches or Protocols → Revocation cache by segment. That status includes segment, issuer fingerprint, local path, signed time window, validation time, and request counts. It never includes the upstream URL, issuer/response bytes, or OCSP request. The older --crl-cache-* flags remain for a single CRL; pair them with --revocation-cache-segment so their heartbeat has an operator-defined segment identity.

An agent runs the shipped connector families it executes — host connectors on its own machine, appliance and cloud connectors as a network relay — in the WASM connector sandbox by default: each deploy, readback, rollback, takedown, dry run or appliance-key operation runs in a fresh instance of the embedded, digest-pinned connector module, whose only effects are the operations its grant allows, performed by the agent through the host profile or the relay's HTTP client. --connector-isolation=in_process runs them as trusted code in the agent process instead; a Windows service keeps the choice across SCM restarts.

For host connector execution, trstctl-agent --host-exec-profile <file> also initializes an encrypted two-generation predecessor ledger. Set --host-rollback-dir <absolute-directory> to choose its durable location; when omitted it is host-rollbacks beside --key. Back up or persist that directory with the agent identity. If it is lost, the exact host predecessor cannot be reconstructed by the control plane and rollback is refused.

Before restoring a predecessor, the agent requests rollback_authorize over its existing mTLS job channel. The control plane checks the exact job, agent, live claim attempt, predecessor revocation state, and any bound identity's revocation or retirement state. The agent opens its saved material only after acceptance. Use updated control-plane and agent binaries together: an older server does not understand this request, so an updated agent refuses restoration. Ordinary lease extension also requires the current claim attempt; an expired or stale attempt cannot renew its lease. Host rollback still redeems no control-plane credential.

The agent keeps one terminal job report at a time in encrypted pending-reports state beside --key until the control plane acknowledges it. If that report cannot be opened under the agent's current registration (after re-enrollment against a rebuilt control plane, a changed --ca-bundle file or --server-name, a missing sealing key, or damage), the agent keeps running with job claims paused. Heartbeats, its own certificate renewal and inventory continue. Its executor census reports the reason (reported_pending_report_error on the agent, shown on Agents), and a deploy or host migration bound to it is refused with HTTP 409 naming the agent. To deliver the report, restore the previous settings. Otherwise stop the agent and run it once with the same flags plus --set-aside-pending-report: it moves the report, intact and unsent, to pending-reports/set-aside-<UTC>.state and exits, and claims resume when the agent starts. A report the agent can still send is never moved, and at most four set-aside reports are kept. A set-aside report is never sent under the new registration; the control plane that issued its job retries or fails that job when the lease expires. To recover one set aside by mistake, stop the agent, restore the previous settings, and move the file back to pending-reports/pending.state.

WASM plugins

The WASM plugin surface is off by default. When enabled, the binary admits only signed modules whose detached Ed25519 signature verifies against the configured trusted keys. CA plugins become external CA entries of type wasm-ca; DNS-provider plugins become a served ACME DNS-01 provider that tenant provider configs can select; connector plugins handle matching deployment work from the outbox.

SPIRE upstream authority plugin

When SPIRE should keep serving workload SVIDs but trstctl should own the upstream CA, configure SPIRE's UpstreamAuthority "trstctl" plugin. SPIRE passes this as HCL plugin_data to the trstctl-spire-upstream-authority process; these are not trstctl environment variables.

Example:

UpstreamAuthority "trstctl" {
  plugin_cmd = "/opt/spire/plugins/trstctl-spire-upstream-authority"
  plugin_data {
    endpoint = "https://trstctl.example.com:8443"
    ca_bundle_file = "/run/secrets/trstctl-server-ca.pem"
    allow_private_cidrs = ["10.96.42.15/32"]
    ca_authority_id = "11111111-1111-1111-1111-111111111111"
    token_file = "/run/secrets/trstctl-spire-token"
    common_name = "SPIRE Server CA"
    ttl_seconds = 3600
    max_path_len = 0
    permitted_dns_domains = ["example.org"]
  }
}

Rate limiting

A per-tenant, PostgreSQL-backed rate limiter sheds load on the guarded routes (429 + Retry-After). See Operations & resilience for the model and the bulkheads it complements.

ACME also has protocol-local abuse budgets above because its public nonce/account/order routes are not REST API routes. The protocol bulkhead limits concurrent work; the ACME quota knobs bound retained ACME protocol state.

When enabled, requests must be positive and window a valid positive duration, or the control plane fails fast at startup.

Bulkheads

Each subsystem runs on its own bounded worker pool. workers caps concurrent work; queue caps accepted backlog before trstctl rejects fast with structured backpressure. Every value must be positive. Defaults are conservative for a single-node or small HA deployment; larger fleets should raise only the subsystem that is actually saturating.

Fleet-size guidance:

trstctl --check-config prints the effective bulkheads.<subsystem>.workers and bulkheads.<subsystem>.queue values, so CI/CD can diff the resolved runtime limits before a rollout.

Every outbox family owns a separate queue and worker set even when all limits are inherited from bulkheads.outbox. In plain terms, a connector endpoint that stops answering can fill only the connector lane; it cannot occupy the workers that issue through an upstream CA, synchronize a secret, publish code-signing evidence, or page an operator. JSON config may override a family with outbox_external_ca, outbox_connectors, outbox_secrets, outbox_secret_sync, outbox_managed_keys, outbox_transparency, outbox_code_signing, or outbox_notifications, outbox_audit_feeds, outbox_tenant_seal, or outbox_fleet_reissuance inside bulkheads.

Config file

Any of the above can also be set in a JSON file named by TRSTCTL_CONFIG_FILE; environment variables override file values, which override defaults.

{
  "server": { "addr": ":8443" },
  "postgres": { "mode": "external", "dsn": "postgres://..." },
  "nats": { "mode": "external", "url": "nats://..." },
  "telemetry": { "enabled": false, "instance_id_file": "data/telemetry/instance-id" }
}

Native TLS discovery for cryptographic inventory

A control plane that inventories endpoints with certificate algorithms unsupported by its Go TLS runtime can use an operator-installed OpenSSL build for discovery. This setting identifies public certificates; it does not establish application availability, issuer trust, revocation enforcement, or complete migration.

Set cbom.tls_probe_openssl in the local JSON configuration, or set TRSTCTL_CBOM_TLS_PROBE_OPENSSL to the absolute path of the actual executable. The default is empty. The executable and its parent directories must be controlled by the deployment operator, and its build must support the endpoint's algorithm. Startup refuses a relative path, missing file, symlink, or non-executable file. No tenant API or scan request can choose this path. Host-agent verification uses its separately configured tls_probe_openssl setting.

Discovery first tries the usual Go handshake. For a failed direct TLS handshake, it may try the configured native TLS 1.3 client within the same endpoint deadline. Scan preview reports at most two connections per normalized endpoint when this option is enabled; neither client sends application data. Native fallback does not bypass a database STARTTLS negotiation. A failed handshake or unreadable leaf remains a partial-scan failure, while successful observations remain available.

Certificate CI export to ServiceNow

Export a certificate's public metadata to ServiceNow without giving the ownership reader write access. This is an explicit export, with its own operator-owned binding and credential reference; it does not turn on continuous synchronization.

The instance must provide the Unique Certificate class (cmdb_ci_certificate), a configured Discovery source choice and identification/reconciliation rules for certificate fingerprints. Use a credential authorized for the IRE write and the exact certificate-class readback. The product uses the ServiceNow IRE API, which applies the instance's reconciliation rules. HTTP 200 can still contain item errors, so each accepted item is followed by an independent Table API read.

{
  "itsm": {
    "servicenow": {
      "certificate_exports": [
        {
          "id": "certificate-writer",
          "tenant_id": "11111111-1111-1111-1111-111111111111",
          "enabled": true,
          "instance_url": "https://example.service-now.com",
          "token_ref": "secret://cmdb/export-writer",
          "data_source": "trstctl"
        }
      ]
    }
  }
}

No bindings are configured by default, and enabled defaults to false. An ID is unique within its tenant. instance_url is an HTTPS origin without credentials, path, query or fragment. token_ref uses secret:// for the tenant's own secret store or file: for an operator-owned credential file; values are opened into locked bytes for one worker attempt. A binding never inherits an ownership reader's or ticket writer's credential. Private instances additionally require allow_private_endpoint: true and exact private_egress_cidrs; DNS-at-dial, TLS, redirect and air-gap rules remain enforced. Changes take effect after restart; a disabled, removed or changed binding blocks retained work until a new export is requested under the current authority.

List bindings with trstctl-cli cmdb certificate-export-bindings list. Write an export request file with only binding_id and the inventory certificate_id, then run trstctl-cli cmdb certificate-exports create -f export.json. The returned operation ID is read with trstctl-cli cmdb certificate-exports show <operation-id>. The console offers the same form under Connect other tools → Permissions and reliable delivery. A queued result is not proof of a CMDB write; wait for verified and inspect the bound CI ID, fingerprint and verification time.

The worker fixes the class to cmdb_ci_certificate and names source objects by tenant UUID plus certificate fingerprint. It sends UTC certificate validity, subject/issuer distinguished names, key size, signature algorithm and the current recorded lifecycle state. It never sends a private key or invents issuer/root CI references from a distinguished name. A certificate with no retained DER/PEM must first use the verified DER-recovery path. An active certificate maps to issued, a revoked one to revoked, and a superseded one to retired; none of these labels claims independent deployment or relying-party revocation enforcement. If lifecycle state changes during the call, the worker retries under that state.

The export has its own bounded outbox.cmdb_exports pool. The optional bulkheads.outbox_cmdb_exports override has workers and queue limits and otherwise inherits the ordinary outbox limits. Verified receipts are immutable cmdb.certificate_export.verified audit events; queue retention cannot erase that evidence. Ownership sync remains a separate relay-only GET workflow. A real ServiceNow instance is required to qualify its class dictionary, ACLs, source choice and reconciliation behavior beyond the local TLS stand-in.

Named ticket-intake schedules

Use several ServiceNow tables or Jira projects without replacing another source's cursor. Each schedule is keyed by tenant, system and name. Existing schedules and clients that omit a name keep default. Names use 1-40 lowercase letters, digits, underscores or hyphens.

{
  "name": "service-requests",
  "system": "servicenow",
  "instance_url": "https://example.service-now.com",
  "token_ref": "secret://tickets/reader",
  "sn_table": "sc_req_item",
  "subject_field": "u_certificate_subject",
  "profile_field": "u_certificate_profile",
  "owner_field": "assigned_to.email",
  "interval_seconds": 3600,
  "enabled": true
}

Send this body to PUT /api/v1/issuance-requests/intake-schedule, or save it to a file and run trstctl-cli issuance-requests intake-schedule set -f intake.json. Create another name with its own table, or use system: jira and jira_project instead of sn_table. List all sources with intake-schedule list; read one with intake-schedule show --system servicenow --name service-requests. The Approvals console lists each named source and offers a source → mapping → review setup. Saving configures a read; only terminal relay coverage claims that tickets were read. Updates and removals refuse an incomplete sweep. A removal also waits for its last pending relay job, then stops future reads while retaining requests and audit evidence. Use intake-schedule delete --system servicenow --name <name>.

Owner mapping is optional and explicit. ServiceNow reference dot-walk fields are requested through sysparm_fields, as documented in its Table API guidance. For Jira, choose a scalar custom email field or a dotted field such as assignee.emailAddress; only its parent object is requested. Jira may omit email under its profile privacy rules. A display name, account ID, requester or reference ID never substitutes for an email. Exactly one existing owner in this tenant must match the mailbox without regard to case. No match or multiple matches leaves owner_id empty and the console shows the request as unassigned. No owner is created automatically.

Re-seeing a ticket never changes its existing request's owner or reopens a denied request. Correct mappings apply to new tickets. An unassigned request needs a new ticket or request that names a unique owner before it can be prepared for issuance. The normal approval and ownership-readiness checks still apply. Upgrade relays before using named schedules; a report that omits the name can settle only the legacy default schedule. The provider credential stays a per-attempt vault reference, and the control plane never dials either provider.