Configuration

trstctl resolves its configuration from, in increasing precedence: built-in defaults, an optional JSON config file (TRSTCTL_CONFIG_FILE), and environment variables. The configuration is validated on boot — a bad combination fails fast rather than starting half-configured.

Inspect the effective configuration at any time (credentials are redacted):

trstctl -check-config

Server

Transport encryption (TLS)

The control plane serves over TLS by default so no credential, token, or session ever travels in cleartext.

• internal (default) — the control plane presents a self-signed certificate it generates at startup, covering localhost, 127.0.0.1, the container hostname, and the Compose service name trstctl. Clients must trust it (or use curl -k); suitable for evaluation and tightly controlled internal or air-gapped use. Public deployments must use server.tls.mode=file with an operator-provided certificate chain from your CA; do not expose the eval self-signed certificate to public clients.
• file — the control plane presents an operator-provided certificate and key. Use this in production with a certificate from your CA. A missing or malformed file fails fast at startup rather than falling back to plaintext.
• disabled — plaintext HTTP. Local development only and mechanically bounded: startup fails unless TRSTCTL_DEV_ALLOW_PLAINTEXT=true and TRSTCTL_SERVER_ADDR is loopback-only (localhost, 127.0.0.1, or ::1). Production TLS termination should use server.tls.mode=file at trstctl or a TLS-terminating proxy in front of a TLS-enabled trstctl listener; disabled mode is not the production proxy pattern.

The control-plane↔signer channel (the private keys live in a separate, isolated process) is independent of this setting. The default (single-binary child mode, and external mode with signer.socket) is a peer-authenticated Unix domain socket — a 0600 socket in a 0700 directory, restricted to the signer's own uid via SO_PEERCRED on Linux — not a TLS channel. For a separately-hosted signer across nodes, set signer.mtls_address (with the signer.mtls_* certificate material): the control plane then reaches the signer over mTLS — TLS 1.3, AEAD-only, with the control plane and the signer each pinning the other's certificate (an untrusted or merely CA-signed-but-unpinned peer is rejected, fail-closed). Exactly one of signer.socket or signer.mtls_address is used in external mode; a partial mTLS block fails closed at startup.

Datastores

trstctl stores its read state in PostgreSQL (the source-of-truth event log lives in NATS JetStream). PostgreSQL is the datastore in every deployment mode — there is no SQLite path.

!!! important "Datastores: bundled single-node for eval, external for production" The serving binary (trstctl, via server.Run) can run a single-node eval stack: bundled PostgreSQL (TRSTCTL_POSTGRES_MODE=bundled, the default — the binary starts and supervises an embedded single-node Postgres with data under TRSTCTL_POSTGRES_DATA_DIR on TRSTCTL_POSTGRES_PORT, default 5432) and embedded NATS (TRSTCTL_NATS_MODE=embedded, the default — in-process file-backed JetStream). Bundled PostgreSQL is available only for host archives with committed runtime pins in deploy/supply-chain/embedded-postgres.json (summarized in Supply chain): currently linux-amd64, linux-arm64v8, and darwin-arm64v8. It downloads that pinned PostgreSQL runtime once on first use, verifies the cached archive before execution, and fails closed if the host archive is unsupported, unpinned, or hash-mismatched. For production, use external for both: TRSTCTL_POSTGRES_MODE=external with TRSTCTL_POSTGRES_DSN and TRSTCTL_NATS_MODE=external with TRSTCTL_NATS_URL, which the Compose stack and Helm chart wire up. There is no silently-failing default: an invalid mode — or external without a DSN — fails fast at startup. (External mode never downloads anything. --migrate / --backup target a managed datastore and require external.)

External datastores

To point trstctl at managed PostgreSQL and NATS, switch both to external mode and supply their connection strings:

export TRSTCTL_POSTGRES_MODE=external
export TRSTCTL_POSTGRES_DSN='postgres://user:pass@db.internal:5432/trstctl?sslmode=require'
export TRSTCTL_NATS_MODE=external
export TRSTCTL_NATS_URL='nats://nats.internal:4222'
export TRSTCTL_NATS_REPLICAS=3

When a mode is external, its connection string is mandatory; trstctl refuses to start without it. External NATS also refuses to serve under-replicated: the event stream defaults to three replicas, startup fails on a non-clustered single NATS server, and /readyz reports degraded if the observed stream later has fewer replicas than configured. The Docker Compose eval stack uses the same external code path but explicitly sets TRSTCTL_NATS_REPLICAS=1 and TRSTCTL_NATS_ALLOW_SINGLE_REPLICA=true; keep that opt-in out of production.

Schema migrations

trstctl embeds its PostgreSQL schema as versioned SQL migrations and applies them itself; there is no separate migration binary.

trstctl --migrate applies pending migrations under the advisory lock and exits; trstctl --migrate-status lists the pending plan (dry run) and exits. See Database migrations & upgrades for the full upgrade/rollback runbook.

High availability (leader election and snapshots)

trstctl is safe to run as more than one control-plane replica against one shared external PostgreSQL and NATS (RESIL-002 / RESIL-004). With the defaults, a single replica behaves exactly as before; adding replicas needs no extra configuration beyond pointing them at the same datastores.

Cross-cluster federation

Federation is disabled by default. When enabled on a passive cluster, the leader worker imports a peer's event log into the local event log, advances a durable peer cursor, and projects the imported events locally. The passive region therefore serves from its own PostgreSQL and NATS after failover; it does not read the primary region's PostgreSQL tables.

JSON config can declare multiple peers under federation.peers; the environment overlay above configures one common peer. Keep ha.leader_election on so one replica owns imports while all replicas serve the replicated read state.

Lifecycle

How far ahead of expiry trstctl renews and alerts. Values are Go durations.

Native connector and external-CA assembly

The 24 native deployment connectors and 14 external-CA drivers are compiled into trstctl, but start deny-by-default: the operator chooses the exact integrations to construct. The connector allowlist and its shared network policy — which native connectors may be constructed, the shared HTTP timeout, private-CIDR grants, and the loopback-only insecure-HTTP escape hatch — are flattened into the environment variables below.

The tenant-bound structured bindings — connectors.local_profiles (local execution profiles), connectors.right_size (tenant-to-endpoint bindings), and every external_cas entry — remain JSON/YAML config-file only, because they carry tenant/provider associations where a parallel-list typo could cross a security boundary.

A local connector needs both an enabled driver and an operator-owned execution profile. Tenant target JSON can then select that profile, but cannot invent a command or escape its allowed roots:

{
  "connectors": {
    "enabled": ["nginx", "f5"],
    "http_timeout": "15s",
    "allow_private_cidrs": ["10.40.0.0/16"],
    "local_profiles": {
      "nginx-prod": {
        "allowed_roots": ["/etc/nginx/tls"],
        "actions": [{
          "logical_name": "nginx",
          "command": "/usr/sbin/nginx",
          "pass_args": true,
          "timeout": "15s"
        }]
      }
    },
    "right_size": [{
      "tenant_id": "11111111-1111-4111-8111-111111111111",
      "connector": "least-privilege",
      "endpoint": "https://entitlements.internal",
      "token_ref": "secret://connectors/right-size-token"
    }]
  }
}

Local action commands must be absolute, executable, regular non-symlink files. Unix shell interpreters are rejected. The one native-administration exception is IIS PowerShell: its profile must pin the complete connector logical_args, set a separate complete operator-owned args list, and leave pass_args false. The runtime compares the connector request with that fixed logical argv and executes only the fixed operator argv, so target data never becomes PowerShell source text.

connectors.right_size binds one tenant plus the playbook's connector name to an operator endpoint and a same-tenant encrypted-secret reference. The durable connector.right_size worker sends an idempotent PATCH, verifies the vendor's mutation receipt, performs an authenticated GET readback, and only then records a delivered connector receipt. Store the referenced token through the tenant secrets surface (secrets.enable_api=true) before enabling the binding. An unconfigured tenant/connector pair fails closed.

External-CA credentials use file:/absolute/path references. Files are loaded into locked memory for one outbox attempt and wiped afterward. Network policy, private CIDRs, custom roots, and mTLS identities are operator-owned:

{
  "external_cas": [{
    "id": "aws-pca-prod",
    "type": "awspca",
    "name": "Production AWS Private CA",
    "endpoint": "https://acm-pca.us-east-1.amazonaws.com",
    "region": "us-east-1",
    "certificate_authority_arn": "arn:aws:acm-pca:us-east-1:123456789012:certificate-authority/UUID",
    "access_key_id": "AKIA...",
    "secret_access_key_ref": "file:/run/secrets/aws-pca-secret",
    "network": {"timeout": "15s"}
  }, {
    "id": "azure-managed-hsm-ca",
    "type": "azurekv",
    "name": "Azure Managed HSM Issuing CA",
    "tenant_id": "11111111-1111-4111-8111-111111111111",
    "endpoint": "https://production.managedhsm.azure.net",
    "managed_key_ref": "https://production.managedhsm.azure.net/keys/issuing-ca/0123456789abcdef",
    "ca_cert_file": "/etc/trstctl/azure-issuing-ca-chain.pem",
    "network": {"timeout": "15s"}
  }, {
    "id": "entrust-prod",
    "type": "entrust",
    "name": "Entrust CA Gateway",
    "endpoint": "https://entrust-ca.internal",
    "ca_id": "production-ca",
    "network": {
      "root_ca_file": "/etc/trstctl/entrust-server-ca.pem",
      "client_cert_file": "/etc/trstctl/entrust-client.pem",
      "client_key_file": "/run/secrets/entrust-client-key.pem",
      "server_name": "entrust-ca.internal",
      "timeout": "15s"
    }
  }, {
    "id": "letsencrypt-prod",
    "type": "letsencrypt",
    "name": "Let's Encrypt",
    "directory_url": "https://acme-v02.api.letsencrypt.org/directory"
  }, {
    "id": "shell-ca-prod",
    "type": "shellca",
    "name": "Isolated Shell CA",
    "command": "/usr/local/libexec/trstctl-shell-ca",
    "args": ["--profile", "production"],
    "env_refs": {"CA_TOKEN": "file:/run/secrets/shell-ca-token"},
    "network": {"timeout": "15s"}
  }]
}

External-CA endpoints require HTTPS. The separate network.allow_insecure_http escape hatch is accepted only for an HTTP localhost, 127/8, or ::1 development emulator; the runtime client also refuses any non-loopback resolution or cross-origin/scheme-changing redirect. allow_private_endpoint and private CIDR grants never authorize plaintext.

The Azure CA block carries only routing and public trust material. tenant_id binds the authority to one tenant, managed_key_ref is the opaque HTTPS key id returned by that tenant's managed-key lifecycle, and ca_cert_file is the public issuer chain whose leaf public key must match the managed key. Do not put an Azure bearer token, key_name, or key_version in external_cas: Azure credentials and private-key operations live in the isolated signer's managed_keys backend.

Let's Encrypt account JWS signatures also cross the signer transport. The control plane derives a stable account handle from the external-CA id and directory URL, but it receives only the public key and signatures; the ECDSA account private key never enters the HTTP process.

Entrust always requires the complete network mTLS identity shown above. The client pins root_ca_file, verifies server_name, and presents the paired client certificate/key. Missing, partial, or untrusted client material fails before an Entrust enrollment request is sent; plaintext HTTP is not an Entrust mode.

For shellca, each env_refs key is a logical credential name, not a secret-valued environment variable. The child sees CA_TOKEN_FD=<number> and reads the exact credential bytes from that inherited anonymous pipe. It never receives CA_TOKEN, and trstctl consumes and wipes the byte buffer after the one child attempt. No credential pathname or filesystem-backed secret copy is created for the child.

Supported type values are adcs, awspca, azurekv, digicert, ejbca, entrust, gcpcas, globalsign, letsencrypt, sectigo, shellca, smallstep, vaultpki, and venafi. trstctl -check-config validates every provider's required fields before the server accepts traffic. The exact connector target schemas are listed in Deployment connectors.

Notifications

Notification channels are off until an operator configures them. When enabled, lifecycle expiry alerts are written to the notification.* outbox first, then delivered by the registered channel workers. Tenant-scoped routing policy authoring and channel-test delivery are served from the console; channel secrets remain in operator-managed secret references or files.

PagerDuty and OpsGenie credentials are copied into locked, non-dumpable memory at startup and wiped on shutdown. Both integrations send deterministic vendor idempotency identifiers, require the vendor's exact acceptance receipt, and redact remote error bodies. *_ALLOW_INSECURE_HTTP exists only for a loopback development emulator; even when set, it cannot enable cleartext delivery to a non-loopback host.

Code signing

The shipped code-signing routes stay fail-closed until code_signing.enabled is true. Tenant-to-key and tenant-to-OIDC associations are structured JSON only: putting these parallel lists in environment variables would make it too easy to attach one tenant's trust to another tenant. A minimal key-backed plus GitHub Actions keyless configuration looks like this:

{
  "code_signing": {
    "enabled": true,
    "keys": [{
      "tenant_id": "tenant-acme",
      "id": "release-key",
      "handle": "acme-release-code-signing",
      "algorithm": "ecdsa-p256",
      "create_if_missing": true
    }],
    "github_oidc_tenants": [{
      "tenant_id": "tenant-acme",
      "issuer": "https://token.actions.githubusercontent.com",
      "audience": "sigstore",
      "jwks_file": "/run/trstctl/github-actions-jwks.json",
      "allowed_owners": ["acme"]
    }],
    "ephemeral_algorithm": "ecdsa-p256",
    "rekor": {
      "endpoint": "https://rekor.example.com/api/v1/log/entries",
      "timeout": "10s",
      "log_public_key_file": "/run/trstctl/rekor-log-public-key.pem"
    }
  }
}

create_if_missing provisions a code-sign-purpose-constrained handle inside the separate signer process; the control plane never receives the private key. Keyless requests also create and destroy their one-use key inside that signer. Pin the Rekor log public key from the Rekor operator's authenticated distribution channel. Startup fails if this trust anchor is absent or malformed, and every outbox delivery verifies the returned signed-entry timestamp against it before acknowledging the row.

Code signing uses the shared live policy controls rather than a decorative, code-only allow list. With ca.policy.enabled=true, OPA receives action code_sign, the authenticated actor, the configured key ID (or keyless:<verified-san>), and attrs.digest_sha256; evaluation errors and policy-pool saturation fail closed. With ca.policy.require_approval=true, a denied response returns approval_required:codesign:<sha256>. A distinct certs:issue approver posts {"action":"sign"} to /api/v1/identities/{that-resource}/approvals, after which the requester retries the same signing tuple with a new Idempotency-Key. The resource hash binds tenant, authenticated requester, key/keyless identity, and exact digest.

TRSTCTL_CODE_SIGNING_REKOR_ALLOW_INSECURE_HTTP is accepted only for a loopback development emulator. It cannot relax transport security for a remote Rekor service.

Telemetry

Telemetry is off by default and never sends anything unless you opt in. When enabled, it sends only coarse, anonymized, non-PII data.

See Telemetry for exactly what is and is not collected.

Air-gap and outbound egress

Air-gap mode arms the no-phone-home outbound HTTP(S) guard. It is off by default; when enabled, public destinations are blocked unless you explicitly allow a host or CIDR. It is intended for disconnected networks and is paired with the Helm values-airgap.yaml overlay described in Air-gapped install.

When TRSTCTL_AIRGAP_ENABLED=true, trstctl rejects TRSTCTL_TELEMETRY_ENABLED=true and TRSTCTL_AI_MODEL_MODE=cloud at startup. Local OTLP collectors and local AI runtimes are still allowed when their hosts or CIDRs are private or explicitly allowlisted.

ITSM and ServiceNow bindings

ServiceNow ticket creation is fail-closed until an operator pre-registers the destination URL and credential reference. Requests to /api/v1/itsm/servicenow/tickets and ServiceNow response-dispatch destinations must match that binding exactly; callers cannot send the ServiceNow token to an arbitrary instance_url or opt into a private endpoint unless the binding permits it.

JSON config supports the same policy as itsm.servicenow.bindings[] with instance_url, token_ref, allow_private_endpoint, and private_egress_cidrs. Private endpoint requests also require the caller to hold the dedicated egress:private permission.

OpenTelemetry export

OTLP export is off by default and sends data only to the collector endpoint you configure. It is separate from product telemetry: use it to feed your own OpenTelemetry Collector, Splunk, Datadog, or SIEM pipeline with served HTTP traces and audit-event log records.

In air-gap mode, point OTLP at an operator-owned collector on a private address or add the collector host to TRSTCTL_AIRGAP_ALLOW_HOSTS; public collector SaaS endpoints are blocked by the egress guard.

Audit

The audit trail is a projection of the event log; these settings govern its evidence export and retention policy. See Audit trail & compliance for the trust model and what trstctl enables vs. what you must operate.

The audit query (/api/v1/audit/events) and signed export (/api/v1/audit/export) endpoints are wired into the serving binary, so they return real data — not an error — out of the box. Protect the signing key file and back it up; distribute its public half to auditors out of band.

Privacy Retention

Non-audit personal data is retained by class, then pseudonymized after the operational need ends. This is separate from audit retention: the immutable event trail remains the source of truth, while tenant read surfaces stop carrying stale names, emails, subjects, SANs, comments, profile authors, approval actors, and free-form evidence. The worker emits privacy.retention.enforced and projects the anonymization from that event, so rebuilds replay the same result.

Subject erasure also pseudonymizes matching raw subject bytes in the hot JetStream event store. The control plane records privacy.subject.erased, builds a sequence-preserving replacement generation with the erased subject changed to its tenant-bound erased:<subject_ref> placeholder, and signs the old/new continuity evidence before switching authority. The old generation remains authoritative if the operation stops before that switch; only after the signed replacement is active are its superseded subject bytes securely scrubbed. Replay, projection tailing, retention, and backups share a history barrier with that cutover. This keeps future hot-log replay, audit queries, and full backups taken after the erasure from carrying the raw subject without ever purging and rebuilding the sole authoritative stream. Backups and signed audit archives created before the erasure are not rewritten automatically. Record the outcome of your legal-hold, WORM, backup-deletion, or cryptographic-shredding procedure with POST /api/v1/privacy/archive-erasure-attestations; inspect the tenant evidence ledger with GET /api/v1/privacy/archive-erasure-attestations or trstctl privacy archives attest/list. The attestation event stores subject_ref and redacted evidence refs, not the raw subject.

Operators can trigger and inspect the same served path with POST /api/v1/privacy/retention-runs, GET /api/v1/privacy/retention-runs, or the matching trstctl privacy retention run/list CLI commands.

Browser SSO

Browser sign-on is optional. Scoped API tokens still work when browser sign-on is off. When it is on, each verified OIDC, SAML, or LDAP / Active Directory user must map to exactly one trstctl tenant by subject, tenant claim, directory group, or an explicit single-tenant fallback. Missing mappings fail the login closed instead of silently dropping a user into the wrong tenant.

SCIM provisioning

SCIM 2.0 provisioning is optional and separate from browser sign-on. Enable it when your identity provider should push users and groups into trstctl instead of an operator maintaining role membership by hand. The served endpoint is /scim/v2: IdPs use POST /scim/v2/Users, PATCH /scim/v2/Users/{id}, POST /scim/v2/Groups, and PATCH /scim/v2/Groups/{id}.

Each SCIM bearer token is bound to exactly one tenant in config. trstctl reads the raw token from auth.scim.tokens[].token_file, hashes it at startup, wipes the raw bytes, and keeps only the hash. The token selects the tenant for every SCIM request; tenant ids in SCIM payloads are ignored. Groups map to configured RBAC role names, so an IdP group with display name viewer grants the built-in viewer role to its members.

Example multi-token SCIM config:

auth:
  scim:
    enabled: true
    tokens:
      - name: okta-payments
        tenant_id: 22222222-2222-2222-2222-222222222222
        token_file: /etc/trstctl/scim/okta-payments.token
      - name: entra-platform
        tenant_id: 33333333-3333-3333-3333-333333333333
        token_file: /etc/trstctl/scim/entra-platform.token

Example SAML config:

auth:
  saml:
    enabled: true
    entity_id: https://trstctl.example.com/auth/saml/metadata
    metadata_url: https://trstctl.example.com/auth/saml/metadata
    acs_url: https://trstctl.example.com/auth/saml/acs
    idp_metadata_file: /etc/trstctl/idp-metadata.xml
    session_secret_file: /var/lib/trstctl/saml-session.secret
    tenant_claim: tenant
    groups_claim: groups
    tenant_mappings:
      - subject: alice@example.com
        tenant_id: 11111111-1111-1111-1111-111111111111
        roles: [admin]

Example LDAP / Active Directory config:

auth:
  ldap:
    enabled: true
    url: ldaps://ad.example.com:636
    bind_dn: cn=trstctl-reader,ou=service-accounts,dc=example,dc=com
    bind_password_file: /etc/trstctl/ldap-bind.secret
    user_search_base_dn: ou=people,dc=example,dc=com
    user_filter: "(sAMAccountName={username})"
    group_search_base_dn: ou=groups,dc=example,dc=com
    group_filter: "(member={user_dn})"
    group_name_attribute: cn
    email_attribute: mail
    session_secret_file: /var/lib/trstctl/ldap-session.secret
    tenant_mappings:
      - group: payments-trstctl-admins
        tenant_id: 22222222-2222-2222-2222-222222222222
        roles: [admin]

ABAC deny overlay

ABAC is an optional deny-only overlay on top of RBAC. RBAC must grant the permission first; the ABAC Rego module can then block the request using request attributes, identity tags, operator-provided environment state, and time. This is useful for rules such as "prod certs may issue only during change windows" or "break-glass actions may run only from the platform project."

The module must declare package trstctl.abac, define boolean deny, and may define string reason. Bad Rego fails startup. Runtime evaluation errors deny the request, and policy-worker saturation returns 503 instead of allowing.

In a config file, the keys are auth.abac.enabled, auth.abac.module, and auth.abac.environment.

Example ABAC config:

auth:
  abac:
    enabled: true
    environment:
      change_window: "false"
    module: |
      package trstctl.abac
      default deny := false
      default reason := ""

      deny if {
        input.permission == "certs:issue"
        input.resource.env == "prod"
        input.env.change_window != "true"
      }

      reason := "prod certificates may issue only during a change window" if {
        deny
      }

Break-glass lifecycle and reconciliation

Break-glass emergency issuance has two served paths. POST /api/v1/breakglass/issue performs online m-of-n emergency issuance when a signer-backed break-glass issuer is configured. First open the exact CSR/reason/TTL ceremony at POST /api/v1/breakglass/issue-ceremonies; then distinct configured operators approve that ceremony through the shared authenticated CA-approval route. The execution request carries ceremony_id and no approver names. The server derives quorum only from immutable ca.ceremony.approved event actors, consumes the ceremony once, returns a self-verifying bundle, and records breakglass.issued before responding. The same signer-backed lifecycle exposes purpose-bound rotation and target-CA cross-sign ceremony/execution pairs under /api/v1/breakglass/*-ceremonies, /api/v1/breakglass/rotate, and /api/v1/breakglass/cross-sign. POST /api/v1/breakglass/reconcile handles recovery after an offline ceremony: it accepts signed offline bundles, verifies them against trust anchors pinned in process config, and records verified facts as breakglass.issued events in the hash-chained audit log. Requests cannot supply their own verifier keys.

breakglass.enabled pins reconciliation verifier material. The files may be DER, or PEM with CERTIFICATE and PUBLIC KEY blocks, and startup fails closed when either is missing or unreadable. breakglass.online_enabled additionally requires one tenant, an already-persisted dual-control signer handle whose public key matches both files, a distinct operator roster, and a threshold of at least two. Reconciliation- only deployments do not need the online fields.

Example break-glass reconciliation config:

breakglass:
  enabled: true
  ca_cert_file: /etc/trstctl/breakglass-ca.pem
  public_key_file: /etc/trstctl/breakglass-public-key.pem

Online lifecycle adds the custody and roster fields:

breakglass:
  enabled: true
  online_enabled: true
  ca_cert_file: /etc/trstctl/breakglass-ca.pem
  public_key_file: /etc/trstctl/breakglass-public-key.pem
  tenant_id: 10000000-0000-4000-8000-000000000001
  signer_handle: recovery-breakglass-ca
  operators: [recovery-operator-a, recovery-operator-b, recovery-operator-c]
  threshold: 2

To create that handle without a test-only or library-only path, leave online_enabled=false, use trstctl-cli ca ceremonies start plus distinct ca ceremonies approve calls, and finish with ca authorities create-root. Save the returned certificate_pem and signer_handle; derive the public-key file from the certificate (openssl x509 -pubkey -noout), set the online fields above, and restart. Startup proves the certificate, derived public key, and persisted signer handle all name the same key before it exposes any online break-glass route.

Secrets (credentials at rest)

Upstream CA and connector credentials — API keys, passwords, client secrets — are stored encrypted at rest using envelope encryption (R3.1): a fresh random data-encryption key (DEK) encrypts each credential with AES-256-GCM, and the key-encryption key (KEK) wraps the DEK. Only ciphertext is ever persisted; the plaintext never appears in the database, in config dumps, or in logs. The cryptography lives behind the platform's single crypto boundary.

Every default-binary idempotency result is outer-sealed through the same tenant-aware crypto resolver before PostgreSQL retains it. A legacy tenant uses the deployment KEK explicitly. An opted-in tenant names exactly one separately provisioned wrapper and never trial-decrypts or falls back to the deployment KEK. The local wrapper file is opened only inside a tenant-scoped shared database fence; its domain key lives in locked memory for that callback and is then wiped. Before the HTTP mutation surface becomes ready, startup drains historical raw-v0 and sealed-dynamic-lease-v1 cache rows in key-ordered, RLS-scoped batches into the authenticated sealed-row-v1 envelope. Each replacement is a compare-and-swap over tenant, key, binding, codec, and prior bytes. A crash therefore resumes safely, while a changed row or unavailable tenant wrapper fails readiness without logging or embedding the result bytes in the error. The runtime reader rejects both legacy codecs after this drain. When the fleet-ready assertion is set, startup also locks the table, proves every completed row is a CSL sealed container, changes the database default, and validates a permanent sealed-only constraint before readiness. This is a one-way compatibility decision: drain or stop all old writers first.

{
  "secrets": {
    "kek_file": "/run/secrets/deployment-kek",
    "tenant_seal_local_wrappers": [
      {"id": "operator-a", "file": "/run/custody/operator-a-wrapper"},
      {"id": "operator-b", "file": "/run/custody/operator-b-wrapper"}
    ]
  }
}

Wrapper files are operator authority, not ordinary application data. Provision them separately with restrictive ownership/mode before configuring a tenant. Missing, corrupt, or mismatched custody fails that tenant closed while other tenants continue. This local mode makes no outbound call and works air-gapped.

Dynamic-secret providers and outbound sync targets use structured JSON/YAML because each entry is bound to exactly one tenant. Authority-bearing values are references, not inline strings: file:/absolute/path loads an operator-owned 0600 file, while secret://path opens that tenant's encrypted secret-store row for one outbox attempt. Both are copied into locked, non-dumpable memory and destroyed after the provider call.

{
  "secrets": {"enable_api": true},
  "secret_integrations": {
    "dynamic_providers": [{
      "tenant_id": "11111111-1111-4111-8111-111111111111",
      "id": "orders-postgres",
      "type": "postgresql",
      "admin_dsn_ref": "file:/run/secrets/orders-postgres-admin-dsn",
      "database": "orders",
      "schema": "public",
      "allowed_roles": ["reader"],
      "max_ttl": "1h",
      "username_prefix": "trstctl_orders"
    }],
    "sync_targets": [{
      "tenant_id": "11111111-1111-4111-8111-111111111111",
      "id": "orders-github",
      "type": "github-actions",
      "endpoint": "https://api.github.com",
      "owner": "example",
      "repo": "orders",
      "token_ref": "secret://integrations/github-actions-token"
    }]
  }
}

Dynamic type values are postgresql, mysql, mongodb, aws-iam, gcp-iam, azure-entra, kubernetes, and redis. Sync type values are aws-secrets-manager, gcp-secret-manager, azure-key-vault, github-actions, gitlab-ci, vercel, generic-ci-json, kubernetes-secrets, terraform-cloud-opentofu, and vault-kv-v2. Startup validates each provider's required native fields, role bindings, endpoint scheme, private-egress CIDRs, and credential-reference form before accepting traffic. An absent tenant target or provider fails its served mutation closed; it never falls back to another tenant or to a test registry.

AWS, GCP, and Azure sync targets can explicitly replace their static target credential with tenant-owned workload identity. Set exactly the provider switch (aws_workload_identity, gcp_workload_identity, or azure_workload_identity) on the matching target. The switch is false by default and forbids the corresponding static credential fields when true. The workload proof itself stays behind a file: or tenant-scoped secret:// reference. Only the bounded secret-sync outbox worker opens and validates that proof, POSTs the provider exchange, caches the locked short-lived token until its refresh boundary, and feeds it into the existing hand-written target client. The request handler never performs that egress. Air-gapped mode records offline_disabled, fails that queued delivery once, and makes no token or target request.

Azure uses an Entra federated credential that accepts the already validated OIDC proof directly as the OAuth client_assertion. This is simpler than a certificate-signed assertion: it adds no private key or certificate custody. Create the matching federated-credential binding on the Entra application, then configure the Key Vault target:

secret_integrations:
  sync_targets:
    - tenant_id: 11111111-1111-4111-8111-111111111111
      id: payments-azure-key-vault
      type: azure-key-vault
      endpoint: https://payments.vault.azure.net
      azure_workload_identity: true
      # Optional. When omitted, the source's azure_tenant_id selects:
      # https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
      # workload_identity_endpoint: https://login.microsoftonline.com/.../oauth2/v2.0/token

Create the tenant-scoped source through the console or trstctl-cli secrets syncs workload-identities create --body-file source.json. For Azure, the request sets provider: "azure", azure_tenant_id, the Entra application client_id, and an allowed Key Vault target_scope such as https://vault.azure.net/.default. It also binds the exact OIDC audience and subject, target_id, allowed remote-key prefixes, workload_proof_ref, and a JWT/JWKS trust_source_id. Those routing fields are stored under PostgreSQL RLS; proof bytes and minted bearer tokens are never stored in the source row, event, job, or API response.

Every dynamic-provider and sync-target HTTP endpoint must use HTTPS in production. allow_private_endpoint grants a named private destination; it never grants plaintext. Local emulators may set allow_insecure_loopback: true, but only beside an http:// endpoint whose host is exactly localhost, an address in 127.0.0.0/8, or ::1. The runtime client resolves and dials loopback only, so changing DNS or adding a private CIDR cannot stretch this development switch to a LAN, VPC, metadata service, or public host.

Machine-auth methods beyond the HMAC token are configured in the JSON/YAML config file under secrets.machine_auth. Each entry names one method: kubernetes, aws-iam, gcp, azure, oidc, or jwt. JWT-family methods require audience plus jwks_file or jwks_json, and must set either tenant_claim (credential-bound tenancy) or tenant_id (tenant-pinned config). AWS IAM must set tenant_id and allowed_accounts or allowed_arns because STS does not carry a trstctl tenant claim.

Treat the credential-store KEK like the audit signing key: protect it and back it up (a lost KEK means sealed credentials cannot be opened) with the same care described in the disaster-recovery runbook. This credential-store KEK is still a local key file. Do not confuse it with Helm externalKMS, which applies to the signer's CA key-store DEK wrapping described in Signer topology and CA custody. On reload, local KEK, auth-secret, and session-secret files are accepted only if they are regular files, not symlinks, owned by the process user with 0600-or-stricter permissions or mounted as root-owned Kubernetes Secret files readable by the pod's fsGroup, and all parent directories reject group/world writes. Unsafe restored files fail startup instead of silently weakening key custody.

Backup

Event-log backups (trstctl --backup) are always integrity-protected (a SHA-256 trailer, plus an HMAC derived from TRSTCTL_AUDIT_SIGNING_KEY_FILE when one is configured). They require the live deployment's external PostgreSQL DSN as well as external NATS: PostgreSQL holds a shared history-generation barrier for the complete export, preventing a concurrent privacy rewrite from changing which JetStream generation is authoritative mid-backup. A full backup (trstctl --full-backup-dir) additionally captures operational secrets — the audit signing key, the signer authorization secret, and the sealed signer key store — so production full backups require an operator-held encryption key.

See the disaster-recovery runbook for the full backup/restore procedure, including what each artifact covers and how to restore a signer host.

License

trstctl ships as a single open-core binary. Enterprise-tier features (managed keys, PCAS, HA support, FIPS artifact posture, remediation, PQC, governance, agent delegation, reconciliation, and verifiable decommission — the full Enterprise row in Editions) unlock through an offline, no-phone-home license check: the file is verified locally against public keys baked into the binary at release time. No configured file means Community edition; a corrupt or untrusted file fails startup loudly; an expired file still loads and walks a grace ladder so licensed read paths stay observable.

The related TRSTCTL_PCAS_* variable surface (about two dozen settings covering delegation, recovery, federation, KEM, checkpoints, monitors, and retirement for patent-covered credential algorithm succession) is off by default and gated by this same license check; it is documented together with the rest of the PCAS material in PCAS operations rather than duplicated here.

Conditional managed-key adapters (six served providers)

The managed-key lifecycle is off by default and requires an Enterprise license plus startup configuration. AWS KMS, Azure Key Vault / Managed HSM, GCP Cloud KMS, PKCS#11, TPM 2.0, and YubiHSM 2 are all required/census-served through the shipped cgo HSM signer artifact. “Conditional” means the route is inert until the operator selects and provisions one provider; it no longer means library-only. The control plane records a tenant event and PostgreSQL outbox command, while the isolated signer constructs the provider and performs the private operation. The control-plane process never receives provider credentials.

Development-only LocalStack shape (not conformance or shipped-runtime proof):

export TRSTCTL_MANAGED_KEYS_ENABLED=true
export TRSTCTL_MANAGED_KEYS_PROVIDER=aws
export TRSTCTL_MANAGED_KEYS_AWS_REGION=us-east-1
export TRSTCTL_MANAGED_KEYS_AWS_ENDPOINT=http://127.0.0.1:4566
export TRSTCTL_MANAGED_KEYS_AWS_ALLOW_INSECURE_LOOPBACK=true
export TRSTCTL_MANAGED_KEYS_AWS_ACCESS_KEY_ID=test
printf '%s' test > /tmp/localstack-kms-secret
chmod 0600 /tmp/localstack-kms-secret
export TRSTCTL_MANAGED_KEYS_AWS_SECRET_ACCESS_KEY_FILE=/tmp/localstack-kms-secret

Private endpoint CIDR grants control where HTTPS may go. They never weaken TLS. The three ALLOW_INSECURE_LOOPBACK switches are deliberately separate and accept only loopback HTTP for a same-host emulator.

Example production shape:

managed_keys:
  enabled: true
  provider: aws
  aws:
    region: us-east-1
    access_key_id: AKIA...
    secret_access_key_file: /etc/trstctl/aws-kms-secret-access-key

Example Azure Key Vault HSM shape:

managed_keys:
  enabled: true
  provider: azure-key-vault
  azure:
    vault_url: https://trstctl-prod.managedhsm.azure.net
    bearer_token_file: /etc/trstctl/azure-kv-token

Example GCP Cloud KMS shape:

managed_keys:
  enabled: true
  provider: gcp-kms
  gcp:
    parent: projects/prod/locations/us/keyRings/trstctl
    bearer_token_file: /etc/trstctl/gcp-kms-token

Example PKCS#11 HSM shape:

managed_keys:
  enabled: true
  provider: pkcs11
  pkcs11:
    module_path: /usr/lib/softhsm/libsofthsm2.so
    token_label: trstctl-prod
    user_pin_file: /etc/trstctl/pkcs11-user-pin
    key_label_prefix: trstctl-ca

Example TPM 2.0 shape:

managed_keys:
  enabled: true
  provider: tpm2
  tpm2:
    path: /dev/tpmrm0
    owner_auth_file: /etc/trstctl/tpm-owner-auth
    key_auth_file: /etc/trstctl/tpm-key-auth
    persistent_handle_base: 2164326400 # 0x81010000

Example YubiHSM 2 shape:

managed_keys:
  enabled: true
  provider: yubihsm2
  yubihsm2:
    module_path: /usr/lib/yubihsm_pkcs11.so
    token_label: trstctl-prod
    user_pin_file: /etc/trstctl/yubihsm-auth
    key_label_prefix: trstctl-ca

Static no-cgo signer builds fail closed if pkcs11 or yubihsm2 is selected. Use the published artifact built by deploy/docker/Dockerfile.signer-hsm; its release profile enables cgo and installs /usr/local/bin/trstctl-signer-hsm.

When the licensed attach seam succeeds, operators with keys:write can exercise POST /api/v1/managed-keys and the rotate/revoke/zeroize API and CLI shapes. Before each destructive action, two different principals with keys:approve record the exact opaque key_id and rotate, revoke, or zeroize action through POST /api/v1/managed-keys/approvals; the requester cannot self-approve. Requests require Idempotency-Key; lifecycle events omit private bytes; tenant projections use PostgreSQL RLS; and provider work is delivered by the durable outbox to the separate signer. The signer writes an fsync-backed operation intent before provider I/O. Every shipped provider also carries that identity into provider state: an atomic AWS tag, deterministic Azure/GCP resource identity, deterministic PKCS#11 CKA_ID (including YubiHSM), or a full-width TPM public tag plus deterministic handle probing. If a provider commits and the signer dies before journaling the response, restart finds the same effect; revoke/zeroize likewise confirm terminal provider state rather than blindly repeating a mutation. Managed-key outbox circuits are partitioned by provider and current key (or operation ID before a key exists), so one unavailable key does not stall every managed-key command. All six exact backend census rows are SERVED + REQUIRED. For an externally deployed signer, mount the same file-backed provider descriptor and pass it with --managed-keys-config plus the signed --license; never copy a credential into argv or an inline JSON field.

Signer topology and CA custody

The private-key operations run in a separate, sacred process, so the CA keys never live in the API process. Its issuing CA key is persisted, sealed at rest (R3.2) so a restart preserves the CA instead of silently rotating it. The signer can run two ways:

In Helm deployments, the signer key store uses local KEK custody by default: kek.existingSecret or eval-only kek.generate=true mounts /etc/trstctl/kek/kek.bin and the chart passes --kek to trstctl-signer. Regulated deployments can instead set:

externalKMS:
  enabled: true
  provider: awskms        # awskms | gcpkms | azurekv | pkcs11
  keyRef: arn:aws:kms:us-east-1:111122223333:key/trstctl-signer
  wrapCommand: /usr/local/bin/trstctl-kms-wrap
  timeout: 10s

With externalKMS.enabled=true, the chart passes --kms-provider, --kms-key-ref, --kms-wrap-command, and --kms-timeout to the signer and does not mount the local KEK Secret. The signer invokes the adapter without a shell as <wrapCommand> wrap|unwrap <provider> <keyRef>, with DEK bytes only on stdin/stdout. Missing provider/keyRef/command values, unsupported provider names, or a relative command path fail at template time.

Back up the sealed key store, the signer custody input, and the CA cert together (the CA-key recovery set) per the disaster-recovery runbook. For local-KEK mode that custody input is the signer KEK Secret; for externalKMS mode it is access to the same provider keyRef plus the wrapper adapter and provider credentials. The docker-compose.yml runs the signer as its own service in external mode.

Regulated CA governance mode

The individual issuance controls — the OPA/Rego policy gate, four-eyes dual control, a bound default certificate profile, revocation publication, and FIPS — can each be enabled on their own. For a compliance deployment that is error-prone: a single missing control silently weakens the posture. ca.governance_mode=regulated is the one coherent switch that closes that gap. In regulated mode the binary fails startup unless all of these are present together, each with an actionable error naming the field to set:

• the OPA policy gate is on (ca.policy.enabled=true);
• four-eyes dual control is on (ca.policy.require_approval=true) with at least two distinct approvers (ca.policy.required_approvals unset, defaulting to 2, or >= 2) — a single approver is rejected;
• a default certificate profile is bound (ca.default_profile);
• revocation publication is configured — at least one of ca.crl_distribution_points or ca.ocsp_servers — so issued leaves carry a status pointer (composing with the served-leaf profile);
• and, when ca.require_fips=true is declared, the FIPS 140-3 module is active (the binary was built with GOFIPS140=v1.0.0 / make fips-build, or run with GODEBUG=fips140=on).

A complete regulated config boots normally. The default posture (ca.governance_mode unset, or standard) imposes no coupling, so existing single-node deployments are unaffected.

Served AI surface and model adapter

The AI/RCA/MCP surface is off by default. MCP investigation tools are read-only when enabled; MCP write tools require the separate TRSTCTL_AI_MCP_WRITE_TOOLS=true operator opt-in. The model adapter is separately off by default: with TRSTCTL_AI_MODEL_MODE=off (or unset), query/RCA still return grounded citations, but no prompt leaves the process. GET /api/v1/ai/status reports the live enabled state, model mode, endpoint host, egress class, and redaction/refusal posture without echoing the full endpoint URL.

Ollama local mode sends the native generate shape to the endpoint. vLLM and cloud mode send an OpenAI-compatible chat-completions shape. Every model path goes through the boundary redactor and residual-secret refusal gate before the HTTP request is made; if no model is configured, the answer remains citation-grounded and air-gapped.

Served protocol listeners

ACME, EST, SCEP, CMP, SPIFFE, SSH, and KMIP protocol surfaces are opt-in until they are explicitly bound to a tenant. That startup check is intentional: a public protocol endpoint must know the tenant it acts for before it is exposed. KMIP is a raw mTLS TCP listener, not an HTTP route, so it additionally requires server certificate/key files and a client CA trust anchor.

For the blank evaluation stack, TRSTCTL_PROTOCOLS_PROFILE=eval is a bounded shortcut: it assembles ACME, EST, SCEP, CMP, SSH, TSA, and SPIFFE for exactly TRSTCTL_PROTOCOLS_EVAL_TENANT_ID. Those responders stay behind a closed runtime gate until a tenant-authenticated operator activates them in the first-run wizard or with POST /api/v1/setup/protocols/activate. Activation appends an immutable tenant event before the HTTP routes and SPIFFE socket become reachable, and replay restores the gate after restart. The shortcut never enables KMIP. Production can omit PROFILE and use the individual toggles below.

Agent mTLS channel

The served agent gRPC channel is a real listener the control plane exposes for steady-state agent heartbeat, renewal, and command fan-out (WIRE-004 / OPS-005) — separate from the one-shot bootstrap enrollment path, which always works. It is off by default; enabling it without a configured signer (the agent CA is custodied there) is a startup error.

See Getting started for the blank Compose stack's published agent-channel port and the local CA-pinning steps to reach it from an agent CLI.

WASM plugins

The WASM plugin surface is off by default. When enabled, the binary admits only signed modules whose detached Ed25519 signature verifies against the configured trusted keys. CA plugins become external CA entries of type wasm-ca; DNS-provider plugins become a served ACME DNS-01 provider that tenant provider configs can select; connector plugins handle matching deployment work from the outbox.

SPIRE upstream authority plugin

When SPIRE should keep serving workload SVIDs but trstctl should own the upstream CA, configure SPIRE's UpstreamAuthority "trstctl" plugin. SPIRE passes this as HCL plugin_data to the trstctl-spire-upstream-authority process; these are not trstctl environment variables.

Example:

UpstreamAuthority "trstctl" {
  plugin_cmd = "/opt/spire/plugins/trstctl-spire-upstream-authority"
  plugin_data {
    endpoint = "https://trstctl.example.com:8443"
    ca_authority_id = "11111111-1111-1111-1111-111111111111"
    token_file = "/run/secrets/trstctl-spire-token"
    common_name = "SPIRE Server CA"
    ttl_seconds = 3600
    max_path_len = 0
    permitted_dns_domains = ["example.org"]
  }
}

Rate limiting

A per-tenant, PostgreSQL-backed rate limiter sheds load on the guarded routes (429 + Retry-After). See Operations & resilience for the model and the bulkheads it complements.

ACME also has protocol-local abuse budgets above because its public nonce/account/order routes are not REST API routes. The protocol bulkhead limits concurrent work; the ACME quota knobs bound retained ACME protocol state.

When enabled, requests must be positive and window a valid positive duration, or the control plane fails fast at startup.

Bulkheads

Each subsystem runs on its own bounded worker pool. workers caps concurrent work; queue caps accepted backlog before trstctl rejects fast with structured backpressure. Every value must be positive. Defaults are conservative for a single-node or small HA deployment; larger fleets should raise only the subsystem that is actually saturating.

Fleet-size guidance:

trstctl --check-config prints the effective bulkheads.<subsystem>.workers and bulkheads.<subsystem>.queue values, so CI/CD can diff the resolved runtime limits before a rollout.

Every outbox family owns a separate queue and worker set even when all limits are inherited from bulkheads.outbox. In plain terms, a connector endpoint that stops answering can fill only the connector lane; it cannot occupy the workers that issue through an upstream CA, synchronize a secret, publish code-signing evidence, or page an operator. JSON config may override a family with outbox_external_ca, outbox_connectors, outbox_secrets, outbox_secret_sync, outbox_managed_keys, outbox_transparency, outbox_code_signing, or outbox_notifications, or outbox_tenant_seal inside bulkheads.

Config file

Any of the above can also be set in a JSON file named by TRSTCTL_CONFIG_FILE; environment variables override file values, which override defaults.

{
  "server": { "addr": ":8443" },
  "postgres": { "mode": "external", "dsn": "postgres://..." },
  "nats": { "mode": "external", "url": "nats://..." },
  "telemetry": { "enabled": false, "instance_id_file": "data/telemetry/instance-id" }
}