A ctlplne studio product

Current limitations & what's not yet served

trstctl is pre-1.0 and under active hardening. This page states plainly what the running binary serves today versus what is built and tested as library code but not yet wired into the served product, and which surfaces are explicitly Phase 2. Maturity is separate from edition gates: Community self-host includes the core control plane; Enterprise and Provider capabilities activate by an offline signed license behind the ee/ boundary. trstctl's core is source-available under LICENSE (BUSL-1.1, converting to MPL-2.0 four years after each release); Enterprise, Provider and other license-gated features are proprietary material under ee/LICENSE, while PQC and the patent-pending families ship in the core.

If a capability matters to your evaluation, check this page before relying on it.

The 60-second boundary summary

Read this first; use the exhaustive ledger below only for the capability you plan to operate.

Question Short answer What to verify next
Can I run the core control plane myself? Yes. Inventory, lifecycle state, ownership, policy, audit, the REST API, CLI, web console, PostgreSQL tenant isolation, and the separate signer are served. Complete the blank evaluation, then rehearse production operations.
Does every integration work immediately? No. Protocols, connectors, HSMs, cloud discovery, SSO, and commercial workflows are conditional on explicit configuration, reachable infrastructure, and sometimes a signed license. Open the relevant feature page and the matching matrix row below before promising it.
Does “delivered” mean a workload is serving the credential? No. Delivery records what the connector did. Only a successful endpoint handshake earns verified. Review the served status vocabulary and configure a verification address.
Is a green System page a production certification? No. It proves the running process's bounded checks, not an external penetration test, customer topology, disaster-recovery rehearsal, or third-party certification. Use the production journey, threat model, and your own acceptance controls.
Is every feature production-mature? No. trstctl is pre-1.0. The tables below distinguish Served, Conditional, Partial, and Roadmap without turning source code into a product claim. Search this page for the feature ID or plain-language capability name.

The safe decision rule is simple: served says the running binary has a real path; verified says the required external effect was observed; neither word replaces your deployment-specific acceptance test.

Local qualification host limit

The shipped definition-of-done runtime runner targets linux/amd64. Its proof opens the kernel's /proc/<pid>/map_files object to check that the executable being tested is the exact binary built by the gate. On an Apple Silicon Docker daemon using QEMU user emulation, the kernel exposes the emulator's mappings instead of the guest executable's mapping. The runner then stops at its map_files preflight. All 81 census rows can show stub before any capability is exercised; those rows cannot establish whether the capabilities work.

Run this qualification gate on native amd64 Linux, or on a separately validated Docker Desktop Rosetta setup whose map_files proof succeeds. The 2026-10-02 local partner-lab run on the current Apple Silicon host passed its 18 live journeys, but its 81-row runtime census remains unqualified for this specific host reason. Changing Docker Desktop's emulation mode requires a daemon restart; defer that change while unrelated containers use the shared daemon. Keep the runtime proof intact and record its failed native result.

Agent release distribution

As of 2026-10-02, the published v0.6.0 GitHub release lists zero assets. It cannot supply a downloadable host agent, checksum manifest, or SLSA provenance. The current source tree has a four-platform archive builder and a protected release job that uploads and checks those assets. A task-local exact-commit archive has been downloaded with curl, checksum-checked, and run on macOS; that proves the candidate packaging path, not public distribution.

An operator needs a subsequently completed release with the archive for their host, the source-commit manifest, SHA256SUMS, and provenance. Verify the public asset roster, download without repository credentials, check the hashes and provenance, and compare the binary's full reported commit with the manifest before treating the release install journey as complete. Publishing that release requires the protected release pipeline and is outside a local-only QA run.

Historical scheduled-rotation detail

The current live event generation does not serve arbitrary provider error text from schema-v1 scheduled-rotation terminal events. Upgrade sanitation replaces that one field with a fixed status-specific class before reads, projections, retention, or export can proceed. This guarantee applies only to repository- controlled live history. Backup/export files and signed or WORM audit archives created before sanitation are external copies: trstctl cannot rewrite them and does not claim they were erased. Operators must govern those copies under their existing custody and retention policy.

Feature served-state matrix

This matrix is the canonical served-state table for the feature catalog. The docs test suite checks every F* row below against internal/featureparity/feature-map-backlog.json and runs the repo-native wiring census against the current source tree. A feature cannot claim Served while a mapped capability is still library-only, a stub, or unknown; a stale generated JSON receipt cannot certify it.

  • Served means the running binary serves the capability end to end.
  • Conditional means the served path exists but depends on configuration, license activation, or an operator-supplied backend.
  • Partial means a real served path exists, with explicit residual work called out below.
  • Library-only means built and tested, but not yet served by the binary: library code exists with tests, but is not yet wired into the served API.
  • Roadmap means Phase 2 or later work with no current served product claim.

Served

ID Feature Primary docs
F1 Certificate inventory docs/features/discovery-and-inventory.md
F2 Network discovery docs/features/discovery-and-inventory.md
F42 SSH credential discovery and inventory docs/features/discovery-and-inventory.md, docs/features/ssh.md
F49 Agentless cloud certificate discovery docs/features/discovery-and-inventory.md
F35 Secret store discovery docs/features/discovery-and-inventory.md, docs/features/secrets.md
F36 API key / token inventory docs/features/discovery-and-inventory.md, docs/features/secrets.md
F17 Certificate Transparency monitoring docs/features/observability-and-risk.md
Discovery coverage & provenance Served: coverage is measured against operator-declared segments rather than against what discovery happened to find, with a per-segment staleness SLO, declared exclusions carrying their reason, and a named blind-spot register. Every certificate carries provenance — which source last observed it, of what kind, and when — distinct from when trstctl first recorded it. Headlined on the Discovery console. Nothing is rollable into coverage until an operator declares a segment: an inventory built from findings can describe what it found and nothing else, so an estate with no declarations reports no coverage rather than 100% Coverage, provenance and blind spots
Revocation through external issuers Served for letsencrypt (ACME), vaultpki and ejbca — each proven end to end against that authority's own protocol, asserting the AUTHORITY was contacted rather than that the call returned nil. Every other issuer kind reports revoke: false on the served capability matrix with a note naming where to revoke instead. A documented vendor endpoint trstctl does not drive is not counted as a capability. Revocation that cannot reach the authority fails visibly; there is no silent no-op Per-issuer capabilities
Upstream domain validation (DNS-01 as an ACME client) Served: obtaining a certificate FROM a public CA now negotiates the challenge type the authority offers and solves DNS-01 unattended, reusing the provider configs already persisted for the server direction. Wildcard issuance from a public CA works for the first time — DNS-01 is the only challenge that can authorize one. Two independent opt-ins, both default off: upstream_dns01 on the ACME authority (which also requires that authority's own caa_issuer_domain) and allow_upstream_dv on each DNS-01 provider config. The console shows a domain-validation column beside every configured issuer — unattended or manual, with the reason — and an upstream authorization freshness panel reporting when each identifier last actually proved control, as against when it last rode a reuse. DNS-01 only: http-01 upstream would need an inbound listener this architecture does not have Upstream domain validation
F18 Drift detection docs/features/observability-and-risk.md
F19 Credential risk scoring docs/features/observability-and-risk.md
F52 CBOM and cryptographic observability docs/features/observability-and-risk.md
F4 CA-agnostic outbound issuance docs/features/issuance-and-cas.md
F48 Private/enterprise CA hierarchy management docs/features/issuance-and-cas.md, docs/runbooks/key-ceremony.md
F53 Certificate profiles and registration-authority model docs/features/issuance-and-cas.md, docs/guides/profile-authoring.md
F46 ACME Renewal Information (ARI) docs/features/issuance-and-cas.md, docs/features/acme-and-dns.md
F47 X.509 revocation infrastructure docs/features/issuance-and-cas.md
F25 Ephemeral credential issuance docs/features/workload-identity.md
F30 Workload attestation chain docs/features/workload-identity.md
F59 Non-human identity lifecycle management docs/features/workload-identity.md, docs/features/discovery-and-inventory.md
F61 AI-agent / NHI identity broker docs/features/workload-identity.md
F44 SSH deployment and trust configuration agent docs/features/ssh.md, docs/design/ssh-trust-rewrite.md
F45 Attestation-gated short-lived SSH user certs docs/features/ssh.md
F6 Lifecycle automation docs/features/lifecycle-and-pqc.md
F33 Just-in-time issuance with approval flows docs/features/incident-and-jit.md
F38 Ephemeral API key issuance docs/features/secrets.md
F28 Policy engine docs/features/policy-and-governance.md, docs/cli.md, docs/web-console.md
F29 Notification integrations docs/features/policy-and-governance.md
F8 RBAC docs/features/policy-and-governance.md
F9 Audit log surfaces docs/features/policy-and-governance.md, docs/observability.md, docs/configuration.md
F10 REST API docs/features/platform-and-api.md
F11 CLI docs/features/platform-and-api.md, docs/cli.md
F12 Web UI docs/features/platform-and-api.md
F14 Single-binary distribution docs/features/platform-and-api.md
F15 Encrypted control-plane transport docs/features/platform-and-api.md
F40 Multi-tenant deployment topology docs/features/platform-and-api.md
F20 Plugin SDK with capability sandboxing docs/features/extensibility-plugins.md
F21 Credential graph docs/features/graph-query-ai.md
F79 Privacy and data-subject controls docs/features/policy-and-governance.md, docs/privacy-data-catalog.md, docs/web-console.md, docs/configuration.md

Conditional

ID Feature Primary docs
F3 Agent-based discovery docs/features/discovery-and-inventory.md
F5 Built-in ACME server docs/features/acme-and-dns.md
F69 DNS-01 challenge automation docs/features/acme-and-dns.md
F70 DNS-provider plugin framework docs/features/acme-and-dns.md
F71 CNAME delegation for validation isolation docs/features/acme-and-dns.md
F72 CAA policy enforcement and management docs/features/acme-and-dns.md
F73 Multi-method domain-validation policy docs/features/acme-and-dns.md
F74 Automated wildcard issuance and renewal docs/features/acme-and-dns.md
F22 EST server docs/features/enrollment-protocols.md, docs/guides/est-enrollment.md
F23 SCEP server docs/features/enrollment-protocols.md
F55 CMP server docs/features/enrollment-protocols.md
F54 Embedded / IoT enrollment agent docs/features/enrollment-protocols.md
F56 Intune / MDM enrollment integration docs/features/enrollment-protocols.md
F24 SPIFFE Workload API docs/features/workload-identity.md
F43 SSH certificate authority docs/features/ssh.md
F50 Code-signing service docs/features/code-signing-and-timestamping.md
F51 Timestamping authority docs/features/code-signing-and-timestamping.md
F26 HSM integration docs/features/issuance-and-cas.md, docs/configuration.md, docs/compliance.md, docs/limitations.md
F7 Deployment connectors initial set docs/features/deployment-connectors.md
F27 Additional deployment connectors docs/features/deployment-connectors.md
F31 Credential compromise workflow docs/features/incident-and-jit.md, docs/features/discovery-and-inventory.md
F32 Fleet re-issuance for CA compromise docs/features/incident-and-jit.md
F34 Break-glass procedures docs/features/incident-and-jit.md
F37 Secret rotation engine docs/features/secrets.md
F39 Code/CI secret scanning bridge docs/features/secrets.md
F63 Native secret store docs/features/secrets.md
F65 Dynamic secrets docs/features/secrets.md
F80 AWS IAM decoy keys docs/features/secrets.md, docs/web-console.md, docs/cli.md, docs/limitations.md
F66 Encryption-as-a-service and KMIP docs/features/secrets.md
F68 Secret sync / platform integrations docs/features/secrets.md
F67 PKI as a secrets engine docs/features/secrets.md
F58 Platform auth-method framework docs/features/secrets.md
F60 Secret sharing and secret-change approvals docs/features/secrets.md
F62 Cryptographic compliance reporting & posture dashboards docs/features/policy-and-governance.md, docs/compliance.md
F13 SSO/OIDC docs/features/platform-and-api.md
F41 Cross-cluster / multi-region federation docs/features/platform-and-api.md
F75 Unified semantic query layer docs/features/graph-query-ai.md
F76 Pluggable AI model adapter docs/features/graph-query-ai.md
F77 Grounded RCA and natural-language query docs/features/graph-query-ai.md
F78 trstctl MCP server docs/features/graph-query-ai.md

Partial

ID Feature Primary docs
F16 Crypto-agility and PQC readiness docs/features/lifecycle-and-pqc.md
F57 PQC migration orchestration docs/features/lifecycle-and-pqc.md
F64 Developer secrets experience docs/features/secrets.md, docs/cli.md, docs/journeys/manage-secrets.md

Library-only

ID Feature Primary docs

Roadmap

ID Feature Primary docs

Feature pages without an F* catalog row

A feature page can document a surface that carries no F* catalog ID, so the generated matrix above cannot hold it. Those pages record their served state here instead, in the same vocabulary, so no page in the feature catalog is statusless.

Page Served state Why it has no F* row
docs/features/agent-delegation.md Conditional Part of the core since 2026-09-20. attachAgentDelegation in cmd/trstctl/attach_families.go attaches the /api/v1/agent-delegation/* routes, root-event projection and agentid.issue-chain-bound outbox worker in every build. Root registration writes an immutable first-writer event, rebuilds the tenant registry from it, then writes and independently reads the signer's public floor over the authenticated signer transport. The issuance receipt has a bounded status read that binds the immutable result to the exact public credential ID; historical receipts without a binding are not guessed. Legacy SQL-only roots require exact signer readback before event backfill; a one-shot rebuild refuses them. The control plane provisions and independently reads back its public reachability-verdict floor through the signer RPC. Tenant attestors register the same way: an immutable first-writer event binds each attestor key ID to its public key and one attestation method, and the control plane provisions the signer's tenant attestor floor and reads it back. The signer verifies a tenant's attestation evidence only against that tenant's attestors, for the registered method and no higher class, or against a deployment-wide attestor an operator placed in its floor. The signer refuses a chain-bound mint until the required floors are present. Each tenant has its own agent issuing CA in the signer (its key reloads after a restart; the certificate is re-issued over it before expiry), published at GET /api/v1/agent-delegation/issuing-ca for relying services to pin; GET /api/v1/agent-delegation/credential-status reports active, revoked, expired or unknown, so a relying service that checks it stops honoring a revoked delegation within its status window rather than at expiry. Reference integrations on the MCP Go SDK and the Agent Development Kit for Go (test/interop/agentframeworks) run against the real signer in the integration suite, as does an agent-to-agent pair on the official A2A Go SDK: signed Agent Cards bound to the serving agent's certificate, mutual TLS, and the delegation gate on every request. What comes next: a pushed revocation feed for relying services (the downstream revocation plane only acknowledges today), and an issuing CA an operator provisions per tenant.
docs/features/client-sdks.md Served The generated clients under clients/sdk/ track the served OpenAPI 3.1 contract; they package the REST API rather than adding a capability of their own.

Census proof modes

The served denominators quoted in the README and on this page are checked against the repo-native census gate (tools/dodcensus/manifest.json in the repository) (make dod-gate emits the local wiring-census.json receipt). Proof modes are not uniform: 12 of 81 census rows launch the shipped binary; 69 of 81 are proved through the production-assembled handler — the production buildRunDeps output driving the assembled Server.Handler in-process, with a hand-built Deps rejected. Only the process launch differs, so each served row names the mode that proved it. This vocabulary is for maintainers and auditors; a customer reads the README's one-line summary and the journeys.

Status at a glance

One line per domain below, for a reader who wants the answer without the prose. "Served" here always means the running binary, not a library package.

Domain Status Detail
Core inventory, lifecycle, connectors, discovery Served end to end Served by the running binary today
Tenant offboarding & audit retention Served; PostgreSQL rows erased, event log/archive follow separate retention Tenant offboarding boundary
Library-only backlog Empty — nothing is stuck library-only right now Built and tested, but not yet served
Conditional/partial residuals Real served spine; specific operator-facing edges remain Conditional, partial, and residual boundaries
Served status strings Every status is registered with what the code actually did; CI blocks a status spelled stronger than its own flags Served status vocabulary
CA hierarchy expiry horizon Served; year-scale bands, re-alerting on each tightening, leaf-validity-compression check, horizon on the CA API and console, opt-in drafted re-key ceremonies at a chosen band The CA calendar
ACME external account bindings Served; kid persisted on the account, per-credential identifier scope / quota / window / profile enforced fail-closed, runtime disable, served policy replacement and rotation by secret-store reference (both replayed from the ACME state log) Protocols
Certificate Transparency monitoring Served as a headline Discovery capability: exact source replacement, active versus retired watchlist history, immutable per-log success/failure with independent checkpoint progress, unexpected-issuance findings, and remediation hand-off. Covers only the domains and logs configured Served by the running binary today
Key custody per credential kind CI-checked table; every enrollment protocol, and the identity API given a CSR, generate keys in your environment. Four paths still generate one in the control plane, each named with its successor and each governed by the tenant's control-plane key generation switch: the three deprecated paths (identity issuance without a CSR and its renewal, PKI-as-a-secret by common name and the Vault-compatible pki/issue, the compatibility Workload API socket) and the reissue step of a PQC migration run that predates host-csr migration. Kubernetes TLS auto-issuance never generates a key in the control plane: a k8s_ingress_gateway source bound to the host-role agent running in the cluster hands each TLS block to that agent, which generates the key inside the cluster, sends only a CSR and writes the TLS Secret with a signed receipt; a source without an agent records each resource and reports it not delivered, with the reason Key custody
Key custody per credential Served: custody is recorded on the certificate row at issuance from what the issuing path actually did, returned by the certificate API, and shown on the certificate in the console. Certificates issued before this shipped get the origin their own retained issuance event proves, marked as derived and naming the rule and the event (Key custody lists the rules); the rest of them, and every certificate found by discovery, read as not recorded — which is a different statement from any custody claim, and is never rendered as reassurance. A certificate ordered through an external CA records requester custody (A121), and where a host agent's key inventory found a matching key file the detail lists where it was seen, as an observation rather than a custody claim (A121) Key custody
Agent job ledger Served: agents claim, lease, extend, report and lose work over the mTLS channel; aggregate waiting/claimed health is on Operations. The shipped agent census executes connector.deploy, connector.test, connector.rollback, endpoint.renew, connector.right_size (for bindings that name a relay), endpoint.contain, endpoint.recovery_check (enabled with endpoint.contain), endpoint.verify, discovery.run, ssh.trust.collect (explicit read-only agentless SSH sources), revocation.probe, adcs.inventory, adcs.inventory.enrollment_groups (opt-in bounded nested enrollment links), adcs.inventory.endpoint_probe (opt-in scoped LDAP hostname probes with retained pages), trust.distribute, cmdb.sync, mdm.sync, ticket.sync, agent.upgrade, and agent.self_check (enabled with agent.upgrade); each row's role and agent constraints select the eligible host agent or network relay. Nothing is claimable until an operator names that kind in agent_channel.claimable_job_kinds — including connector.rollback, which must be enabled separately from deploying The agent job ledger
Agent job receipts Served: every terminal report is signed by the agent with the key behind its channel certificate, verified against the certificate that authenticated, stored with the event, and refused fail-closed with an audit event when it does not verify. A successful host-generated renewal uses the v2 statement, which also binds the installed certificate fingerprint plus origin, storage, exportability, and generator; the projected custody event is replayable. Verified and refused counts, and the reason for the most recent refusal, are on Operations. The signature is over the report's facts and a digest of its text — it attests what the agent SAID, not that the appliance changed The agent job ledger
Renewal windows, canaries and SLOs (D6) Served: maintenance windows restrict when the scheduler may renew (lifecycle.maintenance_windows, e.g. Mon,Tue,Wed,Thu,Fri 22:00-06:00 Europe/London); a closed window defers with a recorded reason naming when it reopens, never drops. Fleet re-issuance is a durable outbox-backed batch state machine: start publishes only the canary, an accepted signed agent receipt is required to advance, pause/halt stores the cursor and reason, and resume/restart reuses deterministic ids. Any failed verification fails the gate and any unverified replacement keeps it not_evaluated; later batches remain unpublished after a canary failure. Renewal success SLO with error-budget burn on GET /api/v1/operations/renewal-slo, window and target both operator inputs Renewal windows, canaries and SLOs
Endpoint verification (D2) Served: after a deploy the host agent handshakes the changed listener — the only observation of whether the reload took effect — and a network relay probes the same endpoints as a client would, which is the only witness for an appliance. Divergence is classed (fingerprint, sans, chain, expired, not_yet_valid) because the remedies differ; unreachable is neither a pass nor a divergence. Results are signed: the probe transcript's digest travels inside the agent's receipt, so a verdict is checkable rather than asserted. Verification is opt-in per target: an endpoint with no configured listener address is never verified and never claims to be. verified % on the dashboard is a percentage of OBSERVED endpoints and the tile is hidden entirely until something has been observed. Sweeps re-probe hourly; divergence raises a critical alert (unreachable: warning) through the notification outbox; automatic rollback to the predecessor is available per target, opt-in and off by default Endpoint verification
Connector rollback Served through two honest execution models. f5, kemp, netscaler, a10 re-bind a fingerprint-named object already on the appliance, and fortigate (bind_to), paloalto (ssl_tls_profile + commit_admin), cisco (node_hostname + certificate_roles) and gcp-certificate-manager (certificate_map_entry, executed on the control plane where that family's deploys run) do the same when the target names the binding to move. All 14 host connectors restore the one encrypted predecessor bundle retained only by the exact enrolled host agent, run the connector reload, and reverify the listener when configured. Deploy and rollback share a serialized per-target lane. aws-acm, azure-keyvault and plugin-backed routes have no executable rollback by their APIs' design; the route returns 409 naming that reason, the destination's rollback_readiness says the same, and no rollback-shaped receipt is written. A re-bind target without its binding is refused at request time. Automatic rollback after verify_failed is opt-in per target; manual rollback uses the same job and signed transcript The agent job ledger
CA migration waves (H2) Served for internally issued X.509 identities on host-agent connectors: read-only assessment, reviewed exact-authority manifest, durable trust-before-leaf waves, signed trust/live gates, pause/resume, halt, and newest-first rollback Migration waves
Agent roles (host / network relay) Served: an operator grants host and/or network at enrollment, the CA stamps it into the certificate, and the claim path refuses out-of-role work. Role badges on Agents. Agents redeem credential material just-in-time, once per job attempt. Connector deploys carry a per-row role demand stamped at enqueue from the shipped vantage census — an F5 deploy is claimable only by a relay, an nginx deploy only by a host agent, a cloud-store deploy by no agent. The control-plane dispatcher structurally refuses host-stamped and legacy host-family rows before native lookup or I/O, so 14 host families execute only on the enrolled host agent; an unavailable agent leaves pending work, never a control-plane fallback Agent roles
React web console Served: real embedded Vite build at /, generated API types The React web console
OIDC/SAML/LDAP browser login & tenancy Served behind config flags; each user maps to a real tenant Browser login & sessions
SCIM 2.0 + NHI inventory/posture Tenant SCIM requires Enterprise SSO; SCIM Bulk served on both planes; directory writeback served to a SCIM-serving directory, never to a push-only IdP SCIM 2.0 provisioning
AI / RCA / MCP surface Served, off by default, air-gapped unless an operator opts in AI, RCA, and MCP surface
Secrets, identity frameworks, transit/KMIP Served (six of six frameworks); Vault shim is a partial subset Secrets and identity frameworks
RBAC / ABAC / OPA policy gates Served, fail-closed, off by default Authorization policy gates
Plugin isolation First-party runs trusted in-process; third-party is WASM-sandboxed and signature-verified Plugin isolation
Protocols: ACME/EST/SCEP/CMP/SPIFFE/SSH/TSA Served end to end, each behind its own enable flag Protocols
Revocation (OCSP/CRL) Served, signed by the isolated signer Revocation
Single sign-on detail Served, including SAML single logout, Kerberos/GSSAPI and NTLM (through a domain-joined helper) login on the tenant plane Single sign-on
CA key custody / HSM / BYOK Sealed local key store by default; six of six HSM/KMS backends served CA key custody
Post-quantum cryptography ML-DSA/ML-KEM/SLH-DSA available; compatibility-bounded proofs Post-quantum cryptography
PCAS KEM data re-wrap Not built; the served route prepares successor keys but cannot migrate ciphertext PCAS KEM data re-wrap
Kubernetes deployment Helm chart plus an Operator that converges the control plane's Deployment, Service, default-deny NetworkPolicy, optional Ingress and isolated signer with a generated mTLS pair; it polls rather than watches Kubernetes deployment
Performance & usability NFRs Perf/scale measured in CI; usability evidence-gated, no NPS claim yet Non-functional targets

Served by the running binary today

The trstctl binary assembles and serves a control plane: the tamper-evident event log, the read models it projects, the lifecycle orchestrator, and the REST API, with the signing service supervised as a separate out-of-process child so private keys never live in the API process. What you can do end to end against the running binary:

  • Inventory and lifecycle for owners, issuers, identities, and certificates: create, read, list (keyset-paginated), and drive the lifecycle state machine.

  • Connector delivery and rotation evidence: deployment attempts emit connector.delivery.recorded receipts and scheduled renewals emit lifecycle.rotation.recorded runs, both readable through the API, CLI, and console. The receipt is routing/status metadata only — no private key or secret bytes are returned. Rotation replay compares completed_at at PostgreSQL's microsecond precision: sub-microsecond bits that cannot survive the first database write are one observation, while a difference of one stored microsecond still fails closed as changed terminal evidence. This lets exact JetStream replay converge without weakening the run ID, tenant, outbox, binding, outcome, or sequence checks.

  • Automated endpoint binding: first call the effect-free POST /api/v1/lifecycle/endpoint-bindings/preview with the exact platform, private, or external issuer. The preview reports CA identity, key custody, target revision, writes, queued effects, recovery, verification, and a request fingerprint while performing zero writes and zero external calls. Sending that unchanged fingerprint to POST /api/v1/lifecycle/endpoint-bindings creates the X.509 identity for an existing owner, provisions or references the connector target, pins the chosen issuer, and queues issue/deploy work through the outbox. Missing, unavailable, or changed issuers fail closed; the worker does not substitute the built-in CA. The leader scheduler later renews through that same pinned authority and sends the successor through connector.deploy.

  • Certificate key custody, corrected (B5 re-audit): the four custody columns (key_origin, key_storage, key_exportable, key_generated_by) are now actually persisted. They were added to the schema, set by the issuing code and served by the API, but projections.CertificateRecorded had no fields for them and the projector's INSERT never listed them — so every projected certificate recorded blanks while three layers of the system agreed custody had been captured. B5 was signed off in that state. The key_origin half surfaced while implementing B2, because B2's own claim depended on it; the other three surfaced only on re-audit. A regression test now reads the DATABASE rather than asserting a struct field was assigned — the distinction that made this invisible — and a structural test fails if a custody field exists on the row without a matching field on the event that carries it.

  • Host-generated endpoint keys (B2): a deployment target whose config sets executor: "agent" opts out of credential-bearing delivery entirely. When the lifecycle scheduler renews an identity bound to such a target, the issuance dispatcher queues an endpoint.renew job INSTEAD OF MINTING — the branch is at mint time, not deploy time, because once a certificate has been minted for a server-keygen identity the control plane already holds a private key and no later refusal can unmake that. A host agent then claims the job, generates the subject key on the machine that will serve it, sends a PKCS#10 up through SignJobCSR, installs the returned certificate with its locally held key, and verifies the listener. The rotation run is recorded as succeeded on the HANDOFF, not on a certificate: the certificate does not exist until the agent's CSR arrives, and D3's three-state truth reports the rest. The control plane never holds that private key, and enforceExecutorParity REFUSES — rather than silently falling back — any deploy that would carry key bytes to such a target, so a target cannot read as migrated while still receiving keys. Scope, stated exactly: this is per target and opt-in; targets without the marker keep the control-plane path unchanged, which is the supported default and not a defect. Renewal is host-vantage only — a network relay cannot claim endpoint.renew, because generating a key for an appliance it merely reaches would reintroduce the custody hop this removes. The CSR is authorized against the names the job payload already carries, so an agent cannot widen its request. GET /api/v1/endpoints/key-custody and the Connectors console report, per target, which path it is on and how much of the estate has moved.

  • Expiry-alert delivery: the leader lifecycle scheduler honors the configured alert window, writes notification.expiry outbox work, stamps alerted_at in the same transaction so one certificate does not spam, and the outbox worker dispatches through operator-wired Slack, Teams, email, SMS, SIEM, webhook, PagerDuty Events v2, or OpsGenie Alert v2 channels. The dispatcher holds credentials in locked memory and wipes them on shutdown; the payload carries the owner, approver escalation recipients, severity, and threshold-day metadata. Tenants manage their own channels at /api/v1/notification-channels (Slack, Teams, webhook, PagerDuty, OpsGenie, SIEM) and routing policies that name them; since A98 each tenant-authored channel delivers in its family's own wire shape (a Slack or Teams incoming-webhook message, a PagerDuty Events v2 trigger or OpsGenie create carrying the key the row's credential_ref names in the tenant's secret store, a webhook HMAC-signed under that secret, a bearer-authenticated SIEM event) instead of one raw JSON POST, the credential is opened into locked memory for one delivery and released after it, and a row whose credential cannot be resolved fails only its own delivery with a credential-free reason. Email and SMS stay operator-configured: the API refuses a tenant row of those families because the record cannot carry the SMTP or gateway settings they need.

  • Air-gapped alert delivery: operator-configured PagerDuty, OpsGenie and, since A99, Slack, Teams, SIEM and SMS channels can each opt into the operator's bounded private-egress policy (*_ALLOW_PRIVATE_CIDRS, exact network addresses validated at config load, plus loopback-only *_ALLOW_INSECURE_HTTP for an emulator), so an internal relay, proxy, gateway or collector can receive alerts while every other destination stays behind the SSRF guard; a channel with a policy also carries the air-gap guard when air-gap mode is on, and a channel without one keeps the public-HTTPS-only default. The partner lab grants only 127.0.0.1/32. Tenant-authored channels may use private HTTPS receivers only after the deployment operator grants their exact tenant UUID and networks in notifications.tenant_private_egress. Without that grant they remain public-HTTPS-only. The API and delivery worker use the same grant; DNS is checked again at every dial and redirects cannot change origin. Tenants cannot grant themselves access. Tenant grants refuse loopback, metadata, link-local, wildcard and partly public ranges; HTTPS certificate validation and the air-gap guard still apply. The served private-TLS/outbox test proves one receiver and tenant isolation; fresh-image console and restart proof is recorded separately in the build replay.

  • Deployment connector orchestration serves target metadata, identity binding, outbox intent, receipts, provenance-verified WASM dispatch, and all 24 advertised native connectors. Device proof (E1): every appliance family — a10, cisco, f5, fortigate, kemp, netscaler, paloalto — now carries a faithful in-process double of its management API and tests that drive the REAL connector against it, and the connector catalog reports which families have that proof. This is a different claim from the conformance suite every connector passes: conformance runs against an in-memory double that accepts any request, so it proves a connector respects its capability grant and is replay-deterministic while proving nothing about whether the device would have accepted the call. For an appliance, whose entire implementation is an API conversation, that was the only untested part. Each suite is verified load-bearing by stubbing the connector's Deploy to return nil and confirming the tests fail. A guard test refuses a family that claims device proof without both an emulator package and a test that drives it. Single-box demo (A5): trstctl --demo serves the control plane AND starts a colocated host agent. Evaluating this product otherwise needs two installs, and until both exist nothing the product is FOR can be shown — no deploy executes, no endpoint verifies, no renewal lands on a host. It is also the cheapest defense against the defect this program keeps finding: six capabilities have been complete, tested and unreachable from the running binary, and every one would have been obvious the first time somebody drove it end to end on one machine. The agent is EXEC'd, never linked. cmd/trstctl-agent must not link the control plane and a guard pins that; importing it here to save a process would put both on the same side of the boundary the architecture rests on, and the demo would stop exercising the real channel. Two binaries over the real gRPC channel is what a deployment does, so the demo tests the actual path. --demo FORCES two settings and reports both rather than applying them quietly: the agent channel (off by default — correct for production, fatal for a demo, because the agent has nothing to dial) and a conservative claimable-job set (discovery.run, endpoint.verify, connector.test). An empty claimable allowlist is the trap worth naming: the job ledger is served, hands nothing out, and the agent enrolls successfully and idles while every surface looks healthy — the most misleading possible demo. connector.deploy is deliberately NOT enabled; an evaluation box must not mutate an appliance somebody pointed it at by accident. Settings an operator did set are never overwritten. A missing agent binary is a FATAL error naming where it looked and how to build one, not a warning. --demo is an explicit request for a colocated agent, and a run that serves the control plane while silently omitting the agent is exactly the "looks like it worked" outcome the flag exists to prevent. The bootstrap token is minted through the SERVED enrollment API — not by reaching into the store, because reaching in would let the demo work while the served enrollment path was broken — and written to a 0600 file, never passed as an argument, since process arguments expose bearer credentials and the agent refuses them for that reason. Also fixed here: agent_channel.claimable_job_kinds was the ONE AgentChannel field with no environment key, so a container deployment could enable the channel and had no way to let an agent claim anything. deploy/demo/docker-compose.yml set no agent-channel variables at all while its seed job advertised enrollment tokens nothing could redeem; it now enables the channel and the same conservative claimable set. SECURITY FIX (provider plane, 2026-08-05): the provider plane authenticated NOBODY. operatorFromRequest parsed Authorization: Bearer provider:<id>:<email> for SHAPE and returned an operator with Role: OperatorAdmin and MFA: true — no verification of any kind, and multi-factor asserted on the caller's behalf. /provider/ is mounted on the root mux behind only a bulkhead whenever the provider plane is licensed, so on any provider-tier binary tenant create, suspend, offboard and break-glass were reachable by anyone who knew the token format. The format was in the source. Break-glass was worse: consentBreakGlass did not authenticate at all and took the consenting subject from the REQUEST BODY, so the operator who requested emergency access named whatever approver they liked and consented to their own grant with a second call. Two-person control defeated by a JSON string. Fixed by requiring a configured OperatorAuthenticator. Every other dependency in the provider Config falls back to a working stand-in; this one deliberately does not, because the safe stand-in for "who is this caller" does not exist and a placeholder is precisely how the original behavior came to ship. A nil authenticator now refuses every request — an unconfigured provider plane is closed, not open — and the consenting subject is the authenticated caller, with a subject in the body REJECTED rather than ignored so an integration cannot keep sending one and believe it has effect. Unauthenticated requests answer 401 rather than 403: "I do not know who you are" is a different statement from "I know, and you may not". Both bypasses are mutation-verified against the real handler. The consent test creates a real tenant and a real grant first — an earlier version used a made-up grant id and the mutation check showed it caught the regression only because a nonexistent grant 404s, so its primary assertion never fired. Note that this CLOSES a surface that was open: a provider-tier deployment must now wire an authenticator before /provider/ serves anything, which is a deliberate breaking change and the right direction. THE AUTHENTICATOR NOW EXISTS (L1): provider.oidc in the config federates operator identity to the provider's own IdP — bearer tokens verified OFFLINE against a pinned JWKS (jwks_file/jwks_json; the plane never fetches keys from a URL the token's minter might control), issuer, audience, expiry and nbf all required, and ROLE AND MFA read from claims the IdP signed (role_claim values mapped by admin_values/operator_values; mfa_claim, default amr). A token the IdP vouches for that matches NO configured role is refused outright — federation is not enrollment, and the IdP vouching for the whole workforce must not make every employee someone who can suspend customers. A single-factor operator authenticates and is then refused mutations (403 at the act, not 401 at the door — the gap is MFA and the error should say so). Console sign-in (A222): with provider.oidc.login the console signs in through the IdP's authorization code flow (state, nonce, PKCE S256) and holds an HttpOnly, SameSite=Strict Provider session instead of a pasted bearer; the ID token meets the same offline checks with the client id as audience, no token reaches browser storage, and the directory row is re-read on every request. The session store is in memory, so a restart signs operators out. Federation answers WHO; the per-customer delegation still answers WHICH customers, and the combined test proves a federated admin delegated one customer cannot suspend another. SAML AND SCIM ARE NOW SERVED (AUD-58): Provider SAML mounts a separate SP at /provider/v1/auth/saml/login, /acs, and /metadata, verifies signed assertions plus issuer/audience/time/request correlation and mapped role/MFA, then issues a Provider-only HttpOnly session protected by double-submit CSRF. Provider SCIM mounts /provider/scim/v2; its file-backed bearer is hashed and wiped at startup. Join/update/leaver changes are immutable operator events. When SCIM is enabled, OIDC and SAML query that directory row every request, so a deprovisioned operator's still-valid credential is refused immediately and the same projection revokes all standing customer authority. Provider admin + MFA list/grant/revoke/role routes expose identity source, customer, operation, expiry, last use, and retained revocation evidence. Provider SCIM Bulk (POST /provider/scim/v2/Bulk, A89) applies User operations in order with bulkId references, each under its own idempotency key so a replayed request appends nothing twice; group operations are refused per operation. Custom Provider roles (A90): GET /provider/v1/roles serves the built-in admin and operator roles, the defined custom roles, and the closed permission vocabulary (tenants:provision, tenants:lifecycle, quota:write, brand:write, operators:admin, roles:admin, isolation:drill, breakglass:request, breakglass:consent, breakglass:results); PUT/DELETE /provider/v1/roles/{name} (operators holding roles:admin) define a role as a subset of that vocabulary or delete one nobody holds, each an immutable provider.role.upserted/deleted authority event. Every Provider mutation now asks for its permission rather than for the admin role; a custom role is assigned through the role route, the access console, or a SCIM roles value, and its effective permissions are intersected with the ceiling the signed IdP role allows (admin or operator), so a custom role never grants what the IdP would not. SCIM Groups expose the two built-in roles as provider-admin and provider-operator (A221): an IdP group push adds, removes and replaces members (PATCH in operation order, in the Okta and Entra shapes, and PUT), and a member it removes or leaves out loses the role as a provider.operator.role_revoked authority event naming the group and the SCIM token. The operator stays active with no role, which OIDC and SAML refuse on the next request; a user created without roles holds none until a group push assigns one. Standing customer delegations stay until a leaver event (active:false or DELETE) revokes them, and SCIM Bulk still refuses group operations per operation. What comes next: a live Okta and Entra push (today's proof is handler and PostgreSQL tests built on the request shapes their documentation shows). Crypto migration sequencing (M2): GET /api/v1/graph/crypto-readiness and a Risk console panel order every observed crypto asset by WHO DEPENDS ON IT, not by severity alone. The CBOM already said which algorithms are weak; it could not say which change is hard. Sorted by severity only, a forgotten lab box exhibiting RSA-1024 outranks a load balancer twelve services authenticate through, and the migration gets planned in the wrong order by a table that looked authoritative. The traversal walks BACK from a crypto asset along EXHIBITS to the resources using it, then BACK again along CONNECTS_TO / GRANTS_ACCESS / DEPLOYED_TO to the parties that depend on those resources. Both hops are reverse because every edge involved points from the dependent toward the thing depended on — a first cut walked out-edges from the resource, found nothing, and reported zero dependents for everything while looking perfectly well-formed, which is the worst failure available to a surface whose whole job is counting dependents. Each dependent is served with the resource it was reached through, so a sequencing recommendation can be checked rather than trusted, and owners are attributed so a blocker has names against it. Two honesty constraints are built into the wording rather than left to the reader. Dependents are what DISCOVERY HAS OBSERVED: the graph is built from scans, so an asset with zero dependents renders identically to one sitting on a resource nothing has scanned, and no row is ever labeled safe to rotate — the recommendation for a weak asset with no observed dependents says in words that zero observed is not zero. And a CBOM usage recorded with NO LOCATION has no place on the graph and no computable blast radius; it is counted separately as unplaceable rather than sorted to the bottom, because an asset that cannot be placed is unmeasured, not low-risk. AUD-64 fixes the production source and direction of those claims. A host/network reporter sends a metadata-only service_dependency finding through the tenant-bound agent.mtls.ReportInventory channel. Source kind and finding kind must BOTH be service_dependency; workload, target, and the target ref must be present and agree; and the verified agent certificate stamps the observing host and agent ID. A completed run binds that ID before the graph treats the observation as authority; manual discovery findings cannot forge this edge. The immutable discovery event survives restart/rebuild. graph.Build joins the exact workload name to that tenant's owner row and emits workload -> resource CONNECTS_TO; an unknown owner or mismatched target fails the graph read rather than silently reporting zero. Credential owner attribution separately follows the INCOMING production OWNS edge (workload -> credential), so tests no longer reverse that relationship. Scope: this sequences only dependencies an agent actually reports. It does not infer network traffic, and its ordering is only as complete as the reporters feeding it. AUD-65 makes this one exportable workflow instead of a Risk-only readout. GET /api/v1/graph/crypto-readiness is the canonical tenant dataset consumed by Risk and CBOM/Posture; it includes a digest and event-projected owner actions. POST /api/v1/graph/crypto-readiness/actions binds an existing core PQC campaign finding to the exact row digest and current attributed owner. Missing, foreign, unlocated/unowned, or later-changed topology is refused rather than guessed; stale actions stay visible but return 409 on mutation. GET /api/v1/graph/crypto-readiness/export returns the same ordered rows as bounded CSV and NDJSON plus an audit-key JWS/JWKS, and compliance evidence-pack v5 embeds the same dataset in its signed manifest. This still cannot discover dependencies no reporter observed; the coverage guidance travels in every format. CORRECTION (D2/D3, B2, R1, F1, H2 — 2026-08-05): FIVE agent-claimable job kinds dead-lettered before any agent could claim them. The control-plane dispatcher is the sole handler for every outbox sweep, and its default branch returns a hard error for an unrecognized destination — right for a genuinely unknown one, fatal for work an AGENT is meant to execute. A hard error burns the row's attempt budget and lands it in status='failed', and ClaimAgentJobs only ever hands out rows in pending. So endpoint.verify (D2/D3), endpoint.renew (B2's host-generated renewal — the epic's central path), revocation.probe (R1), adcs.inventory (F1) and trust.distribute (H2) each enqueued rows that died on arrival, while the API had already told the operator the work was queued. This was already understood for two of the nine claimable kinds: connector.rollback and connector.test carry explicit deferral cases whose comment spells out this exact failure. Nobody applied it to the other five. The dispatcher now derives the answer from agentJobKindAllowlist — the one place that decides what an agent may execute — so a kind added there cannot be forgotten here, because there is nothing here to forget. A genuinely unknown destination still fails closed: the dispatcher must not become a global silent ACK, which loses a row quietly instead of dead-lettering it loudly. Related and separate: RunEndpointVerificationScheduler was also never registered as a runtime worker, so nothing produced endpoint.verify rows at all. Registering it is what made the dead-lettering visible — the scheduler began producing work that immediately died. Both halves are fixed; either alone would have left the capability dark. That makes SIX instances of one defect in this program: D2's VerifyAddress with no producer, B2's endpoint.renew with no enqueue, B5's custody projection never written, J2's restore drill with no production caller, the unregistered verification scheduler, and this dead-letter path. The shape never varies — the capability is complete, its unit tests drive it directly, and the composition root does not reach it — and unit tests cannot catch it by construction, because they stand in for the caller that does not exist. Two guards now do: TestEveryDeclaredRuntimeWorkerIsActuallyStarted fails on any declared Run*(ctx) scheduler that is not started, and TestNoAgentClaimableKindIsEverDeadLettered walks the claimable allowlist and fails on any kind the dispatcher would refuse. Both mutation-verified; the second names all five kinds when the fix is removed. Offline renewal resilience (A5, PARTIAL): a host agent now schedules its next rotation from the credential's REMAINING LIFE on its own clock, not from --rotate-every alone. The old timer was the configured interval and nothing else, which let three ordinary situations end with an agent asleep past its own expiry: an interval set longer than the certificate lifetime, a control-plane outage spanning a whole interval so the single attempt inside it failed and the next was a full interval away, and an agent restarting while holding a credential already most of the way through its life. All three end identically — the agent wakes with an expired certificate and can no longer authenticate to renew it, which needs a person on the box, for a fleet whose value is not needing one. The next attempt is therefore min(two-thirds of remaining life, --rotate-every), jittered ±10%. The min() is the guarantee and only one half of it carries any weight: two-thirds-of-remaining is strictly below remaining, so no configured cadence can push an attempt past expiry. A first draft also clamped to remaining - remaining/10 with a comment calling that the load-bearing line; mutation-testing showed deleting it changed no outcome, and it was removed rather than left as dead code under a false claim. The jitter is not cosmetic: agents installed by the same automation hold near-identical expiries, and without spread a fleet renews in one second — arriving during recovery from the very outage that synchronized it. Local clock, deliberately: the agent that cannot reach the control plane is exactly the agent that most needs to renew early, and it has no other clock to consult. A host with a badly wrong clock renews at the wrong time, so the wrong clock is now made visible rather than corrected: every heartbeat carries the agent's clock and its answer carries the control plane's, and a drift past agent_channel.clock_skew_warning (default 2 minutes) is named on the Agents page and in GET /api/v1/agents with the measured drift, counted by trstctl_agents_clock_skewed_total for the TrstctlAgentClockSkew alert, and logged by the agent itself. Renewal timing still uses the host's clock, and an agent built before this release sends no reading. A5's other items have since landed and their honest scope lives with the staged-upgrade entry below: rings run as dispatched, per-agent-targeted jobs when a campaign publishes artifacts; the agent's -self-upgrade executor verifies the pinned sha256, keeps the previous binary as .old, reports a signed receipt and restarts; and --demo exists. Agent/control-plane version skew IS enforced — agentProtocolInterceptor refuses a handshake outside MinSupportedVersion..MaxSupportedVersion — and was met before this program, so it is not claimed as new work. Constrained edge sub-CA (B6, THE one deliberate exception to AN-3/AN-4): a host with no path to the brain issues leaves locally under a delegated CA, and every bound that makes that defensible is enforced, not documented. Minted BY the central isolated signer over a CSR (the edge key never travels), with name constraints taken from the SEGMENT'S POLICY — never the request — in the certificate itself (permitted and excluded subtrees, exclusion beating permission), path length pinned to zero, and a 30-day ceiling on life that is refused rather than clamped. Default OFF: minting requires a declared segment's explicit opt-in, and the opt-in is one declaration carrying the pinned TPM attestation roots, the segment's identifiers, and a closed allowed-key-provider list. That list defaults to tpm2 only. An un-attested host is refused. In the default TPM2 lane, an attestation over a different key than the CSR's is refused (a TPM vouching for one key must not license delegating another). The delegation is revocable from the brain and its serial lives in the parent CA's issued ledger, so OCSP and the CRL answer for it with no new machinery; expiry is the certificate's own clock. Local issuance (agent -edge-issue) enforces the constraints read FROM THE DELEGATION CERTIFICATE and fails closed on an out-of-constraint name; every issuance lands in a journal in exactly the reconcile request's shape, and trstctl edge delegations reconcile -f <journal> posts it when a path or a courier exists. Reconciliation re-verifies each leaf (signature chains to the delegated CA; names re-checked) and records an out-of-constraint leaf AS A VIOLATION — visible, never silently dropped and never silently accepted — while reconciled leaves enter the certificate inventory so there is no shadow estate. Key custody, stated exactly: trstctl-agent --edge-csr defaults to a persistent TPM2 signing object and writes only an opaque public handle; --edge-attest has a separately certified TPM AIK run TPM2_Certify over that exact object and CSR, verifies the resulting WebAuthn TPM envelope locally, and can write a complete reviewed CLI mint body; --edge-issue reopens the delegated key handle and asks the TPM to sign. PKCS#11 is a configured alternative whose shipping module creates a token object with CKA_SENSITIVE=true and CKA_EXTRACTABLE=false. It must be explicitly allowed in the segment policy because the host's WebAuthn TPM attestation authenticates the host and CSR request, not the separate token object; the evidence therefore says host_attested_operator_claim, not hardware-key attested. Exportable software PEM custody exists only behind BOTH --edge-key-provider software --edge-allow-software-key and an explicit segment-policy allowlist entry. Its durable evidence says file, key_exportable=true, and host_attested_software_exception; the console renders that exception rather than borrowing the host TPM's stronger label. Neither hardware path silently falls back when its device/session is absent. Issuances made while the host is unreachable are invisible until its journal reconciles, and the console panel says so rather than rendering silence as inactivity. A host agent run with --edge-journal-upload sends the journal over its mutual-TLS channel when it connects, after a heartbeat that follows failed ones, and when the journal grows; the control plane reconciles it through the same path as the CLI, counting each certificate against the delegation of that host that issued it, as the agent. Only a host-role agent whose channel certificate names the delegation's host reaches it, and <journal>.uploaded records the acknowledged prefix, so only new entries travel. Without the flag the CLI or console reconcile stays the path. Renewal is opt-in twice: the segment policy's renewal_enabled (with renewal_window_days, default 7) and the host agent's --edge-renew. Inside the window the TPM key signs a fresh CSR, TPM2_Certify vouches for it over the new challenge, and the agent asks over its channel; the control plane holds the renewal to the mint's gate (the same attestation verifier, the segment's current roots, constraints and provider allowlist), requires the same delegated CA key, and mints a delegation with the old one's lifetime that names it in renewed_from. The 30-day bound stays: an expired, revoked or already renewed delegation does not renew, and a refused attestation leaves the delegation to expire. Renewal attests with TPM2_Certify, so it serves the TPM2 lane; a PKCS#11 or software delegation asks again through the attested mint. Proven against the TPM fixture end to end and, for the attestation itself, against swtpm. The exact TPM2, PKCS#11, and software-exception command/configuration lanes are documented in Disconnected edge CA key custody. Authority agreement (C4): the pipeline now has a PRODUCER. A scheduled round observes the store-backed authorities, and when two signed digests commit to different state the scheduler hands every disagreeing pair to a sink that builds the witness, signs it in the isolated signer, records it in the event ledger (with both signed digests on the event, so offline verification needs nothing else) and passes it to quarantine admission. An earlier audit found the opposite: rounds were never scheduled, witness.Recorder.RecordWitness had no production caller, and every ingredient of "we checked and your authorities agree" shipped except anything doing the checking — the collecting=false state and its wording exist because of that finding, and they still guard the zero-schedule deployment today. Authority agreement (C4, core reconciliation): GET /api/v1/reconcile/agreement and a Posture console panel report whether the configured authorities agree about what was issued, and where they do not. XREC already built the hard part — canonical records per authority, signed Merkle digests over them, and witnesses naming the exact differing subset — and had nowhere to say so: the drift projection accumulated every witness class per authority and was never exposed on the runtime, so no route could read it. The system could detect that two authorities disagreed and could not tell anybody. The design constraint is that SILENCE RENDERS AS AGREEMENT. "0 open witnesses" looks identical whether reconciliation found nothing, is unconfigured, or has consumed no events yet, so the surface refuses to show a reassuring zero it cannot stand behind. A deployment with no reconciliation runtime reports that fact and states plainly that it is NOT a report that the authorities agree; a configured but cold projection says its counts are not yet evidence; and every response carries the replay watermark beside the count, because a projection lagging the event log reports an old world confidently. Resolution time is a MEDIAN over a stated sample size — one witness left open over a weekend drags a mean into fiction, and a median over a single sample is not a trend. Scope, stated exactly: this surface is READ-ONLY. Remediation is deliberately not reachable through it, because a corrective operation runs only through a plan verified inside the signer, and a REST route that could start one would be a way around that verification wearing the same URL prefix as a status page. Authorities are ranked worst-first; an authority that has never been collected from does not appear in the table at all, which is why configured is a field rather than something a reader is left to infer from an empty list. Relay migration parity (E1, OPEN — four migrated, three architecture exceptions, six unimplemented): the connector catalog and Connectors console publish the thirteen-family source-plan denominator, not the smaller list the current relay binary happens to execute. The exact families are a10, AWS ACM, Azure Key Vault, Cisco, Envoy, F5, FortiGate, GCP Certificate Manager, Kemp, MySQL, NetScaler, Palo Alto, and PostgreSQL; PostgreSQL and MySQL make the plan's formerly vague "+ DB/API variants" phrase executable. Every row carries a closed disposition: four are migrated, three are open architecture_exception rows, and six are unimplemented. The control plane REFUSES a migrated family's deploy when the tenant has a network relay enrolled. Before this, the A3 role stamp reserved an appliance deploy for a relay by writing required_agent_role on the outbox row and nothing enforced it: the outbox claim query has no predicate on that column — it is read by ClaimAgentJobs when an agent asks for work, never by the dispatcher — while the dispatcher swept every connector.* row on a one-second ticker. The stamp was correct in the column and decided nothing. The refusal is CONDITIONAL on a relay actually being enrolled, and that condition is the design rather than a hedge. Refusing unconditionally would turn "this estate has not deployed a relay yet" into "this estate's appliance deploys stopped working", and would retire the only path with end-to-end proof through the served API — the DoD connector suite drives a10, cisco, kemp and netscaler through the control plane to their device doubles and enrolls no relay. So what is promised is narrower and keepable: if you run a relay, the control plane will not do its work behind its back. An estate with no relay deploys exactly as it did before. A failed relay-presence lookup DEFERS rather than falling through, because falling through is the direction that silently removes the guarantee. Migrated today: a10, kemp, netscaler, f5 — device proof, rollback, readback, a published support row, a relay deploy proven byte-for-byte against the device double, and the refusal above. F5 closes the last gate it needed: an F5 HA pair keeps its certificate objects in separate stores, so a deploy that reached only the active node reported success while the standby served the old certificate until a failover surfaced it as expired. The relay now drives an HAPair over both peers when a peer endpoint is configured — a deploy must reach BOTH or it fails, a rollback re-binds both, and a readback reports the pair serving only when both peers are bound to the deployed certificate; the two-peer deploy is proven end-to-end through relay.Execute against two device doubles. cisco migrated once a target could name its ISE node and certificate roles (node_hostname, certificate_roles; A122): its deploys then import fingerprint-named ISE system certificates bound to those roles, a rollback moves the roles back, and readback reports the certificate serving them; a target without a node keeps the legacy import and reports both as not configured. fortigate and paloalto migrated once a target could name the binding to move (bind_to; ssl_tls_profile with commit_admin): the same binding is what rollback re-points and readback asks about, and a target without it reports both as not configured. The parity surface and console now say this explicitly (cp_retained + a scope note) so "not migrated" cannot be read as completion; each family's support-matrix known limits state the same constraint. These rows KEEP E1 OPEN. Re-check on new PAN-OS / FortiOS / IOS-XE majors: a vendor API that grows an addressable installed object re-opens that family's migration through the same gates, not around them. Of the six families the source plan omitted, the three cloud stores now have relay execution (A59): AWS ACM, Azure Key Vault and GCP Certificate Manager carry network-relay constructors, a relay execution proof against their API doubles, and the control-plane refusal while a tenant has a network relay enrolled; where no relay is enrolled the control plane's egress-guarded client remains their executor, for deploys, previews and GCP's re-bind rollback alike. GCP Certificate Manager passes every gate and is migrated; ACM and Key Vault are the two architecture_exception rows because their APIs cannot express a re-bind or a readback (A58). Envoy, PostgreSQL and MySQL execute on co-resident host agents and stay unimplemented: the database connectors write the server's own TLS files and run a local reload, which is not relay work by nature, and Envoy is host work by default; an Envoy target that declares vantage: network is driven by a relay instead (A61) without that making the family migrated, since SDS cannot express a re-bind or readback. Their rows name those facts instead of disappearing from the denominator. The executor census remains a separate fail-closed list: publishing an E1 row does not let an agent claim work it cannot construct. Device-generated CSRs are a custody choice, not a migration gate. An F5 BIG-IP (a single device), NetScaler, A10, PAN-OS (with an SSL/TLS service profile) or Cisco ISE destination (with node_hostname and certificate_roles) can declare key_generation=appliance and pin a network relay: the appliance generates the key and a CSR, the control plane signs the CSR, and the key never leaves the appliance. Otherwise the control plane generates the key (under the tenant's control-plane key-generation switch), seals it to the deploy job, and the relay redeems and installs it. The appliance operations are proven against the management-API doubles, not hardware, and the PAN-OS certificate generate and CSR export shapes need a firewall to confirm; each renewal's key object stays on the appliance and nothing prunes them yet; a PAN-OS renewal picked up again after its certificate import stops instead of finishing, and the same holds for a Cisco renewal picked up after its bind. ISE names each request itself, so the relay keeps ISE's request ID for the job on the control plane before it asks for a signature and a retried attempt reuses that request; the ISE node must already trust the issuing CA. Readback answers for every migrated family: kemp, netscaler and a10 were listed as readback-capable before their connectors could read back, so their relay deploys reported an appliance that "did not answer"; the census is now checked against the code (internal/connector/readback_census_test.go), and a NetScaler certkey that no SSL virtual server uses is reported as installed but not bound. Served DR posture (J2): GET /api/v1/platform/dr-posture and trstctl platform dr-posture report when this deployment's backup was last VERIFIED — meaning its artifacts were re-hashed and matched — rather than when one was last taken. The distinction is the point: a nightly job that writes a corrupt file runs perfectly, and reading a manifest's recorded checksums back only proves the manifest agrees with itself, since one process wrote both in one pass. The failures this catches are silent — bit rot, a truncated copy, an artifact restored from the wrong directory — none of which change the manifest, and all of which are otherwise discovered during the restore that was supposed to save you. The posture reads TRSTCTL_BACKUP_DIRECTORY only: a set copied to the object store (--full-backup-upload) is verified when it is fetched back (--full-backup-download re-hashes every artifact against its manifest), not by this served check or the scheduled drill. Three states are kept apart because only two are actionable: no backup directory configured (not a fault — many deployments back up through infrastructure this product does not see), a configured directory that cannot be read (a real finding, and the state that goes unnoticed until a restore), and one read that did not verify. A green verdict reports how many artifacts it covers and how many carried no recorded checksum at all, because "verified" over two of eleven artifacts is not the same claim as over all eleven. An artifact with no checksum is reported unverifiable rather than verified, and an EMPTY backup is never verified — green on an empty set is the most misleading answer available. Restore drills (J2): a drill restores the COMPLETE manifest into an isolated target through the same full-restore function production recovery uses. It creates and later drops a real ephemeral PostgreSQL database, starts a private file-backed JetStream beneath a temporary directory, copies the separately-custodied KEK and backup-decryption key into that directory, and redirects every restored signer, certificate, audit, socket, and archive path there. The production deployment is read only. A drill with its own simplified restore would prove the simplified one works, which is the one nobody runs at 3am; a drill aimed at live targets could cause the outage it is meant to rehearse. After event replay and both projection rebuilds, the drill imports and re-exports every RecoveredFromPostgresBackup table and compares exact table counts. It then starts the shipped signer over the recovered sealed key store, assembles a recovered control plane, and requires its real /readyz PostgreSQL, JetStream, and signer probes to pass. The attestation carries event count, independent-row/table counts, restored artifact names, and each health result. restored is impossible unless the full set and every runtime predicate pass; a missing required artifact or an event-only replay is failed. The attestation records FAILURES as readily as successes, because without one "no attestation" is ambiguous between "nobody ran a drill" and "the drill failed". A restore that completes having replayed ZERO events is recorded as a failed drill, not a fast one — it proved nothing and is the result most likely to be mistaken for reassurance. A backup that does not verify is not restored at all, so the cause reads as a sentence rather than being buried in a restore error. RPO is measured from the backup's own manifest, so it is what was achievable with the artifacts on disk rather than what was configured — the two diverge precisely when a backup job has been quietly failing. RTO is a FLOOR and the attestation says so in its own signed bytes: real recovery also includes provisioning, networking and people, and that caveat is signed alongside the number so it cannot be edited off a document whose signature still verifies. A deployment that has never drilled serves no drill rather than a zero-valued one, since zeros render as an instant, complete recovery. Each scheduled result now becomes a versioned backup.restore_drill.recorded event per live tenant. The deployment-wide attestation is signed through the isolated signer's purpose-constrained audit-export handle with the exact restore-drill/v1 evidence domain; the signed body includes the complete attestation, recovery-objective thresholds, derived alert reason, signer key id, algorithm, and public JWKS. The projector verifies against the deployment-trusted key before writing an immutable backup.restore_drill evidence row, so changing an outcome, detail, limitation, metric, threshold, required alert, signer, or verification key is rejected. last_drill is derived from the newest durable row and drill_history exposes the newest 50 through the authenticated tenant API, generated SDKs, trstctl platform dr-posture, and the Platform console. A cold projection rebuild reproduces the history; it is no longer a process-memory pointer. The console shows signature status and signer identity and downloads the whole portable signed JSON evidence object. The drill RUNS: RunRestoreDrillScheduler is registered as a runtime worker and fires on backup.drill_interval (daily by default, "0" to disable, and an unparseable value fails startup rather than silently defaulting; the default is resolved at config-parse time so a zero reaching the scheduler can only mean the operator asked for none — an earlier cut applied the default in the scheduler too, which made the documented "0" run the drill daily instead of disabling it). It is worth recording that the first cut of this work did not have that. The drill, the attestation, the ephemeral target and the endpoint were all built and all tested, and nothing in the running binary ever called any of it — the fourth instance in this backlog of a capability that is complete, tested, documented and unreachable, after D2's VerifyAddress with no producer, B2's endpoint.renew with no enqueue, and B5's custody projection that was never written. Unit tests cannot catch it by construction: they are the caller that production lacks. Two tests now assert the wiring itself, and the DR panel on the Platform page reads the attestation the scheduler writes rather than a second source that could agree with nothing. A deployment that CANNOT drill — no external PostgreSQL to build a throwaway database in — records a skipped attestation saying so, rather than reporting that it has never drilled. Those are different facts, and the second is the one an operator would read as an oversight worth chasing. Failed and skipped outcomes create critical/warning alerts respectively. A successful restore whose measured backup age exceeds backup.drill_rpo (default 24 hours), whose isolated restore time exceeds backup.drill_rto (default one hour), or both, creates an objective-breach warning. The signed evidence row and its idempotent notification.restore_drill outbox intent are projected in one tenant transaction, so a crash cannot commit a red drill while silently dropping its alert. The scheduler logs structural/signing failures instead of discarding its returned error. Enrollment diagnostics (I4): refused ACME, EST, SCEP, and AD CS enrollments now produce a diagnosis naming the protocol, the step that failed, a cause from a CLOSED set, and a remediation. Each protocol emits at its shared refusal choke point, so a new refusal cannot silently bypass the recorder. AD CS keeps only an allow-listed stable HRESULT (0x80094800 or 0x80094801); its free-form upstream status is never retained because IIS or a proxy can echo submitted credentials. The design constraint is that it must DECLINE rather than guess. A tool that says "your DNS record is missing" when the responder was unreachable sends an operator to the zone file for an hour, and they will doubt it afterward on the occasions it was right. So the classifier matches only unambiguous evidence — RFC 8555 problem types, EST status codes, the two stable AD CS error phrases — and returns unknown for everything else, which is a first-class answer carrying NO remediation rather than a fallback. SCEP's badRequest is deliberately unclassified: it is the value a SCEP server returns for most refusals, so mapping it to any single cause would invent a diagnosis from a value that carries none. "Unreachable" is detected from typed network errors, never from error wording, so an error that merely mentions a connection is not reported as one failing. All three properties are mutation-verified. Diagnoses are served at GET /api/v1/enrollment/diagnostics, available as trstctl enrollment diagnostics, and shown on the Protocols console — where an unclassified failure renders "cause could not be established" rather than an empty remediation cell, because a blank reads as a rendering bug and the honest answer is a real one. Repeats collapse only when the stable diagnostic id names the same protocol, operation, identity, and endpoint. Two devices refused at the same step remain two rows; a retry storm for one exact operation remains one counted row. Each observation is a versioned immutable enrollment.diagnostic.observed event whose envelope carries the request tenant. PostgreSQL projects the exact refs into a FORCE-RLS table and retains the newest 200 operations per tenant. Version-1 history replays under a deterministic legacy: id; version 2 carries the exact id and refs. Restart, snapshot restore, and full replay therefore reproduce both the refusal and its verification link instead of clearing a process-global map.

    POST /api/v1/enrollment/diagnostics/{id}/prove-fixed requires certs:issue and an Idempotency-Key. It appends an immutable queued receipt, then projects the link and writes one network-role endpoint.verify intent in the same PostgreSQL tenant transaction. If the event append wins but that transaction rolls back, startup reconciliation recreates the exact intent from the event. The intent contains one explicit deployment-target address and server name; it never guesses SAN:443 and never treats the enrollment server as the workload endpoint. The operator must first retry enrollment successfully. A newer active certificate issued after the refusal supplies the expected fingerprint; without that evidence the action returns 409 and queues nothing. A real relay performs the TLS handshake; verified means the signed transcript reached that endpoint and matched the expected certificate. diverged and unreachable remain red, and a queued row never reads as success. The result is available through GET /api/v1/endpoints/verifications/{id}, its matching CLI command, and the Protocols console's signed-evidence link. While a row is queued, the console performs at most 30 bounded read-only refreshes and stops immediately when a terminal signed result appears.

    The ordinary offline support bundle remains tenant-data-free by default. An operator can explicitly add authorized, redacted aggregates with trstctl support-bundle --include-enrollment-diagnostics plus TRSTCTL_URL and TRSTCTL_TOKEN. The fetched shape can contain only protocol/cause/actionable/count aggregates and an unknown count; tenant ids, times, diagnostic ids, and exact operation/identity/endpoint refs cannot fit in its type. The fetch rejects redirects, bounds the body to 64 KiB, and never copies an upstream error body or bearer token into its error. Relay revocation cache (R3): a network-role relay started with --revocation-cache-config serves issuer-specific CRL and OCSP paths to relying parties in one named segment. The JSON file declares a bounded listener, the segment, and multiple issuers; each issuer points at a public certificate file and may declare a CRL GET path, an OCSP POST path, or both. The old --crl-cache-listen flags remain a single-issuer CRL compatibility path and now require --revocation-cache-segment. Revocation checking is the part of PKI that fails quietly — a client that cannot reach a distribution point usually proceeds rather than refusing — so a segment with no route silently stops checking, and nobody finds out until a compromised certificate is used. The relay SIGNS NO REVOCATION OBJECT: it holds the CA's signed bytes and hands them over, so a compromised relay can withhold a CRL (visible: the fetch fails) but cannot forge one (which would not be). It verifies a fetched CRL against the configured issuer before caching, which is not the relay adding trust — a relying party checks the signature regardless — but the relay declining to store what no client would accept, such as a captive portal's login page. The OCSP path accepts only bounded application/ocsp-request POST bodies. It proves that the request's issuer hashes match the configured issuer, forwards the exact request, and accepts a response only after the issuer/responder signature, requested serial, nonce equality, thisUpdate, and nextUpdate all validate. Nonce-free responses are cached in a 4,096-entry per-issuer bound; nonce-bearing responses are never reused because a nonce binds one response to one request. Expired entries are evicted first. The relay returns 503 with no OCSP response bytes when the upstream is unreachable or its answer is stale, malformed, for a different issuer/serial, or incorrectly signed.

    IT FAILS CLOSED ON STALENESS, and that is the property the feature exists for. A stale CRL is dangerous precisely BECAUSE it still verifies: nextUpdate has passed and the signature is good, so a relying party accepts it and trusts a certificate revoked yesterday. Past nextUpdate the relay serves 503 and no bytes — not the stale list with a warning header, because a client that receives bytes will use them. --crl-cache-grace can extend the window, defaults to zero, and is deliberately an operator decision: serving a list the CA declared expired is a risk only they can weigh. A CRL whose number went BACKWARD is refused, since replaying an older list is how a revoked certificate comes back to life. A failed refresh keeps a still-valid cached list rather than discarding it. Both fail-closed properties are mutation-verified. Before opening its steady-state channel, the relay signs a normalized metadata-only row per segment, issuer, and protocol with the same certificate key used by mTLS. The server verifies the certificate signature, tenant/agent binding, timestamp, network role, and monotonic order, then event-projects it. GET /api/v1/revocation/caches and Protocols → Revocation cache by segment show fresh, stale, empty, and error separately. Upstream URLs, issuer bytes, OCSP requests/responses, CRL bytes, and credentials never enter the heartbeat, event, API, or console. A missing report remains unobserved rather than green. The repository proves multiple issuers, signature-valid local CRL/OCSP clients, nonce policy, stale refusal, signed assembled-agent heartbeat, tenant isolation, replay, snapshot, and cold rebuild against controlled responders. A domain-joined Windows AD CS deployment using its real CDP/OCSP endpoints remains external lab evidence rather than a repository claim. Enrollment proxy for dark segments (A4): a network-role relay started with --enroll-proxy-listen, --enroll-proxy-segment, --enroll-proxy-public-url, and --enroll-proxy-upstream serves ACME, EST and SCEP on the LAN so hosts and devices with no route to the control plane can enroll through the one outbound pipe the relay already has. Stock clients (certbot, sscep, estclient) point at the stable public URL unmodified. The agent listener is HTTP; the public URL is HTTPS, so the segment's TLS-terminating load balancer terminates that public TLS connection before selecting a relay. Every redundant relay for one segment uses the SAME public URL. That detail is load-bearing for ACME: directory, account, order, authorization, challenge, finalize, and certificate resources are absolute URLs, and a different authority would send the stock client outside the relay path or strand it on the process that died. IT MAKES NO TRUST DECISION, and that is the design rather than a caveat. The relay sits inside the customer's network, which is where an attacker with a foothold already is, so a proxy that interpreted a challenge, cached an authorization, or spoke with its own agent identity would hand that attacker the ability to mint certificates. Requests are forwarded byte-identical — an ACME JWS is signed, so a proxy that reformatted it would invalidate every request — and the proxy attaches NO credential of its own: the client authenticates as itself through EST's TLS client certificate, SCEP's challenge password or ACME's account key, exactly as it would reaching the control plane directly. Both properties are mutation-verified. The control plane independently requires every ordinary ACME outer JWS protected url to equal that stable public request URL byte-for-byte before consuming its nonce. Missing URL, path, authority, and scheme differences fail unauthorized; the assembled two-process relay journey proves the relay preserves the same URL a stock client signed through failover. The forwarded paths are an allowlist, not a catch-all: a segment able to reach /api/v1 through a relay would hold the control plane's entire administrative surface, which is a far larger grant than "devices here can enroll". Two kinds of failover stay separate. Several --enroll-proxy-upstream values let ONE relay choose another control-plane endpoint after a transport failure. Several relay processes behind the SAME public URL let the segment frontend select a secondary relay after the primary process dies. Both work because the proxy is stateless — an ACME order lives in the control plane, not in a relay — so the client retries through another path and its order is still there. A 5xx FROM the control plane is NOT a failover trigger: it is an answer, and retrying elsewhere would ask a second endpoint the same question while hiding the real error. Every heartbeat preserves the relay's segment, public URL, verified, unavailable, and not-yet-verified upstream counts, forwarded/refused/failure counters, and last-forward/last-upstream-failover times in the immutable event stream. The authenticated agents API and Protocols console group those certificate-bound rows by segment AND public URL. That lets an operator distinguish no report, proxy off, no verified upstream, degraded upstreams, one relay, and actual relay redundancy. Counters reset when the relay process restarts; the projected timestamps remain durable evidence of the last observed activity. Third-party connectors in the relay (E4): a relay started with --connector-plugin-dir executes signature-verified WASM connectors inside the customer's network, under a capability grant its own operator sets. The control plane never loads partner code. Every guarantee is re-proven in the new location rather than assumed to have traveled with the code: an unsigned, tampered, or untrusted-key module is refused AT LOAD and fails the whole runtime rather than being skipped — a relay serving a subset of its configured connectors would be missing precisely the one somebody tampered with. An out-of-grant operation is denied at runtime and FAILS the deploy, so a module reaching outside its grant cannot report success. Both properties are mutation-verified. A module reads its deploy's public inputs (input.read: the certificate, the fingerprint, the target and its configuration) and asks the relay to make its HTTP calls (net.http): the relay adds the target's credential, splices the private key into the body by reference, and removes every redeemed value from the reply, so the module deploys as the appliance's administrator without holding the password or the key. The control-plane plugin path does not offer these two capabilities; third-party connectors that call appliances run on relays. Everything is operator-owned: the modules, the publisher keys, the pinned build digests and the capability grant. None is read from the module or pushed from the control plane, because a publisher who could widen their own grant by editing a file they ship would make the sandbox a formality. A directory with no trust keys is refused, and an unset grant is refused rather than defaulted — a grant nobody set and a grant that permits nothing are indistinguishable at that point, so the reading that cannot surprise anyone is the one that refuses. AUD-34 adds a signed, metadata-only census to that boundary. The census comes from the exact verifier-and-loader that instantiated the module; it names the stable plugin name, verified module and publisher fingerprints, the fixed network_relay_wasm execution context, and the effective grants plus normalized constraints. The relay signs the tenant ID, its certificate common name, that normalized list, and the issued-at time with the same key used by its mTLS certificate. The server rebuilds both identities from the authenticated certificate, verifies the signature, and refuses a tampered, stale, or cross-tenant report. A host-role certificate cannot report relay plugins. Accepted reports enter the immutable event stream and rebuild the newest tenant-isolated projection after restart. The connector catalog API and the Connectors console show each reporting relay separately. A signed empty census means “this relay loaded no third-party modules”; no report means an older agent cannot make the claim. The census is also what routes work (A63): a deploy or preview for a connector no shipped family owns is stamped for the network role when a relay in the tenant has reported carrying that plugin, and the control plane defers the row to it; without such a report the control plane's own plugin surface remains the executor. The report contains no module bytes, publisher keys, credentials, or secrets. The control plane still does not distribute modules or operator trust keys. Third-party connectors also never receive redeemed credential material; a module that could read an appliance password would make the sandbox decorative. Live appliance readback (E2): after every appliance mutation the relay now asks the device what it actually has, alongside D2's TLS handshake against the served listener. The two are not redundant and only together locate a fault: a handshake showing the old certificate with a readback showing ours-and-bound means the VIP is fronted by something else; with a readback showing theirs it means the binding never moved (the deploy patched the wrong profile); with a readback showing nothing it means the deploy did not take. A handshake failure alone reads as "the deploy failed", and the most common cause is the middle case — a binding the deploy never touched, on a device that reported success at every step. A readback that contradicts a passing handshake DOWNGRADES the outcome to verify_failed, because a handshake can pass against a second listener while the object this deploy installed is not the one bound. Verdicts are a closed set: serving, installed_not_bound, diverged, absent, unknown — and unknown (the device named an object but not which certificate it is) is NOT a pass. Readback is available only where the API can address an installed object separately from uploading one: f5, kemp, netscaler, a10, and fortigate / paloalto / cisco targets that name their binding (bind_to; ssl_tls_profile; node_hostname with certificate_roles). A fortigate, paloalto or cisco target without its binding cannot be asked, which is reported as no readback rather than as a clean one. The readback travels inside the deploy's signed verification report, as its verdict and sentence (appliance_readback), and the control plane admits verified only when the handshake passes and the readback confirms; the verification receipt names a contradicting readback (appliance_readback_<verdict>) and keeps the device's sentence. Earlier builds appended the sentence to that report's JSON, so a deploy with both a verify address and a readback answer was refused as undecodable. Capability disclosures (K3): the connector support matrix and the per-issuer capability matrix are both generated by one tool from the censuses the API serves, with a Go test failing on drift in either. The issuer matrix gained an issue_proven column that is deliberately SEPARATE from issue, because "we implement this authority's API" and "we have run an issuance against it" are different claims and a table showing only the first states the stronger one everywhere. All thirteen advertised authorities now have high-fidelity protocol issuance proof in TestDODExternalCAUniversalProductionAssembly: each driver is constructed by the production server assembly, talks to its own nonce-bound authority-protocol substrate in the expected wire order, and returns a chain that is independently verified. That proves the adapter and custody path. It is not live vendor-account certification and does not prove a customer's policy, entitlement, network, appliance, or service version; operators must still run one scoped issuance against their configured account before depending on it. The revoke and unattended-DV columns remain separately census-checked against the implementations. Every row names the tests backing it, and a manifest-derived guard fails both when a row outruns its proof and when new executable proof remains incorrectly labeled "not tested." Support matrix (E3): docs/features/connector-support-matrix.md is GENERATED from the same census the API serves, and a Go test fails if the two diverge — so a capability cannot be removed while its published row survives. It deliberately publishes no firmware compatibility range: nothing in this repository runs against a physical or vendor-hosted device, so a version claim would be unbacked, and it is precisely the line an operator would plan a migration around. What it publishes instead is the API or local execution contract each connector speaks, the operations exercised by repository tests, and the family's known limits. Eleven rows are proven against faithful management-API doubles: the seven appliances, the three cloud stores and Envoy's SDS management surface (A59, A60). PostgreSQL has no device API to double; its proof instead drives the connector against the pinned bundled PostgreSQL runtime started with TLS on, and a real SSLRequest handshake sees the renewed certificate after the allowlisted pg_ctl reload. MySQL keeps sandboxed filesystem and process operations as its only proof: no MySQL server is bundled or available to the repository tests, and its row says so. A guard fails if a version claim appears, and all thirteen rows explicitly say no external target was tested. Rollback, restated after that work: cisco re-binds only through ISE's system-certificate calls, which a target selects by naming its node and certificate roles (A122); the legacy import is a single call that both uploads and installs, so a cisco target without a node still cannot re-bind. fortigate and paloalto re-bind only when the target names the binding to move (A44), and the same is now true of gcp-certificate-manager through a certificate map entry (A58). aws-acm and azure-keyvault stay absent from the census for the same kind of reason: ACM re-imports in place under one ARN and keeps no previous version, and Key Vault keeps versions but serves the newest by name with no call that makes an older one current. The census tests pin each absence in both directions and the refusal names the family's reason. buildRunDeps constructs the operator-selected production registry; strict target schemas bind endpoint/filesystem/process and same-tenant secret references before a served issue/deploy route can enqueue work. Credential-bearing connector.deploy payloads exist only while needed, travel through the durable outbox, and are wiped after delivery. The acceptance proof requires provider-specific mutation plus independent external readback before a connector counts as served. Inventory: nginx, Apache, Caddy, Envoy, IIS, HAProxy, F5, NetScaler, A10, Kemp, Cisco, FortiGate, Palo Alto, Postfix, Traefik, AWS ACM, Azure Key Vault, GCP Certificate Manager, Java keystore, PostgreSQL, MySQL, RabbitMQ, Elasticsearch, and Tomcat.

  • Discovery control plane + relay-owned network/SSH, cloud-certificate, CT-log, and drift execution: the running binary serves discovery sources, schedules, and runs under /api/v1/discovery/* — create/list a source, create/list a schedule, queue a run (idempotent, deduplicated by Idempotency-Key), read runs and findings (keyset-paginated), and read GET /api/v1/discovery/monitoring for the centralized continuous-monitoring view across sources, schedules, last runs, findings, and inventory counts. GET /api/v1/certificates/health and trstctl certificates health serve the estate-wide expiry/source dashboard over the same inventory projection, including certificates issued elsewhere and later imported or discovered. Queuing a run is an immutable discovery.run.queued event. For network and SSH sources the same transaction resolves the bounded target list and stamps the declared segment, required network role, and optional exact relay ID into both the event and outbox command. The control-plane dispatcher refuses to dial those targets and leaves the row claimable. An enrolled relay executes the sweep on its bounded worker lane, signs the result receipt, and the control plane validates the command/report binding before projecting metadata-only findings, counts, segment freshness, and the executing agent ID. For a cloud_certificate source the worker executes AWS ACM, Azure Key Vault, and GCP Certificate Manager enumeration through credential references — served cloud-certificate discovery execution. For a ct_log source the worker polls configured RFC 6962 log fixtures or public logs, checkpoints each log, and records unexpected issuance as ct_unexpected_issuance findings — this is served through the served discovery worker, queuing notification alerts the same way expiry alerts do. CT monitoring is a headline capability on the Discovery workspace, not a footnote: one surface carries the watched-domain and log watchlist, per-log checkpoint/health state (so you can see whether a log succeeded, failed, or has never been reached), retired audit history that is no longer polled, unexpected-issuance findings with certificate detail, and a one-click hand-off to the rogue-certificate remediation path. It previously appeared there only as a single count shared with drift detection, so the capability was effectively unfindable. It covers only the domains and logs configured — a domain you have not listed, or a log you do not poll, produces no finding, and the surface says so beside the counts so an empty list is not read as an all-clear. Replacing a named watchlist retires URLs absent from the new source configuration in the source event's same tenant transaction; one failed active log does not discard a peer's successful checkpoint or finding. The console polls queued/running CT runs to completion, refreshes per-log checkpoints in place, follows the parent Discovery refresh, and shows a bounded credential-redacted diagnostic for a failed discovery run. It also only sees what a CA chose to log, which in practice means public issuance. For a drift source the worker compares configured credential paths against expected fingerprints/permissions and records credential_drift findings through the same served discovery worker. For an ssh source the worker runs a non-invasive SSH host-key scan through the discovery outbox worker, applying the same reserved-address guard as network scans, recording metadata-only ssh_key findings (fingerprint/key-type/ location) and never authenticating or storing private key material. A manual source records its supplied findings.

  • CBOM scan and migration inventory: POST /api/v1/cbom/scans runs the cryptographic bill of materials scanner against TLS endpoints and host config files, records cbom.asset.observed events, and projects tenant-scoped crypto_assets. GET /api/v1/cbom/assets returns the inventory plus migration targets and migration_progress. With PQC in the core, the targets are the concrete FIPS 203/204/205 algorithms and migration_progress counts which assets are already post-quantum-ready.

  • Credential-compromise incident execution is part of Core: the old POST /api/v1/incidents/executions single-identity mutation refuses with conflict and points to the H2 fleet route; it cannot prove exact trust scope, trust-before-leaf ordering, or signed live health before revocation. Historical evidence remains readable via GET /api/v1/incidents/executions{,/{id}}. Automated remediation playbooks are also served without a commercial license: GET /api/v1/remediation/playbooks, POST /api/v1/remediation/playbooks/{id}/runs, and GET /api/v1/remediation/playbook-runs{,/{id}} cover revoke, rotate, and NHI right-size. Revoke and rotate run the identity lifecycle gate; they cannot carry an exact dual-control approval yet, so under dual control they refuse and the operator uses the identity transition route, which can. Rotate revokes the original only after the replacement's signed, verified delivery, so it serves an X.509 identity bound to a deployment target whose delivery an enrolled host agent or network relay verifies and signs. A control-plane connector delivery (a cloud store while the tenant runs no relay) carries no such signature: that rotation keeps the original valid and raises a rotation-replacement-unverified finding at its deadline, and the operator completes the change through the identity transitions. Owner-driven self-remediation is served through GET /api/v1/remediation/owner-actions and POST /api/v1/remediation/owner-actions/{id}/accept: a bound owner can accept the CAP-POST-01 least-privilege recommendation, and trstctl records the same remediation.playbook_run.recorded evidence plus a connector.right_size outbox intent. When the operator configures the matching tenant/connector binding, the shipped dispatcher authenticates to the entitlement service with a tenant-secret reference, sends the stable outbox idempotency key, applies the requested scope removal, reads the effective scopes back independently, and advances the queued connector receipt to delivered or failed. An absent binding and every unknown connector.* outbox kind fail closed. The rule is simple: a queued receipt is not external-effect evidence. Only the verified delivered receipt is. SIEM/SOAR/chat/ITSM response dispatch is served through POST /api/v1/incidents/response-integrations/dispatch, which records response.integration.dispatched and queues Splunk HEC, Jira issue, configured Slack notification, and ServiceNow Table API outbox rows in the same event-backed workflow. The served boundary is dispatch plus evidence. The separately packaged Splunk app supplies two disabled audit-feed correlation searches for renewal failures and revocation bursts, with tenant/index scope placeholders and immutable event-ID deduplication. Its owned scaffold evaluates the actual query text ad hoc and tests native audit-feed delivery and retries; it does not qualify Splunk's parser, scheduler, RBAC or product version. Those vendor-runtime checks remain unverified without an authorized Splunk environment. Custom incident-response correlations, Jira automation rules, Slack app/channel installation, arbitrary third-party SOAR playbook execution, and bidirectional ServiceNow ticket-status sync remain operator configuration. Fleet-wide re-issuance is served separately at POST /api/v1/incidents/fleet-reissuance-runs: it freezes exact certificate/SPKI H1 trust consumers and ordered H2 member bindings before work, installs replacement trust before host-CSR issuance, waits for lease-bound signed trust/live receipts, revokes each exact predecessor only after verification, and seals terminal success/rollback as a compact JWS. The separately authorized game_day mode structurally refuses production owner/target environments. It has pause/resume/current-cohort rollback and evidence export routes under /api/v1/incidents/fleet-reissuance-runs/{id}, matching trstctl incidents fleet-reissuance * CLI commands, and the /incidents console. Online break-glass at POST /api/v1/breakglass/issue is conditionally served when breakglass.online_enabled=true: production buildRunDeps binds the configured CA and public key to one persisted, purpose-constrained dual-control signer handle. The caller first opens an exact-request-bound ceremony, and distinct approvers use their own authenticated tokens at /api/v1/ca/ceremonies/{id}/approvals; the issue request carries no approver names. Only authenticated immutable ca.ceremony.approved event actors in the configured roster count toward quorum, and the ceremony is consumed once with breakglass.issued. Break-glass recovery readback at GET /api/v1/breakglass/issues and GET /api/v1/breakglass/issues/{id} returns bounded tenant receipts and the exact signed online bundle to callers with certs:issue. A lost response does not require reissuance; trstctl does not retain the CSR subject private key. Offline break-glass reconciliation remains served at POST /api/v1/breakglass/reconcile. A disconnected signer uses its sealed, offline-only CA and a pinned public kit; two distinct custodians sign the exact CSR-bound request. Recovery checks their signatures, tenant, deployment, policy epoch, recovery CA key and leaf before recording the public proof as a breakglass.issued audit event. The disconnected runbook gives the setup, outage and recovery procedure.

  • Real X.509 issuance: transitioning an identity to issued mints a leaf certificate from the assembled CA (its key held in the out-of-process signer) and records it in inventory. Exercised end to end in CI.

  • Attested X.509-SVID issuance: workload owners with certs:issue can self-serve attester trust sources at /api/v1/workloads/attester-trust-sources (create, list, get, replace, rotate, revoke, delete) before calling POST /api/v1/workloads/attested-issuance. The route verifies a workload proof (aws_iid, azure_imds, gcp_iit, github_oidc, k8s_sat, or tpm) against tenant trust material or configured process defaults, signs a short-lived X.509-SVID in the isolated signer, records certificate.recorded, binds attestation.bound, and returns the certificate plus verified subject metadata. Mutations require Idempotency-Key, reads and issue are tenant-scoped, forged proofs fail closed, and offboarded trust sources mint nothing. A tenant/source command is recorded before a trust change reaches the event log; pending changes deny old proof while exact-event recovery completes. Completed commands retain a request-bound response snapshot for a lost-response retry, including a retry after a newer trust change. Subject erasure removes any affected response snapshot but preserves its opaque replay fence.

  • Authentication and RBAC via scoped API tokens (Authorization: Bearer), multi-tenancy with PostgreSQL row-level security, and a tamper-evident audit chain. A fresh boot fails closed (every route 401s until a credential exists); mint the first tenant-scoped token with trstctl token create --tenant <uuid> (writes through the store, prints the token once). OIDC, SAML, and LDAP / Active Directory login are served by the binary when their respective auth.*.enabled flags are set — see "Single sign-on" below — under the same RBAC and per-tenant scoping as an API token, with each user mapped to its real tenant. API-token auth remains the default when SSO is disabled. OIDC stays core; tenant SAML and LDAP require Enterprise SSO, inherited by Provider.

  • SCIM 2.0 provisioning requires Enterprise SSO and auth.scim.enabled — see "SCIM 2.0 provisioning" below for the route and event detail.

  • Transport security (TLS), idempotency and the outbox, observability (/metrics, /readyz, W3C trace headers), bulkheads plus per-tenant rate limiting, backup/restore plus disaster recovery, and safe schema migrations.

  • Protocol parity hardening: served ACME supports the explicit ACME trust_authenticated profile mode for scoped EAB-bound internal issuance, plus account-keyed order/hour and concurrent-order limits. Served EST includes /serverkeygen, RFC 9266 tls-server-end-point binding, profile PathID dispatch, and an mTLS sibling route when configured. Served SCEP includes the SCEP Intune JWS challenge gate with tenant/CSR binding and durable single-use nonce rejection across control-plane restarts, per-profile RA material, and per-device rate limiting. The MDM SCEP control surface serves /api/v1/mdm/scep/status, policy CRUD, challenge-rotation evidence, CLI commands, and Protocols UI telemetry. Live SCEP validator trust anchors resolve enabled policy secret:// references per decision; protocols.scep.intune_challenge is a bootstrap fallback. The selectable hmac-dynamic and Jamf modes still lack a served policy resolver. Enabled policies in those modes are now rejected by preview and create/update; disabled policies can be staged but do not authorize enrollment. A tenant-bound validator, durable replay fence, credential configuration path, stock client replay, and cold recovery remain required before enabling these modes.

  • Revocation hardening: RFC 5280 named revocation reasons, bulk revoke routes, delegated OCSP responders, OCSP nonce echo, nonce-free OCSP response caching, and CRL ETag / If-None-Match caching. CT precertificate/final-certificate submission is served through POST /api/v1/revocation/ct-submissions and the ct.submit outbox worker.

  • NHI decommissioning: POST /api/v1/nhi/decommission and trstctl-cli nhi decommission resolve departure, vendor-term, and inactivity signals to tenant-local managed NHIs, then drive event-sourced lifecycle revoke/retire transitions with per-item evidence.

  • NHI posture — metadata-only, drawn from the unified NHI inventory; every route below reports findings with recommendations and has a matching trstctl-cli nhi posture <name> command. Automatic action is opt-in per tenant: an nhi_posture rule in the active lifecycle policy module may ask the leader's lifecycle sweep to revoke a managed identity (through the lifecycle gate, as system:nhi-posture, with RFC 5280 cessationOfOperation) or right-size its usage-backed unused grants (the right-size playbook's connector intent), at most 25 actions per tenant per sweep under the automation switch and maintenance deferral; each action or gate refusal is a playbook run naming the rule and the module digest, and GET /api/v1/nhi/posture/automation shows the plan. Under dual control an automatic revoke is refused and retried after 24 hours; the operator revokes through the identity transition with an approval:

    • Shadow (GET /api/v1/nhi/posture/shadow): unmanaged, unregistered, and ownerless external NHIs from discovery findings; the Discovery findings table lets an operator claim a finding and run rotate/revoke/decommission/ remediation against the prefilled identity.
    • Over-privilege (GET /api/v1/nhi/posture/overprivilege): granted scopes/permissions/roles compared with observed usage; rows without usage evidence are not classified as excessive scope.
    • Stale (GET /api/v1/nhi/posture/stale): stale/dormant activity, unused credentials, and orphaned records; finding-backed rows hand off to revocation/decommission/remediation, and POST /api/v1/nhi/posture/stale/handoff (trstctl-cli nhi posture stale handoff) hands up to 100 rows to a new accountable owner in one step: the ownership assignment, each row's previous owner and owner status in the response, and an ownership.handoff notice to the new owner and their escalation chain on the ownership notification channel.
    • Static credentials (GET /api/v1/nhi/posture/static-credentials): long-lived credentials, static lifecycle markers, no-expiry credentials, and overdue rotation age; managed rows can start the served rotate/revoke/remediation workflow prefilled.
    • Exposure (GET /api/v1/nhi/posture/exposure): internet-exposed NHIs, public endpoints/callbacks, plaintext transport, weak authentication, missing network policy, wildcard reachability, and insecure-deployment markers; URL query strings are sanitized and no credential values are returned. POST /api/v1/nhi/posture/exposure/probe (trstctl-cli nhi posture exposure probe, discovery:write, idempotent) probes those public endpoints and callback URLs live from a network relay: up to 50 endpoints per probe, most severe records first, as one discovery run under the product's exposure-probe source. The relay connects only to public addresses, checked on the address each socket would connect to, so a name that resolves to a private or loopback address is refused, not probed. Each endpoint becomes a finding (nhi-exposure-probe) with its outcome: a served TLS certificate, an unparseable one, a listener that accepted the connection without TLS, a non-public address, or no answer. The exposure posture then shows each record's latest probe (reachability) and counts probed and answering records. The control plane never dials the endpoints itself. A probe no network relay can claim (none with a fresh heartbeat, or endpoint.verify missing from agent_channel.claimable_job_kinds) is refused with 409 naming what to fix, rather than left queued. Provider-native attack-path validation (whether a cloud's routes and security groups allow the path) is what comes next.
  • Rogue and non-compliant certificate posture: GET /api/v1/revocation/rogue-certificates and trstctl-cli revocation rogue-certificates combine unexpected CT findings with active certificate inventory policy checks (weak keys, expired active state, over-long public-TLS lifetimes, missing owners/issuers) and, when ca.policy.enabled is on, the tenant's own certificate_posture rules from its active lifecycle policy module — metadata-only, returning projection evidence refs rather than PEM or key material. A posture rule is ordinary Rego in the same module (certificate_posture contains {"rule": ..., "severity": ..., "reason": ...} if { input.action == "certificate_posture"; ... }) over input.certificate (subject, issuer, serial, fingerprint, DNS names, key algorithm, validity, lifetime and days-until-expiry, status, source, owner, CA, deployment location); each violation appears as customer_policy:<rule> with its severity and reason, the response names the module digest, and POST /api/v1/policy/dry-run with action certificate_posture previews a rule against sample facts. A rule that yields "remediation": "renew" makes the lifecycle scheduler queue a renewal (reason posture:<rule>) for a deployed identity whose served certificate fires it, under the automation switch, maintenance deferral and a 100-per-tenant sweep bound; certificates without a managed, deployed identity are reported, never remediated. A module without posture rules reports nothing, and a certificate whose evaluation fails is counted in policy_evaluation_errors rather than read as clean. CT coverage depends on configured monitored domains/logs. Each finding with a fingerprint lists installed_on: the connector targets trstctl delivered that leaf to, or restored it on by rollback (an adopted predecessor), and what can remove it there (host_purge through the enrolled agent, appliance_takedown through a network relay for f5, kemp, netscaler and a10, or none). POST /api/v1/revocation/rogue-certificates/takedown (trstctl-cli revocation rogue-certificates takedown) queues that removal with a reason; an appliance object the target still serves is reported takedown_bound and left in place until it is re-bound, and only taken_down or purged claims removal. Provider-native revocation remains separate; targets trstctl never delivered to, CT-only findings and cloud or FortiGate/PAN-OS targets have no executable takedown.

  • Compliance and inventory reporting: GET /api/v1/compliance/inventory-report and trstctl-cli compliance inventory-report return the CAP-OBS-02 reporting view (frameworks, report types, routes, evidence refs, inventory counts, tenant schedules). GET /api/v1/compliance/evidence-packs/{framework} and trstctl-cli compliance evidence-pack serve signed CAP-CMP-04 framework packs for eIDAS, NIST SP 800-53/CSF, CMMC 2.0, FedRAMP, NIS2, and the existing PCI/HIPAA/CNSA/FIPS/Common Criteria/CA-audit frameworks. The soc2 pack serves CAP-CMP-05 only when each CC6/CC7/CC8 prerequisite has exact tenant-scoped event/object evidence in the signed 90-day window; absent, stale, malformed, or wrong-tenant prerequisites are explicit gaps. The signed v2 manifest includes the tenant, window, event IDs/sequences/audit-chain digests, object refs, and missing prerequisites. CPA examination and trust-services scope remain residuals, and signature validity is not certification. GET /api/v1/compliance/nhi-report and trstctl-cli compliance nhi-report return CAP-CMP-06 NHI compliance mappings for NIST SP 800-53, NIST CSF 2.0, PCI DSS 4.0, DORA, ISO/IEC 27001:2022 Annex A, FedRAMP, CMMC 2.0, eIDAS, and NIS2. These are evidence mappings, not legal certification — operator scope, policy, authorization packages, qualified status, national transposition duties, and auditor sampling remain explicit residual attestations. POST /api/v1/compliance/report-schedules and trstctl-cli compliance report-schedules create record idempotent, event-sourced audit-export schedule definitions; GET /api/v1/compliance/report-schedules and trstctl-cli compliance report-schedules list read them back. The licensed governance signer and a private mounted audit.archive_dir are prerequisites to enable them. The local scheduler creates signed artifacts and event-backed run receipts with bounded retry and requeue. The archive stays the record of each report. A schedule's dispatch also announces every completed run: one compliance.report_ready notice on each named notification channel (email, webhook, Slack, Teams, PagerDuty, Opsgenie, SIEM; up to 8; each must be configured for the tenant) and one ServiceNow ticket on an operator-approved public binding. The notices and ticket are queued in the transaction that records the completion, restored by outbox reconciliation, and never sent twice; the run receipt names them. They carry the run, its SHA-256 and its download route, never the report body, so a receiver downloads the signed bytes with its own audit:read credential. A binding the operator withdraws later skips the ticket on later runs and the receipt says why. A Jira ticket and attaching the report itself to a notice are what comes next. Report files can contain tenant metadata. The full backup (--full-backup-dir) carries every completed run's report, encrypted, a full restore puts them back where their runs say and checks each digest, and the restore drill attests how many it restored; a log-only restore still cannot recover them. Reports are kept until removed: the opt-in audit.report_retention removes each completed run's report once it is older than the window, unless a person placed a legal hold on the run, and records the run expired only after reading the archive and finding the report gone. A removal the object store refuses, or acknowledges without applying, retries in the outbox and the run stays unexpired. The run receipt and its SHA-256 remain after removal; a backup taken before it still holds the report. With the archive object store under S3 Object Lock (object_store.object_lock_mode and object_lock_retention), every audit archive segment, report and backup set trstctl writes carries that retention, and the report retention reads an object's lock before removing it: a report the lock protects stays, and its run shows until when (a day at a time under a legal hold on the object). A person places an S3 legal hold on one archive object under the tenant's prefix (POST /api/v1/archives/legal-holds, archives legal-holds place) and releases it; the outbox sets or clears it in the bucket and the hold reads applied or released only after the bucket reads it back, or failed with the reason (a bucket without Object Lock). A report legal hold placed while the archive writes under Object Lock also holds the report's object, in the same event, and its release clears that hold.

  • Audit chain anchoring (J1): GET /api/v1/audit/export?format= serves the signed JWS bundle (default) plus NDJSON, CSV, Splunk HEC and Microsoft Sentinel record streams. The JWS response and browser download are one versioned JSON envelope containing the compact JWS, chain head, and complete anchor. NDJSON, Splunk HEC, and Sentinel end with a JSON chain_trailer; CSV adds the compatible trstctl_record and integrity columns and ends with an RFC-safe trailer row. Every saved form therefore retains the record count, archived-prefix predecessor hash, chain head, and complete anchor without depending on HTTP headers. When operators need continuous delivery rather than a download, tenant-scoped GET|PUT /api/v1/audit/feeds[/{id}] schedules bounded Splunk HEC or Sentinel batches through the durable outbox. It persists exact record lag, cursor, retries, safe terminal error codes, and collector request IDs; it does not store credential values or remote response bodies. This native feed is separate from the metadata-only OTLP log exporter and from incident-response SIEM dispatch. The Splunk app consumes this native trstctl:audit stream; generic siem notifications and trstctl:response packets do not enter its two shipped searches. Collector HTTP acceptance alone is not indexing or correlation proof: independently read the stored event and search result. When protocols.tsa is enabled the chain head is countersigned with an RFC 3161 timestamp over a domain-separated imprint; the artifact carries the timestamp info, signature, TSA certificate, and CMS DER token. The shipped trstctl-cli audit verify command checks all five saved formats offline. An auditor first downloads the public-only JWK set through authenticated GET /api/v1/audit/verification-keys. The command trstctl-cli audit verification-keys downloads it for separate --audit-jwks pinning. The TSA issuer root is separately pinned and supplied as PEM or DER with --tsa-root; the verifier never promotes the TSA leaf embedded in evidence into a root. It reconstructs the archived-prefix-aware chain, verifies the domain-separated RFC 3161 imprint, and can enforce a maximum anchor delay with --max-anchor-delay. A present CMS artifact must verify its own signature, exact signer and signed fields against the authenticated manifest. The receipt's timestamp_artifact_verified flag distinguishes that proof from historical manifest-only tokens, which remain verifiable but have no portable CMS artifact. The TSA's own certificates are entries of the issuing CA's ledger, so its CRL and OCSP responder answer for them; POST /api/v1/protocols/tsa/certificates/rotate (trstctl-cli protocols tsa certificates rotate, Protocols console) moves the TSA onto a fresh signer-held key, revoking the outgoing certificate when given a reason, and a certificate rotated out earlier can be revoked later. Every export carries the CA's OCSP answer for each TSA certificate its timestamps rely on, captured when the export is made, and the verifier checks it against the pinned root: a certificate revoked for key or CA compromise refuses every timestamp it signed, one revoked for another reason refuses only timestamps at or after its revocation, --tsa-crl brings a fresher CRL, and --require-tsa-revocation refuses a certificate whose status no evidence issued after its newest timestamp establishes. Its limits: the evidence is the status when the export was made, so a revocation recorded later is seen only through a CRL the auditor fetches; the long-term evidence is the timestamps, their TSA certificates and this OCSP evidence, not a renewed archive timestamp (RFC 4998), so verification still depends on the TSA root and SHA-256 remaining trusted; and an export whose evidence could not be captured says so as an unchecked status. This independently pinned trust is what makes a BACK-DATED head detectable: a chain rebuilt to remove a record hashes differently, so no earlier token exists for it, and a freshly-taken token is dated long after the events the bundle describes. Scope, stated exactly: an anchor proves the head is no NEWER than the timestamp. One anchor does NOT prove the chain was complete when anchored — a record withheld before anchoring was never in the chain. Continuous anchoring at a known cadence is served for that: with Enterprise audit compliance and the TSA served, the leader timestamps each tenant's full chain whenever it has grown, on audit.anchor_interval (hourly by default), and appends the anchor to the tenant's own log as an audit.anchored record; GET /api/v1/audit/anchors (trstctl-cli audit anchors, Audit console) shows the cadence, state and recent anchors. A selection by time or sequence alone exports the hashes its records have in the tenant's full chain (prev_hash is the chain head before its first record), so trstctl-cli audit verify --tsa-root ... --anchor-cadence 1h re-checks every anchor inside it, refuses a record that entered the chain after an anchor it predates, and requires every record to be covered within the cadence. Its limits: a record withheld for less than one cadence plus the clock tolerance (five minutes by default) is not caught; a content-filtered selection cannot carry the series; an anchor taken before a subject erasure that changed the records it covers is reported as superseded, not verified; and anchoring stops while the TSA is unreachable (the status reads failing and the next pass catches up). A deployment without a TSA exports successfully and says in the payload that it is unanchored; that is a weaker claim, not an invalid one, and the export states which it is. The CLI verifies a plain signed JWS using separately pinned --audit-jwks, reporting audit_signature_verified: true and anchor_verified: false. It checks the signed slice's tenant, count, archived-prefix chain and head, without claiming an independent time or complete tenant history. --require-anchor, a supplied TSA root, or a positive maximum anchor delay refuses unanchored evidence. Timestamp failures never fall back to plain verification, and unsigned record streams still require TSA trust. Each tenant's continuous anchors are also appended, in order, to an RFC 6962 anchor log (internal/translog): every export carries a head of that log signed with the audit export key (its own artifact domain) and an inclusion proof for each anchor record it holds, and trstctl-cli audit verify --audit-jwks ... recomputes each leaf from the anchor record, refuses an export from which an anchor between two it holds was cut (the log's entries would skip), and, given --previous-anchor-log with the head kept from an earlier receipt and an export taken with anchor_log_from=<that size>, proves the log only grew in between. Its limits: the head is signed by the deployment's own audit key, so a split view is caught only by an auditor who keeps and compares heads — there is no external witness or gossip; and the newest anchors a lagging projection has not logged yet are reported as an unlogged tail, not refused.

  • Signed invoice evidence (L2): GET /api/v1/provider/usage-evidence (trstctl usage evidence, Platform console panel) serves a per-customer, per-period document that states its own completeness and carries THREE attestation layers, each of which can refuse: coverage (the durable metering store must vouch for the whole closed period), reconciliation (the certificates_issued meter is recounted from the identity_transitions projection of the event log, and a divergence names BOTH numbers and blocks signing — a counter that disagrees with the log must not be invoiced from), and the detached RS256 signature over the canonical bytes, made with the SAME audit-export key every other auditor-facing export uses. An unsignable document still returns 200 with its reason — "your usage is incomplete and here is how" is actionable where an error is not — and NEVER carries a signature; the absence is the point. ?format=csv exports the table with the verdict and digest on EVERY ROW, because a spreadsheet slice detaches headers, and the JWS stays in the JSON document where a verifier can use it. Per-customer quotas are durable (provider_tenant_quotas), administered only through the provider plane behind the per-customer delegation gate (PUT/GET /provider/v1/tenants/{id}/quota), counted against the live read model, and ENFORCED at the served issuance transition with a structured 429 before the orchestrator accepts anything. Reconciliation sources, stated exactly: certificates_issued is recounted from certificate issuance receipts; agents from agent lifecycle receipts (first heartbeat or renewal before the period end, not offboarded before it; existing agents were captured once from their projected rows when the receipts arrived); tenants from the core tenant registration projection; secrets_stored from application-secret mutation receipts, vouched for as independent only while the tenant holds no connector credentials, which have no event history and are then named in the line's note. certificates_stored still has no independent record (receipts carry mints but not revocation or expiry) and its line says so. A checked gauge that diverges blocks signing like a diverging issuance count. The Provider console labels unchecked meters separately from a checked meter that diverged; the document's signable verdict determines whether finance may invoice. The certificates_stored cap is enforced at a served create path. The agents cap is enforced when a new agent first heartbeats over the mTLS channel: the server serializes the live quota/count check through event projection across replicas, returns ResourceExhausted before recording an over-cap agent, and lets an existing agent keep heartbeating after a cap is lowered. A one-time enrollment token and signed bootstrap certificate do not reserve a fleet slot; a cap changed after token mint still applies at the first heartbeat. The secrets_stored cap gates first writes through both the native and Vault KV APIs, and connector credential insertion; those objects share the same live stock count. Each tenant's first writes serialize across replicas. A durable secret command already admitted but not yet projected reserves a slot so a process crash cannot oversubscribe the cap. Signed usage gauges count only materialized objects, not reservations. Lowering a cap still permits exact idempotent retries, rotations, and deletion of existing objects. New writes over the cap return 429; concurrent first writes can return a retryable 503. The licensed tenant band gates the managed-offering create path: POST /api/v1/managed-offering/tenants counts the hosted tenants already registered under the projection lock and refuses the next one with a 429 naming the band and the count once it is full (an unreadable count is a retryable 503, never a zero); the Provider plane's own provisioning enforced the same band already, and GET /api/v1/managed-offering/status now reports managed_tenants_registered beside tenant_band. Quota-read/count outages return a retryable 503 for new capped creation rather than silently treating an unknown count as zero. The tenant-self-service route still serves only the caller's own tenancy and refuses a foreign customer_id. Provider staff use authenticated GET /provider/v1/tenants/{id}/usage-evidence instead: the handler requires that exact customer's read delegation before billing opens the customer's forced-RLS transaction, then returns the same signed JSON document or ?format=csv finance CSV. GET /provider/v1/evidence/verification-keys supplies the separately fetched public-only JWK set; the Provider console reconstructs the exact displayed canonical bytes and shows Signature verified only after checking the RS256 signature, protected billing-invoice domain, key id, and digest. An undelegated customer or wrong-operation grant is refused before metering is touched. Reconciliation compares against the transitions projection, so a rebuild-in-progress can transiently refuse to sign — the refusal names the numbers, which is the correct behavior while the projection catches up.

  • notification routing matrix and inbox: expiry, CT, drift, and workflow alerts resolve through the configured severity-to-channel matrix, dedup by per-subject/threshold/channel, and are inspectable through the served notification inbox with owner/approver escalation fields and dead-letter requeue.

  • MCP-vs-REST parity guard: the served MCP automation surface includes broad route-backed REST tools in addition to the named investigation tools, and CI fails when a served REST route is missing both an MCP mapping and an explicit allowlist.

  • Cert-ops console parity: issuer catalog and Test connection, operations queue, Alerts and delivery, richer certificate filters, dashboard charts, CTA empty states, onboarding carousel, and server-side command-palette search are in the served console.

The trstctl-cli drives this same served surface, including OIDC/SAML/LDAP browser login, the React web console, and the AI/RCA/MCP surface covered in their own sections below.

Tenant offboarding boundary

Tenant offboarding erases PostgreSQL read state for the target tenant: it deletes every tenant-scoped table under that tenant's row-level-isolation context, verifies zero residue, and returns a deletion attestation. The tenant.offboarded event is replayed by projections, so a read-model rebuild does not resurrect the tenant's PostgreSQL rows.

That is the limit of what tenant offboarding deletes — it is not a promise that the append-only event log or a signed audit archive disappears when a tenant is offboarded. Those records are governed by audit/privacy retention policy: configure TRSTCTL_AUDIT_RETENTION plus an archive target (TRSTCTL_AUDIT_ARCHIVE_DIR, or TRSTCTL_AUDIT_ARCHIVE_OBJECT_STORE with an object store) for archive-backed retirement from the served audit view, and use Privacy Retention for non-audit personal data pseudonymization. The underlying AN-2 event envelopes stay retained for rebuild and disaster recovery. WORM/object-store archive cleanup and legal hold decisions remain operator privacy/compliance work, but the product gives that work a queryable evidence ledger: record deletion, legal-hold exemption, or cryptographic shredding with POST /api/v1/privacy/archive-erasure-attestations, inspect it with GET /api/v1/privacy/archive-erasure-attestations, or use trstctl privacy archives attest/list. The attestation stores subject_ref and redacted evidence refs rather than the raw subject.

An empty Change history search reports the tenant's sealed archived-prefix count when one exists. The API returns only its count and chain head, never the archive file, host path or object locator; retention does not silently turn an online audit read into an archive retrieval. An archive custodian must supply the signed JWS files (from the directory, or downloaded from the object store with any S3 client) and pinned verification keys for offline checking.

Retention elects one archiver per audit scope across control-plane replicas. It pins the source generation until the signed segment, audit.archived event, and PostgreSQL checkpoint are durable. Ordinary Provider quota and metering writes can continue during that work; a history rewrite or full backup waits for the active segment to finish. A large or slow archive can therefore delay a privacy-erasure cutover or full backup. The retention worker reports its failure for retry without claiming the unfinished segment is archived.

Built and tested, but not yet served by the binary

No current feature-map row is in this bucket. The empty section is deliberate: future library code that is built and tested but not yet wired into the served API must be listed here as library-only until its required wiring-census proof passes — Phase 2 residual breadth stays under Partial or Roadmap instead of being quietly promoted.

Conditional, partial, and residual boundaries

These notes explain the matrix's conditional and partial rows: some are served only when an operator enables or configures a backend, and some have a served spine with explicit residual work. The matrix above is the authority for whether the running binary serves a capability; this section records the operator-facing edges and follow-up integration work.

PCAS KEM data re-wrap

POST /api/v1/pcas/kem/rewraps creates a tenant-bound request, mints the signed paired successor signing/KEM key record (key_ready), and then moves the epoch's registered data keys through the isolated signer. The ciphertext target is a KEM-wrapped data-key envelope: an operator fetches the epoch's public encapsulation key (GET /api/v1/pcas/kem/public-key, CLI pcas kem public-key), builds the envelope offline (pcas kem envelope encapsulates to that key and seals the data key under the derived key; the data key never reaches the control plane) and registers it (POST /api/v1/pcas/kem/data-keys, pcas kem data-keys register). A re-wrap request snapshots the predecessor epoch's inventory at acceptance and moves exactly those envelopes; an envelope registered later is accepted, waits for the next request, and keeps the retirement gate closed until it is moved. A stage with no envelopes completes only when the request carries empty_inventory_attested; otherwise it stays key_ready instead of reading as migrated data. For each envelope the signer's RewrapKEMDataKey opens it under the predecessor KEM key and seals the same data key for the successor; the control plane then asks the signer to read the new envelope back under the successor key alone (VerifyKEMDataKey) and records an immutable per-target receipt bound to the request id and digest, both epochs, the predecessor ciphertext digest and the data-key digest. Each stage closes with a completion event over its target count; the job row's data receipt turns complete, rewrap_complete becomes true and the retirement gate opens. GET /api/v1/pcas/kem/data-keys serves the successor envelopes for stock readback. Boundaries: the operator owns the inventory (the status shows data_keys against data_keys_rewrapped per stage); the PCAS console shows completion but has no registration workflow yet; the first KEM epoch of an identity comes from an attested-empty request against its predecessor epoch, since no earlier KEM key exists.

Workload-held co-sign mode (off by default)

trstctl-agent --workload-cosign-listen serves the PCAS workload-held predecessor co-sign RPC with the workload's predecessor key, and the control plane now calls it. Register the identity's genesis with predecessor_custody: "workload" and the agent key's public half (public_key_pem); the anchor records that key instead of minting one in the signer. Configure the agent's endpoint under pcas.workload_cosigners (unix:/path for a co-located agent, or host:port with the control plane's client certificate, the agent's CA, its SPKI pin and server name). The identity's first succession request then runs the FIG. 5 flow in the succession worker: the control plane forms the commitment over the agent-held predecessor and a successor the isolated signer generated, obtains the predecessor co-signature over the co-sign RPC, verifies it against the commitment it supplied, has the successor sign inside the signer, binds the record to a tenant-root transparency-log inclusion proof (mandatory for this mode) and records it with the same transaction every succession uses. The agent refuses commitments for any other identity, tenant or deployment. After that first succession the successor lives in the signer and later successions are signer-held. Boundaries: one endpoint per identity, no console flow yet, and the operator supplies the agent's public key at genesis. The listener authenticates callers:

  • host:port requires pinned mutual TLS (--workload-cosign-tls-cert, --workload-cosign-tls-key, --workload-cosign-peer-ca, --workload-cosign-peer-pin). Without them the agent refuses to start and never reads the key.
  • unix:/path creates the socket 0600 in a 0700 directory and admits only --workload-cosign-peer-uid (default: the agent's own uid). Platforms that cannot report a peer's uid, such as Windows, refuse a unix socket; use a pinned-mTLS listener there.
  • Requests are limited to 64 KiB, 16 concurrent streams and 8 in-flight calls, with a 10-second deadline.

Agent certificate renewal keeps revocation and identity

Both renewal paths, POST /enroll/renewal on the agent mTLS listener and the agent channel's Renew, refuse a revoked agent certificate, an offboarded agent, and a CSR that names a different agent than the presenting certificate. Renewal rotates a key; it never restores a revoked identity or changes which agent a certificate names. Such agents get 403 with the reason and must be enrolled again with a new bootstrap token. Revocation and offboarding first commit a tenant-scoped removal command. That row blocks every new agent RPC on an already-open connection before the event projects; startup and the bounded recovery worker publish and project its exact retained event after an interrupted attempt. Agent channel refusal is a direct admission check, independent of CRL or OCSP distribution to other relying parties.

The CA calendar: year-scale hierarchy expiry

Leaf expiry alerting runs on 7/30/90-day windows. That is the right clock for a leaf and useless for a certificate authority: replacing a trust anchor means getting the new one into every relying party first, which is a quarters-long program, so a 90-day warning arrives long after it could have helped. Nothing evaluated ca_authorities.not_after at all before this, and the CA console printed the raw date and called anything past 90 days healthy.

CA authorities now run on their own clock. The leader lifecycle sweep walks every active authority against year-scale bands — 36, 24, 12, 6 and 3 months — and alerts once per band an authority crosses into, with severity scaled to runway: a planning signal beyond a year, a warning inside one, critical inside three months or already expired. The band already notified is recorded on the authority and only ever tightens, so a repeated sweep is silent, each tightening re-fires, and a clock skew cannot replay an alert the operator already saw.

The second check is the one that hides. A CA cannot issue a leaf that outlives it, so once an authority has less life left than the validity its leaves are issued with, every new leaf is silently truncated to the parent's expiry. Issuance keeps succeeding; certificate lifetimes become shorter until something downstream rejects one. That case raises ca.validity_compression rather than a plain horizon alert, because the fix is different: renew or re-key the authority.

Exact contract. Alerts land on the notification.ca_horizon outbox destination as ca.horizon or ca.validity_compression, carrying the authority, its band, how many active certificates chain to it, and the renew-by date; they fan out through the same operator-configured channels as expiry alerts. The v2 immutable decision is appended first; its alert intent and band stamp then commit in one tenant transaction. Startup and each CA-calendar sweep reconcile a retained decision whose database transaction was interrupted, including when a tail projector stamped the band before the intent was queued. A broker append failure leaves neither a stamp nor an intent. The reference leaf validity is lifecycle.leaf_validity (TRSTCTL_LIFECYCLE_LEAF_VALIDITY, default 2160h / 90 days) — a yardstick for horizon reporting that caps nothing. GET /api/v1/ca/authorities carries a horizon object per authority, and the CA Hierarchy console shows the band, the renew/re-key-by date, and the truncation warning. GET /api/v1/certificates/health resolves beyond 90 days into 180-day, 1-year, 2-year and 3-year bands; later now means beyond three years.

The scheduler appends ca.authority.horizon_alerted v2 events for new horizon decisions. Builds before the closed v1 privacy schema was registered could enqueue an alert and stamp its band, then fail the audit append; a broker outage in that older SQL-first flow could do the same. Those historical missing events cannot be treated as present: inspect the retained notification outbox and delivery evidence for those bands. V1 events are audit-only history; v2 events carry the exact alert facts needed to recover an interrupted outbox command without recomputing it. A subject erasure rewrites any still-retained v2 notification outbox command from the rewritten event before reconciliation may execute it again.

Two edges that used to be silent now speak. An authority with no recorded not_after is evaluated from its own stored certificate: when that parses, the sweep's decision records the derived expiry on the row and alerts in whatever band it lands in; when it does not, the sweep raises one ca.horizon alert at warning severity, stamps the reserved unknown-expiry band, and the served authority carries horizon.expiry_unknown so the console shows a warning rather than a blank. An unknown expiry is a real state, not a passing one. An authority beyond the widest 36-month band announces itself once as a planning entry (severity low, band 37, "on the expiry calendar") and is then quiet until it crosses into a band; the stamp orders before every real band, so each later tightening still re-fires. The compression check also looks back: every active leaf issued under the authority whose expiry equals the authority's (the clamp's signature) and whose validity is shorter than the reference is counted, the compression alert says how many already carry the parent's expiry, and the served authority lists horizon.truncated_leaves with the first fifty truncated_leaf_ids by validity start, so the shortened certificates can be walked to rather than inferred.

Scheduled rollover drafts are opt-in: with ca.rollover_draft_band_months set to a band, an active signer-backed authority entering it gets a rekey_ca ceremony opened for it on the same sweep — the reviewed plan (its profile and lifetime, a "Scheduled rollover" reason) at the approval floor, opened by trstctl:ca-calendar so every custodian may approve it (ca.rollover.drafted). GET /api/v1/ca/authorities and the CA console show the draft with the ceremony's approvals. A repeated sweep drafts nothing new; a cancelled ceremony drafts again only at a tighter band; an authority that cannot be re-keyed online (an offline root needs its offline package) records why instead. Nothing rotates on its own: custodians approve the ceremony and a person executes the re-key with the drafted plan, as the re-key and rotation routes always required.

Agent roles: a vantage in the certificate

An agent runs in one of two places, and where it runs decides what it can be asked to do. On a host, it acts on the machine it lives on. In a network segment, it acts for things that cannot run an agent at all — load balancers, appliances, cloud certificate stores — and probes endpoints the way a client would. The second kind holds the credentials that drive those devices, which is exactly why it is a grant and not a startup flag.

An operator picks the roles when they mint the enrollment token. The grant is recorded with the token, and at redemption the CA stamps it into the issued certificate as an additional SPIFFE URI SAN (spiffe://trstctl.example/tenant/<id>/agent/<cn>/role/<role>). It is never read from the CSR: an agent that could put a role in its own certificate request would be choosing its own capability, and the whole control is that it cannot. A CSR that tries gets a certificate carrying only what the operator granted.

At claim time the served channel reads the roles off the certificate the agent authenticated with — the same certificate the tenant is derived from — and intersects them with the vantage each job kind needs. trust.distribute acts on the agent's own machine and is host work. discovery.run, endpoint.verify, and revocation.probe are observations from a segment vantage and are relay work. A host agent reaching for relay work is handed nothing and the reach is recorded as agent.jobs.role_refused, because it is either a misconfiguration or the thing the gate exists to catch.

Granting the network role additionally requires the agents:relay.grant permission, separately from agents:write. Enrolling a host agent is routine fleet work; placing a relay puts appliance credentials on a machine of the operator's choosing, and those are different decisions.

The agent binary itself is now held to the boundary the roles imply: a CI guard (docs/agent_binary_import_boundary_test.go) fails the build if trstctl-agent links the database driver, the store, the event spine, or any served control-plane package — before the guard existed, two dead store-backed sinks had already pulled all of those into the binary. A companion guard pins the connector core (internal/connector/...) to a host-neutral dependency set, which is what will let the same connector implementations execute on a relay.

An agent enrolled before roles existed carries no role SAN, and is read as host-only rather than as capability-less. That is what those agents already were, and reading them any other way would strand a live fleet mid-upgrade. A renewal carries the roles across unchanged — it cannot gain a capability, and it cannot silently lose one either — unless an operator recorded a role change. An operator holding agents:relay.grant records the complete role set (POST /api/v1/agents/{id}/roles, Change roles on the Agents page); the agent's heartbeat asks it to renew, and the renewal over the agent channel signs exactly those roles, so a host agent becomes a relay without re-enrolling. Roles never come from the CSR: one asking for a role the renewal is not signing is refused by name. An agent built before this release ignores the renewal request and takes the roles at its scheduled renewal; the HTTP renewal endpoint carries roles over and does not apply a grant; and removing a role leaves the old certificate valid until it expires unless it is revoked.

Per-row locality is now gated at the kind the roles share. connector.deploy and connector.rollback remain both roles' work at the kind level, and the row decides: at enqueue, the control plane reads the connector name out of the raw payload — the one moment it is not yet sealed — consults the shipped vantage census (nativeConnectorVantage), and stamps required_agent_role onto the outbox row. The claim SQL filters on that plain column, so a host agent asking for connector.deploy receives the nginx deploy and never the F5 deploy. A cloud-store deploy (ACM, Azure Key Vault, GCP Certificate Manager) is network-relay work since A59, reached over the store's management API from the segment a relay sits in; where a tenant runs no relay, the control plane's egress-guarded client carries it. The stamp is durable in the lifecycle event's side-effect record, so reconcile-replay reproduces it rather than re-deriving it against a possibly-changed census; rows enqueued before the census existed carry the empty demand. The dispatcher reclassifies those legacy rows by connector family before native lookup, so an old nginx row cannot use an upgrade as a path back into control-plane execution.

The census is per connector KIND, written by hand in the composition root, and deliberately not derived from transport: Envoy is HTTP-driven yet classified host-agent, because its admin surface binds loopback in the deployments we ship for. The agent has constructors for the same 14 host families, including Envoy, and a generated parity test compares the two closed sets. A host stamp is carried through the control-plane outbox claim rather than discarded; the dispatcher defers it before unsealing, and its direct handler repeats the family check before registry lookup. Classification failure is therefore a refusal, never permission to touch the control-plane host.

Per-target vantage overrides are served for Envoy only (A61). An Envoy target may declare vantage: network in its configuration: its deploys and previews are then stamped for the network role, need no host assignment, are refused by the control plane like any host-family row, and are executed by an enrolled network relay against the remote SDS endpoint with the same connector the host agent uses. The declaration is validated when the target is saved — any other family, or any value but network, is refused — and the row's role is derived from it at enqueue, so neither an agent nor the console can relocate work by editing anything else. The roles column on the agents read model is still a projection for the console only: writing network into it grants nothing, because the certificate still says host and the claim path still refuses. Host availability also never re-enables the old path: with no eligible host agent the deploy stays pending for Operations to surface.

AD CS template posture, read from inside the domain

A Windows PKI's real attack surface is not its CA — it is the template list. A template that lets the enrollee supply their own subject, grants enrollment to a broad group, and carries a client-authentication EKU is a domain escalation path that looks, in every console the organization owns, like an ordinary certificate template. Nobody has an inventory of these, because the information lives in the directory rather than anywhere a PKI product looked.

An operator creates an adcs discovery source and schedule with an LDAP/LDAPS authority, configuration naming context, read-only bind DN, secret:// credential reference, and optional exact relay UUID. The mutation emits the immutable discovery source/run events and writes one adcs.inventory outbox intent in the same transaction. Only a network-role relay can claim it; an exact UUID pins it further. The control-plane worker recognizes the job as estate-owned and performs no directory I/O. Concurrent schedule leaders use a database transaction lock and database-clock due check, so one due interval produces one run rather than two.

adcs.inventory reads pKICertificateTemplate and pKIEnrollmentService objects under CN=Public Key Services, capturing schema version, msPKI-Certificate-Name-Flag, the enrollment and private-key flags, EKUs, which CAs publish each template, and the template DACL. It runs in-domain because a domain controller's LDAP is not reachable from a hosted control plane and should not be — an in-domain relay is the only vantage from which this inventory exists.

The source may also name at most 24 enrollment endpoints, each bound to the exact LDAP enrollment-service name and one closed kind: web_enrollment, ndes, or ndes_admin. The relay performs one bounded GET from inside the domain, requests at most one response byte from cooperative servers, never follows a redirect, and closes the body without reading or retaining it. Only the configured URL, closed reachability/authentication state, HTTP status, authentication scheme names, verified-HTTPS fact, and Extended Protection state cross the relay boundary. Cookies, redirect targets, authentication challenge bytes, and response bodies do not. This configured-only behavior remains the default. With endpoint_probe.enabled: true and 1..32 explicit endpoint_probe.dns_suffixes, a selected upgraded relay also derives fixed HTTP and HTTPS /certsrv/ and /certsrv/mscep/mscep.dll candidates from each LDAP dNSHostName. It never generates mscep_admin, follows redirects, or treats a response as successful SCEP enrollment. CES metadata is a separate surface.

The adcs.inventory.endpoint_probe capability requires the server allowlist and the exact relay's fresh enabled census; it also supports nested-group reads when both options are selected. The planner rejects address literals, URL/port syntax, wildcards and hosts outside the configured DNS suffixes. Explicit/generated duplicates are probed once, with both origins retained. Public addresses are accepted through the shared resolved-address SSRF guard. Private endpoints require the source's explicit allow_private_endpoint opt-in and a bounded private_egress_cidrs list limited to RFC1918 or IPv6 ULA sub-prefixes. The resolved address must fall inside that list; link-local/cloud metadata, loopback, multicast, unspecified, and CGNAT addresses remain blocked. The relay's own destination grant is an additional check; LDAP hostname data never grants egress or disables HTTPS verification.

Each opted-in run represents at most 24 targets with four HTTP workers, five seconds per attempt and a 90-second overall budget; the legacy job keeps its 60-second budget. Larger plans carry a source/run-bound digest and next offset recovered from retained immutable evidence. The next scheduled or manual run advances that cursor; a failed read retries its page and a changed plan starts at zero. Every service identifies its place in the latest page, including invalid/out-of-scope hostnames and services outside that page. Cancellation before an attempt yields not_checked, with no fabricated status or TLS fact. Partial or unverified coverage produces ADCS-ENDPOINT-PROBE-INCOMPLETE. The API, CLI, console and compliance evidence preserve provenance, page coverage and timestamps. The latest page never implies simultaneous full-estate evidence. The bounded posture service view flags its 500-row ceiling; retained observations carry the full directory plan. See the endpoint probe runbook. Windows IIS authentication/EPA and live NDES enrollment still need a Windows lab; Samba and private HTTP/TLS stand-ins establish only the implementation boundary.

The combinations are named, not the flags. Nobody spots an escalation path by scanning four boolean columns across ninety templates, so the analysis reports consequences: ADCS-ESC1 fires only when supplies-subject, an authenticating EKU, the absence of manager approval, and an observed broad/low-privilege enrollment trustee are all present, because removing any one of them changes the answer. Everyone, Authenticated Users, Builtin Users, Domain Users, and Domain Computers are recognized by canonical SID; an unknown custom SID is never guessed broad. Enrollment-agent templates get their own rule (ADCS-ESC3-AGENT) rather than being folded into the client-auth checks, because the primitive is different — an agent certificate requests on behalf of any principal, so one of them is a master key rather than an impersonation of one account — and so is the remediation: restricting who may enroll is not enough, the CA must also bound which templates accept agent-signed requests.

The same sweep checks the surfaces outside template LDAP. Plaintext published CES URIs, reachable plaintext Web Enrollment/NDES, anonymous mscep_admin, and Windows-authenticated Web Enrollment whose Extended Protection is disabled or unobserved produce separate findings with the exact endpoint/status/header facts. msPKI-Enrollment-Servers is treated only as the CES metadata it is; it is never mislabeled as proof that legacy /certsrv or NDES exists or is absent. On Windows relays, certutil.exe queries each publishing CA's CA\\EnrollmentAgentRights: a present policy is enabled, Microsoft's stable missing-value result is disabled, and access/transport/tool failures are unobserved. A non-Windows relay reports requires_windows_relay. Missing evidence therefore produces an explicit visibility finding instead of becoming either a vulnerable or hardened guess.

Every finding carries the attributes and values it was derived from. A posture finding an operator cannot check against the template's own property page is one they have to take on faith, and the first false positive they cannot check costs the credibility of every true finding after it. A test asserts a deliberately hardened template set produces no findings at all, which is the harder half of getting this right: firing a rule is only half the problem; keeping it quiet is what makes the output worth reading. The SAN variant is reported separately and is the more urgent of the two, since SAN-based mapping is what Windows authentication actually reads. An empty EKU list counts as authenticating, because unrestricted is not harmless. Every finding names the specific change that removes it, and reports whether a CA actually publishes the template — a dangerous template nobody publishes is a latent risk an operator can fix calmly.

Read-only, structurally. The directory interface has exactly one method and it is Search; a test asserts that. An inventory tool pointed at a domain controller must be incapable of modifying one, not merely careful. The queries name the attributes they use rather than requesting a wildcard, are bounded to 2,048 templates and 512 enrollment services, and are scoped to the Public Key Services container. The template query requests nTSecurityDescriptor with Microsoft's critical LDAP_SERVER_SD_FLAGS_OID (1.2.840.113556.1.4.801) and the DACL-only flag; owner, group, and SACL data are not requested. A plain ldap:// connection is upgraded with StartTLS or the read does not happen: a template inventory is the map of a domain's escalation paths, and reading it in the clear publishes that map to anyone on the segment. Anonymous binds are refused rather than attempted — where they would succeed, the directory is misconfigured in a way worth reporting rather than quietly relying on. The redeemed bind credential is attempt-scoped and never enters the intent, event, report, or posture row.

The Posture console shows it: configured directory readers with pending/running/succeeded/failed lifecycle, templates worst-first, what each one permits and the specific fix, whether a CA publishes it, canonical Windows SIDs granted the enrollment extended right, and which relay observed it when. It also shows each enrollment service, its published CES URIs, configured live endpoint probes, CA restriction state/source, exact finding evidence, and the same observation time/relay. The Discovery form exposes the bounded endpoint targets; the control plane never probes them. The empty state distinguishes "no relay has read a directory yet" from "no AD CS estate", because those are opposite facts an empty table cannot tell apart. The posture table is only an event projection. The immutable adcs.template.inventory.observed v2 event is the authority, snapshot/restore includes both template and enrollment-service projections, and a cold replay rebuilds the same rows atomically. Historical v1 template-only events remain replayable but cannot masquerade as complete service evidence. Observation time and relay identity stay visible so restored evidence cannot masquerade as a fresh directory read.

Changes between sweeps are reported semantically. A textual diff of two directory dumps is useless — attribute values are bit fields, and "msPKI-Certificate-Name-Flag changed from 0 to 1" tells nobody anything. Each change says what it means and which way it moved: somebody turning on enrollee-supplies-subject, or removing manager approval, or publishing a template that carries findings so a latent risk became an offered one. Only a change for the WORSE emits an alerting event. An operator who has hardened a template does not need waking, and a tool that alerts on improvement teaches people to mute it — after which it will not reach them on the day it matters. Better and neutral changes are still recorded, because an incident timeline needs them.

Those records are now first-class, tenant-scoped adcs_template_drift discovery findings rather than event-log-only notes. Each one is bound to the real source, run, relay UUID/name, domain, and observation time, and preserves the semantic change plus exact normalized before/after facts. Trustee changes carry canonical SIDs (who gained or lost enrollment access), never raw security descriptor bytes. GET /api/v1/posture/adcs/drift, the generated SDKs, and the Posture console expose the same bounded history. A worsening v2 drift event projects the immutable finding and one notification.drift outbox intent in the same PostgreSQL transaction, so it appears in the notification inbox after one sweep; first, unchanged, neutral, and improved sweeps do not page. Event ID and outbox-key replay guards keep a retried signed receipt from duplicating either. Pre-v2 drift events lacked run/source authority and therefore replay as audit history only rather than being attached to a guessed source.

Licensed compliance evidence packs copy the latest internally consistent service observation for each domain in the bounded window plus every authoritative v2 drift record into the signed manifest. Each carries tenant-local event ID, type, sequence, chain digest, source, run, relay, and observation time. The outer API copy is the same semantic object for the console; offline verification still treats the signed manifest as authority. Template/service findings are recomputed from normalized event facts before export, so an agent cannot omit an uncomfortable verdict while keeping the facts that require it.

A first sweep is deliberately not drift. Reporting an entire estate as "added" the first time anyone looks would bury the real change that comes next under ninety notifications. The template is stored exactly as the directory reported it so the next sweep diffs against what was really there; a row written before that column existed is skipped and re-baselines on one quiet sweep, rather than being reconstructed into a template whose flags all read false and reported as becoming dangerous.

Nested enrollment links are served with an explicit opt-in. Set nested_groups_enabled: true, membership_domain_dn to the domain naming context, and relay_agent_id to the exact upgraded network relay. Enable adcs.inventory.enrollment_groups in agent_channel.claimable_job_kinds and start that relay with --relay-claim. Preflight and run admission require its fresh authenticated census to advertise the enabled reader. An older relay cannot silently ignore the nested settings. Default adcs.inventory retains direct-trustee collection.

The reader resolves canonical grant/deny SIDs inside that domain and uses the LDAP in-chain rule 1.2.840.113556.1.4.1941, with a critical single-naming-context control and no referral following. Known direct member links positively check the matching rule. Users/computers appear only with a reconstructed path through observed security groups. Direct and expanded denies win conservatively; conditional/compound permission ACEs keep the result incomplete. Unreadable, unresolved, range-limited, unsupported or budget-limited evidence carries no verified member list. An empty LDAP result alone never proves an empty group. The console and trstctl-cli posture adcs show completeness, observation time, domain, path and trustee. A failed later read is a coverage change, never proof that rights were removed. Signed receipt retry, source-log replay and tenant isolation preserve these facts. The served member list is checked against its retained immutable observation; a changed SQL path is refused until replay.

Bounds are 256 membership queries, 4,096 returned nodes, 1,024 descendants per trustee, 16 path nodes, 256 KiB of membership attributes and 384 KiB of serialized membership evidence across templates. The nested collector has 45 seconds inside a 60-second relay attempt. LDAP authentication has a 20-second deadline, and searches have 20-second request timeouts. The TLS-only byte-backed bind wipes the borrowed password immediately after authentication; directory frames are capped at 2 MiB and the connection at 16 MiB before library allocation. Exceeding a bound is an explicit incomplete/failed read, not successful absence.

What is not inferred: a complete effective Windows token. The parser handles self-relative DACLs, ordinary/object allow and deny ACEs, Generic All, the certificate-enrollment extended-right GUID, SID byte order, exact-trustee deny, null DACLs and malformed-value refusal. Link evidence does not evaluate conditional ACEs, primary groups, SID history, cross-forest tokens or full Windows AccessCheck; it is never permission to issue. Focused served tests use real Samba AD 4.15.13 and a user → inner → outer enrollment grant. Samba's lab TLS identity override is explicitly unverified. Live Windows forest and trusted production directory identity qualification remain release-lab work. See the nested enrollment runbook.

External Windows-lab boundary. The repository fixture runs the real LDAP client over a BER wire, accepts exactly the two bounded SearchRequest operations, checks the critical DACL-only control and named attributes, returns a binary self-relative descriptor, and proves the normalized enrollment SID reaches the inventory. The served journey separately proves real PostgreSQL, JetStream, out-of-process signer, mTLS relay claim, single-use credential redemption, signed result, event projection, cold replay, and console/API readback. This development environment does not contain a licensed Windows Server forest with AD DS and AD CS, so forest policy, domain-controller authorization, and Microsoft implementation interoperability still require the release lab. A release must run the same read-only account against that lab and retain the domain-controller audit showing Bind/Search only, certutil read evidence, and IIS probe outcomes. Repository tests use live-shaped HTTP/certutil outcomes and cross-compile the Windows collector; they do not claim a real CA's registry encoding or IIS Extended Protection configuration was observed here.

Segment sweeps run from inside the segment

Network scanning ran from the control plane's worker, which meant it could only ever see what the control plane could route to. For a hosted deployment that is the public internet — so the scan inventoried the estate's least interesting surface and reported it as the estate. The segments that actually hold unmanaged certificates are the ones behind a firewall: a management VLAN, a DMZ, a lab nobody admits owning.

discovery.run is now a relay job. A network-role agent already sits in those segments, and the scanners needed no changes to allow it — they were already dependency-light, and the one thing keeping them out of the agent binary was a store-backed sink whose only caller was a control-plane test. Both modes travel: TLS sweeps for served certificates, SSH sweeps for host keys.

The source must bind a declared segment; it may also bind one relay_agent_id. Declare the denominator first with idempotent POST /api/v1/discovery/segments (name, ranges, and the segment's staleness_hours SLO). That mutation emits discovery.segment.upserted, so a cold event replay reconstructs the declaration instead of relying on a seed-only database row. Queueing then resolves CIDRs into at most 10,000 stable host:port targets and persists that exact command with required_agent_role=network in the same event/outbox transaction. The generic control-plane worker recognizes both new commands and legacy network/SSH rows and defers them before receiver I/O, so an upgrade cannot make an old pending scan dial from the control plane. A wrong role or wrong exact agent cannot claim the row.

The reserved-range guard travels with the scanner, not with the control plane. A relay does not escape it by being somewhere else: loopback, link-local and multicast targets are refused inside the scanner (a network source's allow_loopback admits loopback for an authorized local diagnostic; link-local, which covers cloud metadata, and multicast cannot be enabled), and the sweep reports how many were blocked rather than presenting a refused range as an empty segment. Attempted, discovered, failed, rejected and blocked are all reported, because a sweep that reached nothing and a segment with nothing in it must not read the same.

Findings come back over the channel the agent opened and the control plane writes them, as with every other agent finding — a relay holds no database. The result is accepted only while that exact lease generation is held and only after its receipt signature, mode, assigned targets, counts, metadata bounds, and optional exact agent selector validate. A projection error leaves the claim retryable; a replay after completion is refused. The run API and Discovery console show the segment, role/agent binding, blocked count, and the relay ID from the verified terminal receipt.

What changed in the vantage table: discovery.run was host work when it meant "enumerate this machine's filesystem". It is now a segment sweep, which is a vantage question, so it demands the network role. A host agent's own filesystem inventory still travels on the inventory path rather than as a claimed job, so nothing was taken away from it.

Revocation distribution points are monitored and recorded

Every inventoried certificate has carried its CDP and OCSP URLs since discovery shipped — internal/crypto/certinfo parses them — and until now nothing fetched one. That gap is quietly serious. A CRL whose nextUpdate has passed does not announce itself: relying parties either fail closed and break the service, or soft-fail and stop checking revocation at all. Neither appears on any dashboard until an incident, while the CA's operator believes revocation works because publishing succeeded once.

revocation.probe is now an hourly, database-clocked relay job, and an operator can ask for one now: POST /api/v1/revocation/health/probe (certs:write), trstctl-cli revocation health probe, or Probe endpoints now under Certificates → Revocation & CT records who asked and queues the same targets under the request's own bucket, so an operator who fixed a relay's trust bundle or a distribution point need not wait for the next hour; requests inside one minute share one probe. The leader scheduler derives distinct CRL distribution points and OCSP responders from inventory, matches every leaf to its real issuer key, and carries the exact public leaf/issuer certificate context needed to verify the answer. It reads them from the certificates whose signed bytes trstctl holds: those issued through trstctl or its external CAs, adopted from a host, or imported with trstctl-cli certificates ingest, which keeps the certificate it is given (an import never replaces the public chain of a certificate trstctl issued). A certificate known only from a network scan carries its metadata, not its bytes, unless agent_channel.probe_scanned_certificates is on: then a relay's TLS sweep keeps the leaf and the first issuers served after it (each checked against the hash the relay reported), and the scheduler also checks that certificate's own serial at its CRL distribution points and OCSP responders, verified against the served issuer that signed it; its certificate detail shows those signed answers, including whether the CRL lists its serial. A leaf no served or known issuer verifies is not probed, scans by relays older than this setting keep no bytes, and a certificate known only from a cloud scan is not probed; import it to have it watched. It queues at most 32 endpoints per bounded network-role command through the event/outbox transaction. Concurrent leaders converge on the same command ID; if the representative leaf changes, its OCSP serial changes the ID and creates new work instead of colliding with an old command.

The relay performs a real fetch. For CRLs it parses the signed list, verifies the issuer signature, and records thisUpdate/nextUpdate, latency, and revoked count. For OCSP it builds a request for the exact leaf, POSTs the RFC 6960 media type, verifies the signed response, checks the serial and response status, and applies the same freshness window. Endpoints are deduplicated before probing: one CA's endpoint is named by every certificate it issued, and walking the raw list would be monitoring that causes the outage it watches for.

It is a RELAY job deliberately. The distribution points that matter most are internal — an AD CS CRL on http://pki.corp.internal/certenroll/ is unreachable from a SaaS control plane by design — so monitoring only what is reachable from outside would inventory exactly the endpoints least likely to break.

Five outcomes, kept distinct because they need different people. fresh and expiring differ by a warning window (the last quarter of the answer's own signed lifetime, at most 24 hours, so a responder that keeps signing short-lived answers reads fresh), and expiring is the one worth alerting on: after nextUpdate passes, relying parties are already failing. stale means that has happened. unreachable is deliberately not the same as stale. And unparseable catches the case that fools status-code checks: a proxy or captive portal answering 200 with HTML. An AD CS LDAP CDP (ldap:///CN=<CA>,...,CN=Configuration,DC=...) is read through the tenant's AD CS discovery binding whose configuration DN covers it (A107): the batch is pinned to that source's relay, which redeems the bind credential for one attempt, connects only to the binding's directory under TLS, reads the entry's certificateRevocationList, and judges it like an HTTP CRL. When it cannot read the directory the code says why: ldap_connect_failed, ldap_tls_failed (with the TLS stack's own reason), ldap_bind_failed (the directory refused the credential) or ldap_credential_unavailable (no credential was redeemed for the attempt). A directory certificate with a negative serial number is refused by the relay's TLS stack; Samba's autogenerated certificate draws one about half the time, so give such a DC a certificate with a positive serial, as an AD CS-issued DC certificate has. An LDAP CDP no AD CS source covers reports unparseable as needing a binding, rather than unreachable, because sending someone to check a network path that was never the problem wastes the hour that mattered. Whether the signature was checked is reported explicitly — "we did not check" and "it verified" must never read the same. The signed terminal receipt is accepted only for the live lease and exact command, and its evidence digest must be the SHA-256 of the findings document it reports. A report that can never validate (an older relay's report carried no digest, so before B13-7 no real relay's observation was ever recorded) is refused once: the receipt is kept, the job fails, and the next hourly sweep asks again, so one bad report cannot hold a relay's other work behind it. It emits immutable queue and observation events, projects tenant-scoped endpoint health, and queues one warning/critical notification per unhealthy endpoint. GET /api/v1/revocation/health and Certificates → Revocation & CT show the endpoint, issuer/certificate context, relay, evidence digest, status, latency and freshness window. Before a signed observation exists, the API says observed:false and the console says unknown; an empty projection never reads as healthy.

The view gives client-context guidance, not a universal recommendation. CRLs and OCSP are different relying-party mechanisms. A fresh responder proves the relay could validate that answer at that time; it does not prove every client is configured to check it, nor whether a client fails closed or soft-fails when the endpoint disappears. Operators must test those client policies separately.

External lab boundary: the repository acceptance test uses a real embedded PostgreSQL/JetStream/signer stack, an enrolled network relay, and controlled cryptographically valid CRL and OCSP responders. It does not claim a run against a domain-joined Windows AD CS CDP/OCSP estate. That externally served AD CS lab remains infrastructure evidence to collect when the Windows lab described under F4 is available. LDAP CDP reading (A107) is exercised against a stand-in directory in the repository tests and against a lab Samba AD DC on the running product: through the AD CS binding (StartTLS, simple bind) the relay read the CRL entry under CN=CDP fresh and signature-verified with its one revocation, and a CDP naming no entry read ldap_entry_missing. A run against a production AD CS forest is not claimed.

Just-in-time credential leases: the brain ships references, not secrets

A relay executes against things that cannot run an agent — an F5, a NetScaler — which means it needs the credentials that drive them. Handing a relay standing credentials would put a copy of the estate's admin passwords on a machine in the estate, permanently, whether or not any work was pending. So nothing standing is shipped at all.

A claimed connector job carries a reference-only intent: what to deploy, where, and the NAMES of the credentials it may redeem. Not the credential, and not the sealed container holding it — the agent has no key for that seal, so shipping it would be pointless, and would leave the tenant's credential ciphertext sitting on a host waiting for a future key compromise. At execution time the relay calls RedeemJobCredential over the same mTLS channel it claimed on, and the control plane resolves the references — opening the sealed payload and reading each secret:// name out of the tenant secret store — into locked buffers that are wiped as soon as the response is encoded.

Redemption is once per attempt, ever. It is a single statement: an INSERT ... ON CONFLICT DO NOTHING keyed (tenant, job, attempt), performed only if the caller currently holds the job's claim lease. A replay inserts nothing. So does a second agent that stole a lapsed lease, and so does a stale attempt number. All three get the same coarse PermissionDenied with nothing to distinguish them; the reason is classified afterward for the audit event only, so probing the endpoint cannot map the claim table. Material is resolved BEFORE the gate is taken, so a custody outage refuses the call without burning the attempt's one redemption — the relay retries rather than failing the job. The redemption's expiry is bound to the claim lease and never chosen independently: a credential must not outlive the claim, or a second agent could take the job while the first still holds live material.

The agent's own words no longer become durable history when it held a credential. A1 already kept agent free-text out of outbox.last_error behind a closed set; the event log took it raw, which was safe only while agents held no secrets. An appliance password is short and word-shaped — no redactor recognizes hunter2-lab, and no entropy floor fires on it — so any attempt that redeemed material records a closed-set marker instead of the agent's text. The operator still gets the failure reason, the redemption's audit reference, and the evidence digest. The relay keeps its own record on its host: a bounded failure log (--deploy-failure-log, beside --key as deploy-failures.jsonl by default, mode 0600, the newest 200 entries) that names the step that failed (intent, executor, redeem, material, connector, sandbox, predecessor or verify) and why, with every redeemed value (and every line of a multi-line one), every PEM block, and the value of an authorization header or credential-shaped field removed. An operator with access to that host reads it with trstctl-agent --show-deploy-failures. (Earlier builds discarded the error, despite this promise, then kept it only in the process log.) An attempt that redeemed nothing never held a secret to echo, so its detail flows through redaction as before.

Operations shows credential custody directly: how many redeemed credentials are held by relays right now, how many have ever been handed out, and how long the oldest live one has been held — counts and one age, never a tenant, agent, reference name or value. A live count that does not fall, or an age past the maximum claim lease, is a stuck attempt holding material.

The relay executor ships. internal/agent/relay in the agent binary claims connector.deploy, redeems the credential for that attempt, builds the same connector implementation the control plane would have built — same constructors, same sandbox, same capability grant — drives the appliance over its API from inside its own segment, wipes, and reports. It is armed by --relay-claim and only when the agent's certificate actually carries the network role; an agent without the role says so at startup instead of polling forever and presenting as a stalled queue.

The order inside an attempt is deliberate. Work this build cannot execute is refused BEFORE redemption, because a credential redeemed for an attempt that was never going to run is material outside the seal for nothing — and it burns the attempt's one redemption, so no other agent can take the work either. Redeemed values are moved straight into locked buffers and the wire copies wiped, so the only surviving copy is the one destroyed on the way out, including on panic. What a connector or an appliance says on failure is never forwarded: the relay reports a closed phrase and keeps the target's words local, because an appliance can and does echo the credential it received back in an error body. A sandbox denial is reported as a failure, not as a deploy with a footnote.

Ten connectors are relay-executable — f5, netscaler, a10, kemp, cisco, fortigate, paloalto and, since A59, aws-acm, azure-keyvault and gcp-certificate-manager — and the Agents console shows exactly that set for network-role agents, derived from the agent package's own census so the console cannot advertise an executor the binary lacks. Host-role agents see their host job kinds and host connector families instead. The server uses that same job-kind role census at claim time. That list describes this server build and certificate roles. Since A62 each agent also reports its own executor census on its heartbeat — the job kinds and connector families the enrolled binary ships, its version, and the enabling flags that process was started without — and the console shows that reported list, role-filtered the same way, with the server census beside it (capability_source says which one describes the agent; an older agent that has never reported one shows the server census alone). The report authorizes nothing: certificate roles and the server census still gate every claim, and terminal receipts remain the evidence that work completed.

Host connector execution ships too. The thirteen file/exec connectors — nginx, Apache, Caddy, HAProxy, IIS, Postfix, Traefik, Java keystore, PostgreSQL, MySQL, RabbitMQ, Elasticsearch, Tomcat — now execute on the host agent that serves the machine, not against the control plane's own filesystem. The connector implementations moved unchanged: they were always host-neutral, and what changed is which filesystem they resolve against.

Envoy is the fourteenth host connector. It pushes SDS over HTTP instead of writing files, but the shipped target is a co-resident loopback listener. The agent therefore supplies its own HTTP client and the control plane refuses the row exactly like every file/reload family; transport shape does not make a loopback address reachable from the right machine.

The exec profile moved with them, and had to. An allowlist naming /usr/sbin/nginx is a statement about a host; leaving it on the control plane while the exec happened on an agent would mean an operator authorizing a binary on one machine and a different binary running on another. It is now a file on the host (--host-exec-profile), read and validated at agent startup so a mistyped path surfaces when someone is watching rather than an hour later during a renewal. Without it an agent claims no file/reload deploys at all: there is no safe default for "which commands may run on this machine", so an absent profile refuses rather than permits. NewLocalOps re-canonicalizes the roots and re-Lstats every command on the host that will run them, which is the point — the check and the execution finally happen on the same machine.

One binary serves both vantages. A relay claims appliance work, a host agent claims file/exec or co-resident Envoy work, an agent granted both roles claims both, and the per-row role demand stamped at enqueue decides which agent may take a given job. The two executor sets are disjoint by test.

Rollback executes now through two custody-safe models (D4/G1). After CSR-first issuance the control plane holds no subject key, so it never rebuilds a predecessor bundle and never sends one back down.

The executable form is a re-BIND. The predecessor is already installed on the appliance; what a deploy changed was which installed object the listener points at, and a rollback points it back. Nothing is uploaded, no key moves, and the operation is possible precisely because the control plane holds nothing.

That required a change to deploys, in addition to a new operation. Every appliance connector installed under a name derived from the target, so each deploy overwrote the object before it — there was never a predecessor to bind back to. Deployments now install under a name carrying the certificate's fingerprint, which makes two deployments two objects (and keeps deploys idempotent for free, since the same certificate computes the same name). The listener-facing object — an F5 Client SSL profile, a NetScaler certkey, an A10 client-SSL template, a Kemp virtual service — keeps its name, so existing bindings are untouched.

Six appliance families ship re-bind: f5, kemp, netscaler, a10 — the ones whose API addresses an installed object separately from uploading one, which is the property a re-bind needs — and fortigate and paloalto when the target names the binding to move: FortiGate bind_to (admin re-points system.global admin-server-cert, ssl-vpn re-points vpn.ssl.settings servercert) and PAN-OS ssl_tls_profile with commit_admin (the shared SSL/TLS service profile is set and a partial commit scoped to that administrator pushes only its own changes). One cloud store joins them (A58): gcp-certificate-manager with certificate_map_entry (<map>/<entry>) creates a fingerprint-named Certificate resource per deploy and points the map entry at it, so a rollback re-points the entry at the predecessor still in the project; because that family's deploys run on the control plane's egress-guarded client, its rollback row is stamped control_plane, no agent can claim it, and the outbox worker performs the re-bind and records the same rolled_back receipt an agent would. A fortigate, paloalto or GCP target without that binding deploys in place under the target name as before and its rollback is refused at request time as not configured, never reported as done. A cisco target re-binds when it names its ISE node and roles, and one without them is refused as not configured rather than offered a rollback that would return success having changed nothing; aws-acm (one ARN, re-imported in place, no previous version) and azure-keyvault (versions kept, newest always served by name, no call to make an older one current) are refused with those reasons on the route and in rollback_readiness. An operator's rollback also pauses automatic renewal of that identity on that target until an operator resumes it (POST /api/v1/identities/{id}/renewal/resume, trstctl identities renewal-resume): the identity's renewal_paused list and the target's renewal_paused_identities show the hold, the lifecycle plan reports it as paused_by_operator_rollback, expiry alerts keep firing, the scheduler and posture remediation never redeploy there on their own, the identity's other targets keep renewing, and the resume is an audited event (identity.renewal_resumed) carrying the operator who made it. An explicit operator-driven renewal transition is not held by the pause. Plugin-backed connectors are refused because the plugin ABI exports deploy only. A rollback whose predecessor object is no longer on the appliance fails, with a distinct error, because reporting success there tells an operator that a bad certificate stopped serving traffic when it did not.

All 14 host connectors use a different model. After a successful host deploy, the exact enrolled host agent stores active plus one predecessor bundle in a machine-local AES-GCM ledger (--host-rollback-dir, default beside the agent key). The key file and state files are 0600; plaintext certificate/key bytes exist only in locked memory while recording or restoring. A rollback is pinned to that exact agent, reopens the ledger after restart, restores the predecessor, runs the same connector reload, and repeats local listener verification. A second predecessor is never retained, so this is a rollback buffer rather than a key archive. Losing that directory means the host predecessor is gone and the job is refused; the control plane cannot reconstruct it.

Two things to know before relying on it. First, an install that predates fingerprint naming holds its last certificate under the old target-derived object name. An appliance rollback tries the fingerprint-named predecessor first and, when none exists, re-binds to that legacy object (f5, kemp, netscaler, a10); the receipt then reads rolled_back_to_legacy_object and says the object's contents were not re-read, so verify the listener independently. A host agent adopts the certificate and key it finds installed before its first deploy to a target as its retained predecessor, reports the adopted leaf's fingerprint inside its signed receipt together with the public leaf, and the control plane records that leaf as an observed certificate and names it as the target's predecessor, so the first trstctl deploy over a pre-trstctl install is rollable through the served route; an agent that did not retain the pair still refuses the restore. The control plane also names as predecessor the latest delivery to a target by any host agent, so a re-enrolled agent can restore what the host served. A target with no earlier trstctl delivery and no reported adoption becomes rollable after its second deployment. The single-target read (GET /api/v1/connectors/targets/{id}, the Connectors console's destination detail) now says so up front: rollback_readiness applies the same refusals the rollback route applies — no executable route for the family, a fortigate or paloalto target without its binding, no bound identity, no predecessor certificate, a revoked predecessor, a predecessor never delivered to this target through trstctl under its fingerprint-carrying name, or a host target without a second successful agent deploy — and names what would make the target rollable. A refused rollback still reports that the predecessor object is not installed, which remains the truthful answer at execution time.

Second, objects now accumulate. Each deployment leaves its predecessor on the appliance rather than overwriting it, which is the entire point, and nothing prunes them automatically: trstctl removes an object from a customer's appliance only on an explicit takedown request (f5, kemp, netscaler and a10), never as housekeeping. On a target renewed every 90 days that is a handful of objects a year; on a short-lived-certificate target it is not, and operators running those should expect to prune. Automatic pruning is deliberately not served: deleting a crypto object that something else might be bound to is a worse failure than leaving one behind.

Deploy and rollback now use the same connector.bind:target:<id> effect lane. The agent claim query admits one unexpired holder per lane and returns at most one row from that lane in a batch, so two agents cannot race a deploy and restore against the same listener. Unrelated targets remain parallel.

Automatic rollback is served when a target explicitly sets auto_rollback_on_verify_failure; manual console rollback queues the same command. Unsupported connector families and targets with no predecessor are refused at request time with no memo-only success receipt. A plugin-backed connector executes on a network relay once one of the tenant's relays has reported, in its signed census, that it loaded that plugin (A63): the deploy and preview rows are stamped for the network role from that census at enqueue, the control plane leaves them for the relay while it is enrolled, and falls back to its own plugin surface only when no relay carries the plugin any more. Plugins have no rollback. connector.right_size executes on a network relay when its connectors.right_size binding sets executor: network (A64): the relay redeems the binding's token for one attempt, runs the same mutate-and-readback conversation as the control-plane worker, and its signed report becomes the operation's terminal receipt; the server verifies the report's shape and digest, not the endpoint, so the readback evidence is the relay's. Envoy TLS posture changes run on one pinned agent: a co-resident Envoy's on its assigned host agent, and a remote Envoy that declares vantage: network on the network relay its required_agent_id names (A104), which keeps the predecessor in its own encrypted store so the rollback returns to it. No job kind is claimable by default: agent_channel.claimable_job_kinds must name each kind before any of this moves, which is deliberate rather than unfinished. connector.rollback is named separately from connector.deploy on purpose — an operator should be able to enable undoing a deployment without enabling deploying, and during an incident that is the order they will want.

A relay runs its own egress policy, not the control plane's. The control plane validates a target's endpoint at admission and drives it through an SSRF-blocking, egress-guarded transport. A relay cannot run that guard: it exists to reach devices on private, non-routable addresses inside its own segment, which is precisely what the guard refuses. Instead every connection a relay opens to run a job refuses link-local addresses, cloud instance metadata, unspecified and multicast addresses, and --relay-egress-allow refuses everything it does not list (see Configuration). What that does not cover:

  • Without an allowlist, the rest of the network is reachable. The always-on refusal covers only the addresses no job needs. An operator granting the network role without an allowlist grants every other address the relay's own network can reach.
  • Addresses, not names. An allowlist names addresses and ranges; a host name cannot be listed, because a connection sees only the address it resolved to.
  • Two probes are checked before they run, not at the dial. The PQC posture and dry-run listener probes may fall back to the system openssl, which dials outside the agent, so their destination is resolved and checked first; a name that resolves elsewhere between that check and the connection is not caught for those probes.
  • Plugins keep their own grants. A plugin's dials follow its grant and the reserved-address refusal plugins always had, not the relay allowlist.
  • The Agents page does not show it yet. The policy in force is printed at the agent's startup; the agent census does not report it.

The agent job ledger: served executors and signed receipts

Work that touches your estate has to execute inside your estate. The control plane has no route into a host and never gets one, so the agent comes and takes the work over the connection it opened — the same mTLS channel it heartbeats on, the same certificate-derived tenant, no inbound port anywhere.

Agentless SSH trust collection uses ssh.trust.collect through the selected network relay. It calls only a fixed root-owned read-only helper using pinned host identity and a vaulted PKCS#8 reference. Every target has an explicit outcome; missing permissions, failed authentication, denied egress and oversized readback cannot become an observed posture. The host-agent collector stays the default.

Approved additive SSH trust uses ssh.trust.apply in its own one-worker loop, with local --ssh-trust-fleet --ssh-trust-confirm plus the staged CA and local command policy. It is not claimed by the shared relay pool. The exact skipped roster, approval and signed readback/health/rollback facts are served under /api/v1/ssh/trust-fleet/rollouts; queued jobs remain pending.

The ledger is the outbox, unchanged. An entry is still committed in the same transaction as the state change that caused it, still carries an idempotency key, still at-least-once. What changed is the consumer: ClaimJobs and ReportJobResult on the agent channel let an enrolled agent lease work, extend while it is still going, and report executed or failed.

A claim is a lease, not an assignment. An agent that is killed, partitioned, or stops calling home leaves work behind; because the claim expires rather than sticking, that work returns to the queue without anyone noticing the machine is gone. Claims use SKIP LOCKED, so a fleet polling in lockstep fans out across the queue instead of serializing on its head. Extend, complete and release all require the caller to hold the lease, so a stalled agent whose lease lapsed cannot report on work another agent has since done. GET /api/v1/operations/jobs, trstctl-cli operations jobs and the Operations console show per-kind waiting and held counts plus the oldest wait — counts only, never a tenant identifier, payload or credential. The same posture read publishes Prometheus series (queue depth and oldest wait per kind, live credential redemptions and their oldest age, claims and refusals as counters), so the metrics and the console cannot disagree about what the fabric is doing. Three alert rules ship in deploy/observability/alerts.yml: a stalled queue alerts on the oldest WAIT rather than depth, because depth alone cannot distinguish a busy fabric from a stopped one; a credential held past the maximum claim lease is critical, because that is live material on a machine whose claim should already have lapsed; and refused redemptions alert at all, because each one is an agent asking for material it did not hold a claim for. trstctl doctor gains FABRIC-1, which sweeps agent-claimable work nobody has taken and names the ROLE the waiting work demands — usually the answer, since work demanding a role no enrolled agent holds waits forever and looks exactly like a busy queue. It sits beside DUR-2 rather than inside it because a control plane that is not delivering and a fleet that is not claiming need different runbooks.

Reports are signed, and the signature is what an auditor gets. Every terminal report — executed or failed — carries a detached signature the agent made with the same key behind its channel certificate, over a canonical statement naming the tenant, the agent, the job, the claim attempt, the outcome, the evidence digest, a digest of the report's text and when it was signed. The tenant and the agent name in that statement come from the certificate the caller authenticated with, never from a request field, which is why a forged receipt and a cross-tenant receipt are the same refusal: both were signed over different bytes than the ones the server rebuilds.

This is worth being precise about, because mTLS already authenticates the connection. What the signature adds is evidence at rest. Without it, "agent-7 executed this deploy" is a sentence the control plane wrote about itself, and anyone who can write to the event store can write that sentence. With it, the record is one the control plane could not have produced. The statement and the signature are stored on the event and in a receipt ledger, so the check can be repeated later by someone who does not trust that it happened the first time.

The production event log checks an explicit, closed payload schema for agent claims, refusals, signing, credential redemption, workload issuance and terminal reports. Unexpected fields or undeclared schema versions are refused. Privacy erasure cannot rewrite a signed terminal report: it refuses that rewrite rather than changing the bytes covered by the signature.

Agent audit publication runs after the operation's state change, with the event's identity, time and actor fixed before the first attempt. If the event bus does not acknowledge the append, the server queues that same event, under the same identity, on the control-plane-only agent.audit.retry outbox destination and logs agent job audit event queued for retry with the tenant, event type and event ID, never the payload. The outbox dispatcher re-appends it with the outbox's usual backoff and attempt cap, and because the stream deduplicates message IDs within its 24-hour window, a publish whose acknowledgement was lost is not recorded twice. An append the log refuses for a deterministic reason (a payload the privacy schema does not declare, a missing tenant) is not retried: the server logs agent job audit event could not be stored with the reason. When either line appears, compare the job and retained receipt with the event history; a successful agent response alone does not establish that its audit event was retained, and a retry row that exhausts its attempts is visible as a failed outbox delivery on Operations.

Anything that does not verify is refused fail-closed, with an agent.job.receipt.rejected audit event and a counter on Operations: unsigned, signed by a key that is not the connection's certificate, altered after signing, or signed outside a ten-minute window against the server's clock. That last one is what stops a captured receipt being replayed later — a signature does not expire on its own. Lease extensions are deliberately NOT signed: an extend claims nothing about the world, and signing every keepalive would put the agent's key on the heartbeat path for no evidentiary gain.

Two honest limits. The signature attests what the agent SAID, not what the appliance did — an agent that is lying, or that is wrong about its own outcome, produces a perfectly valid receipt for a false statement; making the claim checkable against the endpoint itself is the verification work (WS-D), not this. For a deploy with a verify address the transcript behind evidence_digest now travels in the report: the control plane checks it against the digest, against its own record of the target and the issued certificate, and keeps it as the verification receipt. A deploy without a verify address has no transcript, and the receipt binds only the agent's signed statement.

What is served, and what is not. connector.deploy has both the network-relay executor from A3 and the host executor for the 14 host-vantage families; a role- eligible agent claims it, redeems its credential for one attempt, and performs the effect from the required machine or segment. Connector test, rollback, endpoint renewal, exact-fingerprint host containment and verification, discovery, revocation probing, AD CS inventory, trust distribution, CMDB/MDM/ticket observation, and self-upgrade also have the executors named in their own sections. The claimable set remains empty by default and agent_channel.claimable_job_kinds is the only way to fill it. That is deliberate rather than unfinished: handing out work nothing can perform fills a queue while the control plane's own worker stops doing it. The kinds the allowlist recognizes — connector.deploy, connector.test, connector.rollback, endpoint.renew, endpoint.contain, endpoint.recovery_check (the read-only check a replacement reads after a containment, enabled with endpoint.contain), endpoint.purge, connector.takedown, endpoint.verify, discovery.run, ssh.trust.collect (explicit read-only agentless SSH sources), revocation.probe, adcs.inventory, adcs.inventory.enrollment_groups (opt-in bounded nested enrollment links), adcs.inventory.endpoint_probe (opt-in scoped LDAP hostname probes with retained pages), trust.distribute, pqc.posture, pqc.posture.rollback, connector.right_size, cmdb.sync, mdm.sync, ticket.sync, agent.upgrade, and agent.self_check (the post-upgrade functional check every polling agent answers, A71) — all have shipped executors. Enabling a kind exposes that already-built executor; it does not turn a placeholder into work. Anything outside that allowlist is dropped even if an operator names it in configuration, so ca.issue and notification.expiry cannot be moved onto a host: those are the control plane's own effects and CA-adjacent work does not belong in the estate.

Agent-signed result receipts are served and verified against the same certificate that authenticated the mTLS channel. Aggregate waiting and claimed counts are served on Operations, together with verified/refused receipt totals. Each agent's live claims are attributed on the Agents page and in GET /api/v1/agents (live_claims: job id, kind, attempt and lease expiry, read from the ledger's unexpired leases, never a payload). agent_channel.max_live_claims_per_agent caps how many leases one agent may hold at once; at the cap the channel withholds new claims until a lease completes, fails or lapses, and the Agents page says so (claim_quota, claim_quota_reached). Unset is the shipped default and means the shared agent bulkhead is the only cap: the quota is an operator choice, not something the server infers from fleet size.

Served status vocabulary: what each status claims

A status string is a claim you act on, so each one has to mean exactly what the code did and nothing more. Three statuses on the deployment surface used to read stronger than the work behind them. They are corrected below, and the correction is enforced rather than remembered.

The enforcement has two closed layers. internal/servedstatus records, for the action-bearing deployment and fleet surfaces, whether an attempt contacted the target, changed it, independently re-read it, or computed a verdict from evidence. The API builds those receipts from registry constants and CI rejects a spelling stronger than its flags or a bare string that bypasses the registry. Separately, docs/status_vocabulary_test.go derives an exhaustive census of every status, outcome, verdict, health_gate, and canary_state field in the generated OpenAPI contract. Each field must bind a closed evidence class to an exact, AST-resolved production Go symbol; a planted future DTO proves the census fails closed. The same gate declaration-checks the assembled negative proofs and console renderers for fleet re-issuance, full-set restore drills, and MDM lifecycle traces. Shared console labels cannot add a green status until its evidence predicate is reviewed.

Status Surface What it means What it does not mean
queued connector delivery Intent committed to the outbox in the same transaction as the state change. That any connector has run.
delivered connector delivery A connector reached the target and applied the credential. That the endpoint is serving it. Live verification is a separate state, served since D2 — see GET /api/v1/endpoints/verifications and the endpoint verification vocabulary below. A delivery receipt says what this control plane did; only a handshake says what the listener answers with.
failed connector delivery The attempt ran and did not succeed. —
config_validated connector delivery POST /api/v1/connectors/targets/{id}/test resolved target metadata, schema, and credential references locally, because an agent-owned target has no eligible connector.test path enabled. Cloud-store targets do not use this fallback. That the target was contacted, reachable, or willing to accept the credential. Nothing was changed.
verified connector delivery A connector applied the credential AND a TLS handshake against the endpoint afterward observed it serving that exact identity. The only delivery state that says the certificate was live at that check rather than merely sent. That it is still live now. A delivery receipt is historical. Compare the endpoint verification row's last_checked_at and probe again before relying on a prior pass.
verify_failed connector delivery A connector applied the credential and a handshake found the endpoint serving something else. That the delivery failed. It succeeded; the endpoint did not take it. This is a renewal that did not land, and it is what triggers rollback where a target has opted in.
verified endpoint verification A TLS handshake observed the listener serving the expected identity at last_checked_at. The record carries which comparisons ran — fingerprint always, name set and chain when an expectation supplied them. The summary percentage counts last recorded checks, not current listener state. That it still serves the identity now, or that every vantage agrees. A local row means the serving host's own agent confirmed it then; only a relay row means a client across the segment could get it then. Re-probe after a restart or other out-of-band change.
diverged endpoint verification A handshake succeeded and the listener is not serving what was deployed. The mismatch class says which way: fingerprint, sans, chain, expired, not_yet_valid. That the deploy failed. It usually succeeded — this is a renewal that did not land, which is exactly the failure inventory-based expiry alerting cannot see.
unreachable endpoint verification The handshake did not complete, so nothing was observed. A divergence, and emphatically not a pass. An endpoint nobody could connect to is not verified.
not_checked endpoint verification No check has established the current expected identity from this vantage. A prior relay check may exist, but if a newer local deployment changed the expected fingerprint or address, that predecessor check cannot qualify the new generation. That the endpoint is fine. An endpoint with no configured listener address stays here permanently — absence of a current check is not absence of a problem.
dry_run_queued connector delivery An effect-free test was queued for the eligible host/network agent bound to this target or for the bounded control-plane outbox worker that owns its cloud connector. That anything is yet known about the target. No executor has reported.
dry_run_planned connector delivery The correct execution vantage validated every prerequisite a real deploy needs and returned the mutation plan. A network relay proves its appliance credential with a read-only request. A host agent validates its operator-owned roots and logical commands and performs a read-only TLS handshake when verify_address is configured; without one, the plan explicitly says live verification is not configured. That anything was deployed or, when no verify_address exists, that a certificate is serving. The test path never invokes a connector's Deploy, writes a file, or starts a reload command, so zero writes is structural rather than promised.
dry_run_blocked connector delivery The correct agent vantage ran the effect-free test and a real deploy would not proceed. The reason names the step that stopped it. Deterministic blocks are terminal test answers rather than endlessly retried jobs. That the target is broken in every respect — one step failed, and the plan says which.
rollback_recorded connector delivery Legacy evidence from releases that recorded a memo instead of executing. New unsupported/no-predecessor requests are refused and do not create this status. That a rollback executed. Nothing was restored.
rollback_queued connector delivery An executable rollback was queued for the required enrolled agent. Host rows also pin the exact agent that retained the predecessor. That anything happened yet. No agent has reported.
rolled_back connector delivery An agent restored the predecessor and returned a signed receipt. reason=rolled_back_and_reverified additionally proves the same agent handshook the listener after reload; otherwise the detail explicitly says verification was not configured. That an unverified restore is live when the reason does not say reverified.
rollback_refused connector delivery An agent declined the rollback before contacting the target — it cannot execute that connector, no supported inverse exists, the named predecessor is absent, the exact host ledger/profile is unavailable, or the sandbox blocked the operation. That the target rejected anything. It was never reached and is unchanged.
rollback_failed connector delivery An agent reached the target and the family-specific re-bind or local restore/reload/reverification did not succeed. The reason names the failed stage. That the target is broken in every respect, or that the predecessor is absent unless the reason says so.
not_evaluated fleet re-issuance health gate No evidence exists from which a verdict could be computed, so trstctl asserts none. It is not a pass.
passed / failed fleet re-issuance health gate Either an operator attested the verdict, or the replacement-deployment gate was computed from signature-verified agent receipts. That a plain delivery row or an unsigned report proved endpoint health.
planned fleet re-issuance batch A partition of the affected identity set that has not been published. That anything in the batch was attempted.
queued fleet re-issuance batch Exactly one durable outbox command exists for the cursor batch. That the worker or a target has run.
waiting_verification fleet re-issuance batch Replacement work was published and the cursor is waiting for accepted signed endpoint receipts. That silence is success or that a later batch was published.
executed / failed fleet re-issuance batch The batch ran as a mutation unit and its signed verification gate passed or failed. That a later batch necessarily ran.
halted fleet re-issuance batch The batch was never published because an earlier signed verification failed. That this batch or its targets failed.

Two spellings are retired and are no longer written: test_succeeded (now config_validated) claimed a successful test on a route that opens no connection, and a batch completed was stamped at planning time before anything ran per batch. Both remain in the served OpenAPI enum and render in the console, because receipts written before the correction still carry them and removing an enum member would put stored rows outside the contract that describes them.

Making these statuses stronger is real work, not relabeling. The dry-run half is now served from every execution vantage. With connector.test enabled, POST /api/v1/connectors/targets/{id}/test queues a job for the bound host agent or network relay. A network relay redeems the appliance-management credential for one attempt and probes the endpoint with a read-only GET. A host agent validates the target paths and logical commands against its local --host-exec-profile, then performs a read-only TLS handshake against the exact verify_address and optional verify_server_name when configured. If the address is absent, preview explicitly marks listener verification skipped while still planning the deploy; the later delivery can say delivered but cannot say verified. It does not redeem a certificate or private key: those do not exist in a target-only test and are supplied only after a deploy is authorized. It redeems only a host-target secret the preflight actually needs, such as a Java keystore password reference. The three cloud-store connectors queue the same event-sourced job on the bounded control-plane outbox worker. That worker resolves the exact saved target revision, leases its provider credential for one attempt, and performs an authenticated read-only list operation before returning the later import plan.

Both vantages return what a real deploy would change as dry_run_planned, or name the first failed step as dry_run_blocked. Zero writes is structural rather than promised: this path never calls a connector's Deploy, never writes a file, and never executes a reload. A deterministic block is reported as a completed test answer, so an unavailable listener or insufficient host profile does not become a rapid retry and credential-redemption storm. A signed connector plugin answers the test through its test entry point under a read-only grant: no file writes, and HTTP requests limited to GET and HEAD, a write being refused before it is sent. A plugin without a test entry point is blocked before it is invoked or any credential is redeemed. Without an eligible agent path enabled, the route keeps the honest local answer, config_validated, rather than queueing work nothing will claim.

Restoring a predecessor is served as a relay-executed re-bind, not a re-upload. After CSR-first issuance (B1) the control plane never holds the subject key, so the rollback command names the predecessor already installed on the target by serial and fingerprint. A connector that supports re-binding can switch to that object and report rolled_back; missing capability, credential, predecessor, or sandbox permission reports rollback_refused before contact, while a reached target that cannot re-bind reports rollback_failed. Where no predecessor exists — a first deployment — the durable evidence says so rather than sending an operator looking for a credential that was never there.

  • Remaining private CA hierarchy operator flows beyond root/intermediate/leaf issuance. Root/intermediate CA creation, existing signer-backed CA chain import, offline-root import, offline-intermediate CSR generation/import, m-of-n approvals, signer-backed leaf issuance, and configured upstream CA issuance are served at /api/v1/ca/ceremonies, /api/v1/ca/authorities, /api/v1/ca/authorities/offline-roots, /api/v1/ca/authorities/imported, /api/v1/ca/authorities/imported/csr and its ceremony-ID readback, /api/v1/ca/authorities/{id}/offline-intermediates/csr, /api/v1/ca/authorities/{id}/offline-intermediates, and /api/v1/external-cas. Public/private direct-CA discovery is served at /api/v1/ca/discovery (returns configured public/private upstream CAs and imported hierarchy authorities, without PEM or key material). Zero-downtime CA rotation activation is served at /api/v1/ca/authorities/{id}/rotate (marks the predecessor superseded, records the successor's replaces_id, keeps the predecessor issue URL live while new certificates route to the successor). Signer-backed renewal/re-key is served at /api/v1/ca/authorities/{id}/rekey (consumes a ca-rekey-v2 ceremony bound to the authority profile, lifetime, and reason; mints fresh CA key/certificate material, records ca.authority.rekeyed, keeps the stable issue URL live), and cross-signing at /api/v1/ca/authorities/{id}/cross-sign, with durable public certificate readback at /api/v1/ca/cross-signs/{ceremony-id}. Offline-root re-key (/api/v1/ca/authorities/{id}/offline-rekey) and cross-certificate verify/import (/api/v1/ca/authorities/{id}/offline-cross-signs) never accept a private key: operators produce the successor and both cross-certificates on the disconnected root system, then submit only certificates under an exact ceremony purpose (see the key-ceremony runbook). The console's Imports workspace now previews and executes offline-root succession under that purpose. Its event-projected public receipt is read by ceremony ID at /api/v1/ca/offline-rekeys/{ceremony-id} after a restart or a lost response; no private key enters the browser or control plane. The offline-root import profile accepts any key algorithm the platform can verify and label — ECDSA P-256/P-384/P-521, Ed25519, RSA-2048/3072/4096 — and the reviewed signature_algorithm must name the key the certificate actually carries (a root reviewed as one algorithm and carrying another is refused before the ceremony starts); the console's offline import and offline re-key forms select it. CAs the served signer mints stay ECDSA P-256. For offline-root import, ttl_seconds caps remaining certificate validity and is checked before the ceremony starts and again at import.

  • All 14 external CA integrations are served when configured. buildRunDeps constructs tenant-bound AD CS, AWS PCA, Azure Key Vault, DigiCert, EJBCA, Entrust, GlobalSign, Google CAS, Let's Encrypt/ACME, Sectigo, shell CA, Smallstep, Vault PKI, and Venafi TPP/TLS Protect clients. The authenticated served issue route journals the request before the upstream call, and the acceptance proof independently validates the returned chain for every provider. The Kubernetes CSR and TrustBundle posture routes beside them serve the authenticated reports the cluster agents' controllers send, and the same census proves both against a real kind cluster, so CA-agnostic issuance (F4) is served end to end.

  • Discovery collectors with residual connector-owned execution: SSH host-key scanning is served through the discovery outbox worker, and on-host SSH/private-key inventory is served through the agent mTLS inventory report path. Connector-specific external secret-store/API-key scanners remain source-plugin or provider-owned unless a native served source kind supplies findings. The network, ssh, cloud_certificate, cloud_secret, ct_log, drift, k8s_ingress_gateway, and manual source kinds are wired through the served discovery worker (see "Discovery control plane" above), alongside the NHI kinds (nhi_cross_surface, oauth_grant, service_account, nhi_behavior, credential_compromise) and observation-shaped api_key sources; secret-repo and third-party artifact scans dispatch through the same worker from their /api/v1/secrets/scans/* routes. A secret_store source runs through the same served secret-manager connectors as cloud_secret (aws-secrets-manager, gcp-secret-manager, azure-key-vault, hashicorp-vault; the kinds share the providers config shape), and a provider outside that set fails with the connector's specific "unsupported provider" error. One accepted kind carries no worker executor: agent sources report through the agent mTLS channel instead of the worker. On the agent side, the shipped binary collects filesystem certificates, OS/Java/NSS/browser trust stores, and private-key material (each behind its own default-off --inventory-* flag), Windows certificate/trust stores (--inventory-windows-stores) and in-cluster Kubernetes TLS Secrets (--inventory-k8s-secrets, widened across namespaces with --inventory-k8s-namespaces). PKCS#11 token collection needs the cgo agent image: the default statically linked agent cannot dlopen a module and fails --inventory-pkcs11-module with "built without cgo" rather than reporting an empty token, while the released trstctl-agent-pkcs11 image (deploy/docker/Dockerfile.agent-pkcs11, glibc plus SoftHSM and OpenSC modules, vendor modules mounted in) reads CKO_CERTIFICATE objects over a read-only session and is exercised against a real SoftHSM token in the repository's container test. Its live proof is a lab SoftHSM token, not a vendor HSM. Discovery schedules tick server-side: a leader-only scheduler sweeps every minute and queues a run for each enabled schedule whose source has no in-flight run and no run newer than the schedule's interval_seconds, through the same event + outbox path an operator-initiated run takes (runs it queues carry requested_by: discovery-scheduler; a failed run counts as an attempt, so a broken source retries next interval instead of hot-looping; one sweep queues at most 100 runs per tenant). The CBOM scanner is also served, through its own /api/v1/cbom/* API rather than the discovery-run worker.

  • Broker-issued agent credentials can be task-scoped: POST /api/v1/broker/agent-identities accepts an optional task_envelope_base64 (the AGID-05 task envelope) and returns the task_envelope_digest the credential binds, so an AI/MCP agent badge can be scoped to one authorized task instead of standing scope alone. Verification is the core AGID gate's: the requester signature is checked over the envelope's canonical bytes against an operator-provisioned requester key (the caller cannot supply its own), plus the expiry window, and the bound digest is the verified envelope's own — a substituted envelope cannot be bound in place of the signed one. Every build attaches this gate; an unverifiable envelope is refused, not ignored: silently returning an unscoped credential in place of the scoped one the caller asked for would be the dangerous outcome. Requests carrying no envelope are the ordinary single-hop badge, unchanged.

  • A migration can be reviewed before it runs: POST /api/v1/pqc/migrations/plan (and trstctl-cli migration plan, available in every build) previews the plan — which assets would be re-issued and to what, which TLS findings would be rolled out, and the residuals it will not touch — without queueing a run, minting a run id, or writing an outbox row. It calls the same plan builder the start path calls over the same CBOM assets, so the preview cannot describe a different migration from the one that would execute. Both paths also preflight each exact TLS deployment target; an Envoy host-agent target requires an enrolled, assigned host, co-resident management endpoint, and loopback served-listener binding. A TLS finding without an explicit operator target binding returns HTTP 400. An ML-DSA host-csr re-issue is refused by the host agent's name when that agent's executor census reports no tls_probe_openssl in its host execution profile (the native probe the agent needs to verify a served ML-DSA leaf; reported_host_tls_probe on the agent), so the run is not queued only to fail fifteen attempts; an agent that reports no census is admitted, and its own refusal phrase then names the cause in the finding. The same census carries the binary's own reason when its host execution profile could not be loaded (reported_host_profile_error on the agent; for example a tls_probe_openssl that is a symbolic link, which the agent refuses while naming the resolved path to pin): that relay claims no host work, so a connector target deploy or an ML-DSA host migration bound to it is refused with HTTP 409 naming the agent and the reason, instead of waiting on a claim that never comes. The same holds while an agent has paused job claims over a pending job report it cannot open under its current registration (reported_pending_report_error: after re-enrollment, a changed CA bundle or server name, a missing sealing key, or damage); it keeps heartbeating and renewing its own certificate, and claims resume after the operator restores the previous settings or runs the agent once with --set-aside-pending-report. A host renewal whose report was set aside keeps its retained key in the host rollback state: a retry of that job resumes with it, but if the control plane fails the job instead, a later renewal of that target is refused as "the encrypted pending host key belongs to another job" until that renewal is rolled back on the target, which clears the slot. When the certificate was issued but the host could not install it, the agent's receipt names the step that failed (the host sandbox refused the write, the connector's configuration test refused the new pair, or the reload action failed) and the finding carries that phrase; the underlying error stays in the agent's own log on that host and is never served.

  • Signing history is verifiable after the fact: GET /api/v1/code-signing/identities (and trstctl-cli code-signing identities) lists recent signing operations with their identity kind (managed for a signer-held key, keyless for an ephemeral Sigstore/Fulcio identity) and the transparency-log state of each — verified, pending, failed with its reason, or not-published. Verification is not a separate flag that could drift: Rekor publication rides the outbox and the handler refuses to acknowledge an entry whose signed receipt does not verify, so a delivered row is a verified entry. The view reads no sealed command bytes, so the plaintext identity assertion and the artifact digest never leave the signer boundary through it.

  • The SSH key estate is readable: GET /api/v1/ssh/fleet (and trstctl-cli ssh fleet) rolls the tenant's discovered SSH key locations up with standing-access and orphaned counts, key types, and the observation window, worst location first. A raw public key can also be the key inside a CA-signed host certificate. The same view now answers whether a host is under the SSH CA (A115): each host agent run with --inventory-ssh-sshd-configs reports its sshd configuration's trust posture, and host_trust judges it against this tenant's SSH CA and served KRL: which CAs sshd trusts for user certificates (this CA only, this CA and others, another CA, none), the host certificate it presents (signed by this CA and valid, renew soon, expired, another CA, a signature that does not verify, none), whether its RevokedKeys file is the current list (by digest), another list, a plain key list, missing (sshd then refuses every key) or absent, and whether authorized_keys still logs in. The control plane names each host from the agent's certificate. Limits: the agent reads the configuration file's global section the way sshd resolves it (first value wins, HostCertificate accumulates, Include in place) rather than running sshd -T, so Match blocks are counted, not applied; the posture is reported when the agent connects, so a changed configuration shows after the agent restarts. It reports metadata only (fingerprints, key types, paths, digests); no private key material is read or stored by discovery.

  • The connector catalog reports sandbox truth, not description: each row in GET /api/v1/connectors/catalog carries native (this build has a native implementation), capabilities (the declared sandbox grant — fs.read, fs.write, net.dial, process.exec), and replay_safety (reconciled when the receiver converges on retry, otherwise at-most-once). Those come from the live registry, so the catalog cannot claim a capability the process would not enforce. Each row also says who executes the family in this estate (executor: host_agent, network_relay, control_plane, or none — listed for its description, nothing runs it until the family is enabled here or a relay reports a plugin by that name) and where its grant was read from (grant_source): a factory-registered family declares its operation kinds at registration from a prototype of its own connector (A75), and a family an enrolled agent executes reports the shipped binary's census for the same connector code (A74), so a host family the control plane never enabled for itself no longer reads as having no implementation. A real attempt's per-target grant narrows the declared kinds to the target's paths and host; it never widens them.

  • The running system is readable: GET /api/v1/platform/system (and trstctl-cli platform system) reports the build version/commit/date, the Go toolchain, process start time and uptime, the live signer topology (child/external/none), whether the FIPS module is active, and per-dependency reachability for the database, event log, and signer. It reuses the same probes as /readyz, so the console and a load balancer cannot disagree about whether the spine is up; it reports no addresses, DSNs, or configuration values. The same response and the System health console page, under Configuration evidence, includes idempotency_results: RLS-scoped counts of legacy, sealed, pending, and indeterminate mutation responses, plus the explicit fleet-readiness assertion and whether PostgreSQL has the validated sealed-only floor. Result bytes and raw datastore errors never enter this view; partial, failed, empty, recovery-required, and complete states carry concrete operator recovery guidance.

  • Worker-pool backpressure is readable: GET /api/v1/operations/bulkheads (and trstctl-cli operations bulkheads) reports each bounded pool's workers, capacity, queue depth, saturation, and its submitted/completed/rejected/panicked counters, so AN-7 pressure is visible without scraping the metrics endpoint. The counters are process-wide operational telemetry — subsystem names and numbers, never tenant or credential data — and a control plane assembled without the bulkheaded surfaces answers served: false rather than 404.

  • SSH trust rewrite (the privileged authorized_keys/CA-trust mutator): the applier that installs a trusted SSH CA and rolls it back on failure is wired into the trstctl-agent binary behind a default-off operator opt-in (--ssh-trust-add-ca) that additionally requires explicit confirmation (--ssh-trust-confirm) before it rewrites trust. The op is additive (it never removes existing trust), validates the new config with sshd -t, reloads, runs a separate operator-supplied post-reload health command (--ssh-trust-health-cmd) as a validated argv command line rather than a shell string, and auto-rolls-back to the last-known-good on any failure — so a bad rewrite cannot lock operators out. Reload success alone is not treated as health. Because weakening sshd/authorized_keys trust is a high-blast-radius mutation, the feature stays off unless the operator turns it on and confirms; with the flag off the agent only discovers SSH trust, it does not mutate it. Trust removal still requires its own explicit confirmation.

  • Approved SSH fleet trust changes (A226) are served through /api/v1/ssh/trust-fleet/{preview,approvals,rollouts}, the matching ssh trust-fleet CLI, and SSH access → Trust rollout. Each enrolled host still needs local --ssh-trust-fleet --ssh-trust-confirm, a staged public CA and its own file/command policy. Skipped hosts stay in the exact reviewed roster and receive no job. Default two-person approval binds the CA, deadline, ready policies and skipped hosts; the requester cannot approve. ssh.trust.apply jobs have their own one-worker loop, and signed receipts prove file readback, effective global trust selection, validation, reload and the configured health command. Pending work never implies success; verified rollback stays attention_required. Completion must match retained source evidence. New work expires after its reviewed window; delayed jobs refuse before writing. This is additive user-CA trust: no remote command selection, trust removal, authorized_keys pruning, or host-certificate install. Backups are memory-only, so process/power loss can interrupt rollback. Do not edit host files concurrently; keep independent recovery access and backups. A receipt proves the configured checks at checked_at, not future host state. Real stock-OpenSSH served tests cover two applied hosts, an unchanged opt-out, strict host-key pins, retry and a checked failed-change restoration. Final image/console/restart replay and covering release gates remain qualification work; see the runbook.

  • Posture collectors and agents: CT-log monitoring and path-based credential drift detection now run through the served discovery worker when operators create ct_log or drift sources; findings are tenant-scoped and alert intents are outbox-backed. The Posture page has dedicated CT-log and drift dashboards (A105): each lists its own source kind's findings a page at a time, filters them by triage status, and shows counts the server takes over every finding of that kind, not a first-page sample. Resolution workflows (A106) run through POST /api/v1/discovery/findings/{id}/resolution and discovery findings resolve: investigate with an assignee and a due date, record a fix with its evidence, accept the risk until a time within a year (the leader's sweep reopens it at expiry), or reopen. Posture reads leave a fix that held and an unexpired acceptance out of the open posture and count them; a fix seen again after it was recorded counts as open. The console's CT-log and drift dashboards offer the same actions on each row, show assignee, due date, overdue, acceptance expiry, fix evidence and regression, and filter by every status with the server's counts. Automatic remediation exists for certificate posture findings whose tenant rule asks for "remediation": "renew" (the lifecycle scheduler renews the deployed identity), not for SSH-trust or drift findings. Generic agent/endpoint discovery is served through the mTLS ReportInventory channel and visible through /api/v1/agents, /api/v1/discovery/findings, /api/v1/graph, and the Agents console. The default SSH host/trust collector runs on the enrolled host agent. An explicit ssh_trust source can instead use an enrolled network relay, pinned host keys and vaulted PKCS#8 credential references to call the installed read-only trstctl-ssh-collect-v1 helper (A227). Its ssh.trust.collect job reads the on-disk configuration through sshd -T and only root-allowlisted public CA, KRL and host-certificate files. The target needs no running agent; it does need Python 3, the fixed root-owned helper/policy and a restricted forced-command account. Match contexts, per-user keys, other SSH files and whether the running daemon loaded that disk configuration remain unassessed. Read failures and denied paths stay unknown; the remote target, source and collecting relay remain distinct in the fleet view. The key is encrypted at rest and in transit and is borrowed into locked relay memory for the bounded attempt; no credential value enters configuration, events, outbox or reports. See agentless trust collection. The credential graph and risk-scoring read APIs (/api/v1/graph*, /api/v1/risk/credentials, /api/v1/risk/contextual-priorities) are also served. The contextual-priorities response carries a canonical urgent_summary: it merges the named credential-score and contextual-priority projections for one tenant, exposes both source counts, and deduplicates the union by credential_id. Dashboard and Risk use that same answer, and a projection read failure is unavailable rather than zero. Critical/high discovery findings enqueue one same-transaction notification.risk / risk.urgent outbox intent from the same score band. status=complete means every named projection read completed; it does not mean an undiscovered machine is safe, because discovery coverage still bounds what these projections can know. The AI/RCA/MCP surface is also served behind ai.enable_api.

  • Ownership depth (I1): owners carry an application/service model — application_id, service, business_unit, environment — plus a stored escalation_chain, attributed ownership_verified_by, verification time, and digest of the exact application/environment pair a human confirmed. The stored chain is deliberately distinct from the computed approver snapshot: the snapshot answers who could approve right now, the chain answers who to wake and in what order, and the approver graph cannot answer the second because it is about responsibility rather than permission. Every field is nullable and EMPTY MEANS UNKNOWN, never "none" — an estate predating this model has owners nobody can retroactively classify, and treating blank as a deliberate answer would hide exactly the rows the queue exists to surface. Store.ListUnownedIdentities reports four distinct reasons rather than one boolean (no_owner, owner_missing_application_model, ownership_never_attested, ownership_attestation_stale) because they need different actions: a data-entry gap, a classification gap, a first trust decision, and a repeated decision. Editing application ID or environment invalidates the prior digest immediately; no timestamp is allowed to pretend it confirms a different model. New steady-state deployments require current evidence at the configurable cadence (90 days by default) or an active identity-bound exception with authenticated grantor, reason, and expiry of at most 30 days. The scheduler writes one immutable request and one outbox notification per stale verification edge. Served through the owner and ownership-exception APIs, trstctl owners attest/exceptions, and the Owners console, which can create/edit the full model, attest it, separate all four queue counts, and grant a bounded exception. Expiry is enforced from event time and needs no cleanup timer to become effective.

  • Asset-specific ownership assignment: POST /api/v1/ownership/assignments, trstctl owners assign, and the Owners action queue accept one durable owner, 1–100 canonical NHI inventory IDs, and an attributed reason. The mutation is tenant-scoped, permission-gated, idempotency-protected, and capped at 2,000 reason characters and 1,024 characters per inventory ID. It emits one immutable ownership.assigned event. The projector updates the current override and, for native identities or certificates, the lifecycle owner_id in one transaction. A cold replay reconstructs both. An asset-specific decision wins over older native or imported owner hints; its event ID stays visible as exact attribution evidence. The NHI inventory and every NHI posture route honor the assignment for every source, so a reassigned discovery finding, API token or agent is no longer reported orphaned.

  • Certificate CI export is opt-in and separate from ownership synchronization. A tenant can queue a certificate at POST /api/v1/integrations/cmdb/certificate-exports only through its own enabled deployment-operator write binding. The binding pins the HTTPS instance, registered discovery source and file-backed or tenant-secret credential reference. The outbox worker sends public certificate metadata through ServiceNow IRE with the class fixed to cmdb_ci_certificate, then independently reads that exact CI and compares every exported field before recording a verification receipt. An HTTP success alone never completes it. /api/v1/integrations/cmdb/certificate-exports/{id} retains the verified receipt after queue retention; an unverified command without a queue row reports recovery required. The Connect other tools console and trstctl-cli cmdb certificate-exports create|show expose the same result. Exports are requested explicitly; later lifecycle changes need another export. Rows lacking certificate bytes need verified DER recovery first. The TLS stand-in proves one certificate CI, duplicate-safe retry, revoked state, independent readback and tenant isolation; a live ServiceNow instance with its certificate class, identification/reconciliation rules, discovery source and read/write ACLs is still required for vendor qualification.

  • Ownership provenance and CMDB reconcile (I2): owners record WHERE an ownership claim came from — ownership_source (unset / manual / csv-import / cmdb), ownership_source_ref, ownership_source_observed_at. All three are nullable and unset means UNKNOWN, never "manual": an unrecorded origin is not evidence a human said so, and the Owners grid renders it as "not recorded". Two ingest paths write it. POST /api/v1/owners/import (trstctl owners import) takes a CSV; PUT/GET /api/v1/owners/cmdb-schedule (trstctl owners cmdb-schedule set|show) configures a per-tenant, leader-only ticker that re-reads ServiceNow cmdb_ci on the tenant's own interval. Both funnel through one reconcile rule: a source FILLS IN what nobody recorded and NEVER overwrites what a human attested — that becomes a row in owner_ownership_conflicts, carrying both values and the source record that caused it, surfaced in an "Ownership disagreements" panel on the Owners console. A change to a value nobody attested IS applied and is still listed, because a change nobody was told about is how ownership data quietly stops matching reality. A blank cell is silence, not a deletion. Read-only is structural, not a flag: the network relay builds only GETs against a fixed /api/now/table/cmdb_ci, and orchestrator.NormalizeServiceNowTable rejects cmdb_ci for ticket writes. Neither path can acquire certificate-export write authority: that requires its own operator-configured itsm.servicenow.certificate_exports binding. A CI naming an owner this estate has never heard of does NOT create one; it is reported as unattributed, because a CMDB assignment group is not evidence that a trstctl owner should exist and auto-creating would build a parallel estate out of the CMDB's typos. The sync is RELAY-ONLY: the leader commits a tenant-scoped cmdb.sync outbox job, and a NETWORK relay inside the segment claims it over its outbound channel. There is no control-plane HTTP fallback and therefore no private-egress hole into the estate. The relay redeems the ServiceNow token per attempt through the job-credential path (the approved ServiceNow binding sets relay_only: true with a secret:// reference; an env: reference belongs to the brain process and is refused), reads the one permitted table through the shared endpoint builder, parses in place, and reports records — never the raw response, bounded at 8MB. The RECONCILE stays in the control plane on the reported records, so the never-overwrite-an-attestation rule has exactly one implementation. The signed result is bounded before ingest and projected before the job closes; stable event identities make a crash/retry converge. One relay job reads at most 500 CIs in strict sys_id keyset order. A full page commits an immutable page event, the exact next cursor, cumulative read count, current expected count, and the next cmdb.sync outbox intent in one tenant transaction. A short or empty page is the only terminal proof. Dispatch and intermediate pages never set last_run_at; GET /api/v1/owners/cmdb-schedule, trstctl owners cmdb-schedule show, and the Owners console expose read_count, optional expected_count, pages_completed, next_cursor, coverage_complete, and the retained failure. Restart recovery derives the same idempotent next job from the event/checkpoint, so it resumes rather than rereading page one.

    The relay verifies the CMDB's HTTPS certificate against host roots plus an optional operator-supplied --relay-ca-bundle PEM. That file is separate from the control-plane --ca-bundle; adding appliance trust must not widen bootstrap trust. An invalid bundle fails agent startup. After startup a changed bundle file is re-read before the next request (checked at most every five seconds) and takes effect with no restart; an unusable replacement keeps the previous bundle and is reported, with the digest and load time of the bundle in force, as reported_relay_ca_bundle on GET /api/v1/agents and on the Agents page. No skip-verification switch exists.

    The source inventory stores only CI key, matched local owner ID, and the four contributed ownership values. At terminal completion it withdraws a vanished/reassigned CI's still-identical, unattested values, never a human attestation or later edit, and serves changed/removed counts. One page is in flight per tenant; a second due tick leaves that incomplete checkpoint untouched instead of stacking an identical read. Scope, stated exactly: relay execution is mandatory. POST /api/v1/owners/cmdb-schedule/run (trstctl owners cmdb-schedule run, the panel's Run CMDB sweep now) starts a sweep without waiting for the interval, as the scheduler's own durable relay job: it refuses an unconfigured (404) or paused (409) schedule by name, answers 409 naming the sweep already in flight rather than stacking a read, replays under its idempotency key, and starts a fresh sweep after a terminal page. A newly enabled schedule is still due immediately and fires within one scheduler tick. A disagreement is CLOSED through POST /api/v1/owners/ownership-conflicts/{id}/resolve (trstctl owners resolve-conflict), which REQUIRES both a reason and an attributed operator — "resolved" with no explanation tells the next reader nothing about which side was right, and an unattributed judgment cannot be questioned later. The command checks the tenant-bound open row under the projection lock before appending; unknown IDs return 404, and a second decision returns 409. The projection also pins resolved_at IS NULL so a second event cannot overwrite the first judgment. The console provides a reason field and Resolve control for each open disagreement. The CI-to-owner mapping joins on the owner reference's EMAIL first (the page asks ServiceNow for owned_by.email, assigned_to.email and support_group.email beside the display values) and on the owner NAME second, both compared without regard to case; an ambiguous key is no match. A CI whose mailbox and label both miss this estate's owners reconciles nothing and is counted as unattributed; no owner is created from it.

  • Issuance requests as first-class objects (I3): the effect-free POST /api/v1/issuance-requests/preview, the mutating POST/GET /api/v1/issuance-requests, and /{id}/approve|deny|cancel (trstctl issuance-requests preview|open|list|approve|deny|cancel) give a request a real lifecycle — requested then approved, denied, expired, or cancelled, and approved is NOT terminal because issuance can still fail. Collapsing approved and issued would make a request whose mint failed read as fulfilled. Denial and expiry are deliberately distinct: a denial is somebody's decision and REQUIRES a reason (a denial with none teaches the requester only that somebody said no, so they re-ask); an expiry is nobody's, and the leader-only sweep records it with an EMPTY decided_by, because stamping a person on it would put a decision in the audit trail that no human made. The requester can withdraw their own request and can never decide it — self-approval would leave an approval record that looks legitimate while nobody independent looked — and only the requester may cancel, so anyone else closing it is a denial and is recorded as one. Permissions reuse the existing certs:request / certs:issue split rather than inventing a parallel certs:approve that could drift out of agreement with the gate guarding direct issuance. Requests default to a 7-day expiry; the list surface serves closed rows too and counts open separately, because one total cannot say whether a queue needs attention or is merely long with history. Direct API/console requests require owner_id. Preview and submit share one admission path: they validate an optional public CSR, resolve a profile name to the exact active version, and accept only a syntactically valid owner UUID that resolves through the caller's tenant RLS context. Preview returns blockers, key custody, approval authority, and later submission effects but writes no event, projection, identity, certificate, outbox job, or idempotency row and contacts no CA. The console refuses to submit a stale, missing, blocked, or unavailable preview and links to Profiles, Owners, and CA hierarchy for configuration. Missing/malformed identifiers return 400 before event append; missing and cross-tenant owners share one 422 response so the endpoint does not reveal another tenant's roster. Historical requests and intakes without a unique configured owner match can still project with no owner; that absence stays explicit and blocks preparation instead of being backfilled with a guessed account. TICKET-DRIVEN INTAKE is now real: PUT/GET /api/v1/issuance-requests/intake-schedule (trstctl issuance-requests intake-schedule set|show|list|delete) configures named ServiceNow or Jira reads within a tenant. name is a 1-40 character lowercase slug; omitted names and retained legacy jobs bind only to default. The list route is GET /api/v1/issuance-requests/intake-schedules; exact reads and removals take ?system=servicenow|jira&name=<name>. Removal preserves requests and audit history and refuses an unfinished sweep or pending relay job. The Approvals console can configure either provider, shows its exact HTTPS destination, secret reference, ticket source, field mapping and enabled state for review, and retains the idempotency key when a save response is uncertain. Retrying the exact same schedule, even with a fresh HTTP idempotency key, returns its retained sweep cursor without replacing the running job. Changing a schedule during an incomplete sweep returns 409 until that sweep reaches a terminal page. The console reports schedule acceptance separately from relay read coverage. Each (tenant, provider, name) has its own schedule, job lane and cursor, so one table or project cannot replace or suppress another source's progress. An optional owner_field reads an exact mailbox: a ServiceNow scalar or dot-walk such as assigned_to.email, or a Jira scalar or dotted user field such as assignee.emailAddress. Jira privacy settings may hide that field. Exactly one matching tenant owner supplies owner_id; missing, ambiguous or foreign owners remain unassigned. The console shows that absence. Existing requests retain their originally recorded owner; correcting the mapping applies to new tickets, and an unassigned request needs a new ticket/request naming a real owner before issuance. Named jobs require upgraded relays, because an old report cannot identify their schedule. ServiceNow is bounded to one of the four request-shaped tables (incident, sc_req_item, sc_request, change_request); Jira is bounded to one validated project key. Both bounds are database CHECKs, because an unbounded source name would aim the intake token at records that are not this tenant's certificate-request queue. The intake opens one issuance request per ticket, IDEMPOTENTLY by ticket reference: a re-seen ticket opens nothing, and a DENIED request does not reopen — the denial was the answer to that ticket, and a fresh ask needs a fresh ticket. The field mapping is explicit (subject_field, profile_field, optional requester/justification); a ticket missing the mapped subject or profile is SKIPPED AND COUNTED, never guessed at, because an intake that opened requests from prose would fill the approval queue with noise. Requests opened here carry origin=servicenow|jira and the exact provider ticket reference, and the existing lifecycle — separation of duties, denial with a reason, expiry (7 days for intake-opened requests) — decides them unchanged. Each sweep first commits a tenant-scoped ticket.sync outbox job. A NETWORK relay redeems the provider's secret:// token for one attempt and performs the only external call; the control plane has no ServiceNow/Jira HTTP or token fallback. ServiceNow advances by strict ascending sys_id keyset pages and Jira advances by the provider's opaque enhanced-search nextPageToken, with a hard 100-ticket page bound. The signed report must echo the exact provider, sweep ID, input cursor, source references, read count, and provider total. The page event and next outbox command commit together. Dispatch or failure updates last_attempt_at and retained error only; last_run_at and coverage_complete=true appear only after a terminal page whose observed count equals the provider total. The API and console serve read_count, optional expected_count, pages_completed, next_cursor, eligible/skipped counts, retained failure, and terminal coverage separately. Replay, relay retries, scheduler restart, snapshot restore, and cold event rebuild therefore resume the same cursor; stable request/page identities collapse duplicates without skipping later tickets. The GITHUB ACTION lives in-repo at clients/github-action (composite, action.yml + README with the sample workflow): the workflow's ambient OIDC token is fetched with the requested audience, an EC key is generated INSIDE the runner (only the public half travels), and POST /api/v1/workloads/attested-issuance with method=github_oidc returns the certificate. One certificate belongs to the GitHub-signed run_id and run_attempt, scoped by issuer, repository and tenant. A fresh runner on a new signed attempt receives its own certificate. Within an attempt, the same effective command/public key recovers the original; a second key or changed effective command returns HTTP409 even under a new caller idempotency key. Unsigned environment changes cannot allocate an attempt. Missing/noncanonical signed coordinates are refused. Issuance evidence retains each certificate's run ID, attempt and original verification time; earlier certificates expire normally. The Action preserves existing runner key and certificate files, so a new attempt needs a fresh output directory. An unfinished mint whose public template was lost requires its original bookkeeping to be restored; the signer journal refuses changed signing input. Fork pull requests are refused locally before OIDC, and the server still verifies the signed repository_owner against allowed_owners. CI extracts and executes the shipped composite run block against the full served test deployment, then proves success, foreign-owner and same-attempt mint refusals, a successful new signed attempt, and unsigned-attempt refusal. Local native tests also cover concurrent issuers and source-backed recovery after cache cleanup/read-model rebuild. A real Actions run still needs the owner's push. The tag release pipeline packages the Action, SHA256SUMS, and SLSA provenance. Repository state is release-ready for v0.6.0, but that immutable tag is not published until an authenticated maintainer pushes the reviewed commits and creates the tag; the current local remediation does neither. The Action no longer needs an API token: a github_oidc trust source created with self_authenticating: true and an allowed_owners pin lets the workflow send its verified OIDC token as the bearer credential of POST /api/v1/workloads/attested-issuance with the tenant in X-Tenant-ID (the Action's tenant input). That principal is github: plus the repository name (for example github:acme/payments), holds certs:issue on the attested-issuance routes only, and must present the same token as payload_base64; the server refuses to flag a source without an owner pin, and a disabled or revoked source closes the door again. A token input scoped to certs:issue remains supported. The live proof on a real GitHub Actions run still needs the Action published from a pushed tag. Approving a request does not itself mint. The served POST /api/v1/issuance-requests/{id}/prepare command creates or recovers one deterministic requested identity, keeps the request at approved, and returns only the stored public CSR plus a stable request-derived issue key. The console sends that identity through the ordinary guarded requested -> issued transition, so RA separation, policy, optional dual control, profile binding, signer isolation, outbox delivery, and idempotency are not bypassed by the convenience journey. Finally, POST /api/v1/issuance-requests/{id}/complete moves the request to issued only when its linked identity has an immutable issuance transition AND inventory contains its real certificate under the exact canonical issue key. The signing worker also completes this receipt, so closing the browser cannot lose it. Recovery after revocation or retirement records historical fulfillment without reactivating the credential or signing again. Reviewer (decided_by) and original issuance actor (issued_by) remain separate facts; issued_at is the retained certificate-recording time, not the later recovery time. A signer or outbox failure therefore leaves an honest, retryable approved request instead of producing a false green status. The console keeps that approved row visible, explains the interruption, and labels the recovery action Retry safely; prepare and issue repeat with the same deterministic identity and stable request-derived issuance key rather than creating a parallel operation.

  • Attested issuance is reachable (AUD-10, I3 prerequisite): attested_issuance in the config file turns on POST /api/v1/workloads/attested-issuance and POST /api/v1/ssh/attested-user-certs. Before this there was NO config key at all — Deps.AttestedIssuance was never assigned anywhere in production, so both routes were registered, documented, and permanently 503 on every deployment, and the six attestors behind them (including the GitHub OIDC attestor a CI pipeline needs) were constructed by code no request could reach. Off by default is correct for a mint that trades a cloud attestation for a certificate; unreachable when on was the defect. Attestors stay per-tenant from the workload attester-trust API rather than process-wide, so one tenant's trust decision does not become every tenant's. The other two members of this family were wired after their dead Deps fields were found: agent_broker can turn on POST /api/v1/broker/agent-identities, while pam reaches the broker constructor for POST /api/v1/access/sessions and its companion routes. The operator can seed tenant-bound PostgreSQL provider references and SSH host/principal allowlists in the validated pam config block; tenant registrars can add and disable targets through the served console, API, and CLI. An enabled broker can start without targets so the first registration is possible. PAM resolves attestors from enabled tenant-managed workload trust at request time. The target model requires an explicit PostgreSQL readonly/writer role allowlist, and rejects an SSH target without an explicit host, port, and principal allowlist. A Postgres DSN must be loaded by an existing tenant PostgreSQL dynamic-secret provider from a protected file or tenant secret. The injected PAM target contains only that provider ID; its issue and remove effects use the provider's durable outbox. PostgreSQL's native role deadline prevents a worker outage from extending access, while PAM's final expiry event waits for confirmed provider removal. The requester proposes an exact session with POST /api/v1/access/session-requests; a configurable quorum of distinct access:approve reviewers must approve its immutable digest before POST /api/v1/access/sessions consumes that authority. An altered command or target binding cannot reuse the approval. The local browser activation, stock PostgreSQL use and expiry, and paired cold restart have been exercised. Early revocation is now requested through the tenant API and console; PostgreSQL completion waits for native role removal, while SSH completion means the served KRL contains the serial and key ID. Hosts must load the refreshed KRL to enforce rejection. A local J4 host with the current CA rejected a freshly revoked PAM certificate before its expiry and retained that rejection through host and control-plane restarts; this proves one watched host, not central fleet delivery. The provider terminal-failure path now projects revocation_failed with a closed error class and exposes a fresh-key retry in the API, CLI, and console. A live PostgreSQL cross-database dependency exhausted the native removal attempts; its console retry removed the exact role and survived a cold restart. The KRL endpoint returns 503 during a catch-up it cannot finish within its read deadline, and an opted-in host retains its installed list and retries. /readyz proves KRL and audit-read readiness after a restart: ssh_krl reads not ready while the revocation list cannot be brought current and audit_read while audit history cannot be read, so a replica restarted behind the revocation stream or before a restored history's cutover stays out of rotation. A host agent refuses a KRL whose version does not advance its installed one (a list that revokes nothing, such as the empty one ssh-keygen -k writes when a host is prepared, yields to a list of the same version, so a new host reads current before the first revocation), and refuses any KRL, whatever its version, that would stop revoking a serial, key ID, key, or key hash the installed list revokes, so a restored or replacement control plane, which counts its own versions, cannot silently un-revoke anything on a host; before this guard a newer list that omitted an installed serial replaced it. A restored or replacement control plane takes over a lineage by adopting the list its hosts hold (trstctl ssh krl adopt): it records every predecessor revocation for any CA and moves its version past the predecessor's, and a served test proves a host refuses the replacement list before adoption and takes it after, with stock ssh-keygen reading the result. One adoption reads at most 4 MiB and 100,000 entries. Per-host receipts exist for a watcher run beside an enrolled host-role agent: each heartbeat carries a receipt, signed with the agent's certificate key, of the digest and version read back from its RevokedKeys file and how its latest attempt ended, and trstctl ssh krl receipts (and the SSH access console page) places each host against the served list (current, behind, refusing, failing, stale). A standalone watcher still reports nothing, a receipt is the host's own account rather than a login attempt against its sshd, and there is no control-plane push. A lab proof with real hosts moving between two control planes, and receipts from a multi-host lab fleet, remain to be shown. PAM sessions created before the verified-attestation event field was added retained an attestation ID and audit summary. Their verified selectors, claims, and verification time cannot be reconstructed from that start event. The API omits attestation for those sessions and the console names the evidence gap; new sessions project the full verified, non-secret facts after replay. A malformed TTL on any of the three leaves zero so the built-in bound applies — silently substituting a LONGER lifetime than the operator wrote is the dangerous direction, and zero cannot do that.

  • MDM device correlation and per-device enrollment trace (I5): GET /api/v1/mdm/devices and GET /api/v1/mdm/{mdm}/devices/{id}/trace (trstctl mdm devices|trace) join Intune and Jamf device records to SCEP transactions and answer the question a status field cannot: WHICH STEP an enrollment broke at. The trace runs requested -> issued -> installed -> renewing and the distinctions are the feature. A step nobody reported is pending only if an earlier step SUCCEEDED — otherwise a device that never asked would look mid-flight and sit in an "in progress" queue forever. Steps after a failure are unknown, not pending: nothing downstream was attempted, and pending suggests it still might happen. Only the FIRST failure is reported as the break, because reporting the last sends an operator to the symptom rather than the cause. An MDM that could not be reached yields unknown, never "not installed", and the list counts unobserved separately from failed — merging them sends somebody to re-push a profile that is already there. Intune's deviceRegistrationState is ALWAYS unknown for certificate installation: it says that the device is registered, not that a SCEP profile or certificate installed. Intune installation evidence comes from a completed CertificatesByRAPolicy export filtered to the exact enabled SCEP profile IDs. Jamf reads the fixed GENERAL, HARDWARE, and CERTIFICATES inventory sections. Neither provider can produce installed=ok until its exact device record contains the signer-minted certificate serial with an active/valid status; a completed read with a different serial is an actionable failure, while no certificate-specific read remains unknown. READ-ONLY IS STRUCTURAL — inventory uses fixed GET paths (/v1.0/deviceManagement/managedDevices and /api/v1/computers-inventory). The one non-GET provider operation is Microsoft's fixed POST /beta/deviceManagement/reports/exportJobs for the CertificatesByRAPolicy READ artifact; it cannot name a policy, assignment, or device mutation path, and the Graph bearer is never forwarded to the signed report-download authority. There is no MDM write route; TestNoMDMCodePathCanWrite rejects PUT/PATCH/DELETE and any mutation-capable resource fragment. Devices with no matching certificate and certificates with no matching device are BOTH reported, because a correlation that showed only its successes would make an estate look covered by hiding the gaps; the join key is the hardware serial, matched case-insensitively, because device NAME would silently join two laptops an admin happened to name the same. THE SURFACE NOW HAS ITS PRODUCER: PUT/GET /api/v1/mdm/poll-schedule (trstctl mdm poll-schedule set|show) configures a per-tenant, per-MDM schedule and a leader-only ticker re-reads Intune/Jamf on that interval. An identity inventory join remains metadata only; the lifecycle trace joins the MDM hardware serial to immutable protocol.scep.request.observed and protocol.scep.issuance.observed facts emitted by the actual SCEP handler, keyed by the exact CSR common name and transaction. A successful issuance fact carries the inspected signer-minted certificate serial, fingerprint, and expiry; every terminal refusal carries its failed stage and remediation. Correlation IDs and identity lifecycle rows never manufacture success. The sync is RELAY-ONLY: the leader first commits a tenant-scoped mdm.sync outbox intent, a NETWORK relay claims it over the outbound agent channel, and that relay redeems the secret:// token for one attempt. Enabled schedules require required_agent_id naming one enrolled network relay with a fresh authenticated heartbeat. The outbox claim and credential redemption both enforce that exact agent ID. Upgrades disable older enabled schedules without an ID and retire their pending unbound jobs; the operator selects a relay and re-enables the schedule. env: references, control_plane execution, and the old allow_private_endpoint/private_egress_cidrs brain-egress grant are refused. The fixed endpoint is parsed in place and only a bounded typed, signed observation returns. The control plane binds that report to the exact durable job payload, projects it before atomically completing the claim and outbox row, and performs correlation through one shared core; it has no MDM HTTP/token fallback. The in-flight pending check reads the provider from the durable mdm.sync intent: a pending Intune read stops duplicate Intune work but does not suppress Jamf (and vice versa). Each provider's schedule stamps its own waiting or completed outcome instead of reporting invented health. RENEWAL-WINDOW AWARENESS FOR OFFLINE DEVICES is served on the device list: each correlated device with a certificate carries renewal_at_risk and a detail naming the dates, computed from the identity's expiry against the DEVICE'S OWN last check-in (Intune lastSyncDateTime / Jamf lastContactTime — never the poll time, which would make every device look fresh on every poll and defeat the check). A device inside its renewal window that the MDM has not seen since the window opened is flagged: a SCEP device renews by CHECKING IN, so nothing fails before the certificate expires in a drawer — and the count is served apart from failed because nothing has failed yet, which is the problem. The window is 30 days unless the schedule sets renewal_window_days. Scope, stated exactly: no built-in OAuth client-credential exchange — the token reference must resolve to a bearer the MDM accepts, rotated by the operator's own pipeline; a device with no certificate gets NO renewal verdict rather than a warning. The trace's renewing stage is now evidence-backed: a later distinct SCEP transaction supplies its request/result, while a certificate inside its renewal window with no later attempt and a stale device check-in is unknown with instructions to bring the device online and trigger an MDM check-in. The console device row opens this complete trace and shows the first failed stage plus its remediation. Intune report export currently uses Microsoft Graph's beta report endpoint; the operator-supplied bearer must already authorize both managed-device reads and report export because trstctl does not perform an OAuth credential exchange.

  • AD CS coexistence, first increment (F4): the /certsrv transport now REFUSES to send a password over plaintext. Basic is base64, not encryption, and on a plaintext hop anyone on the path reads a credential that can issue from the enterprise CA directly — the compromise is not "an eavesdropper saw a request" but "an attacker can mint certificates the whole domain trusts". The refusal is at CONSTRUCTION, so a misconfiguration fails when an operator sets it up rather than silently on the first production issuance. A domain-joined relay can supply Kerberos/NTLM through a WebEnrollmentConfig.Authenticator, an interface rather than a GSSAPI dependency because the credential belongs in the host's credential store and core must have no code path that could serialize a domain password. Configuring BOTH a password and an authenticator is refused: the authenticator would win, leaving a live domain credential in configuration that nothing reads — which is exactly what somebody later "fixes" by making it take effect. A failing authenticator fails the request and never falls back to Basic, so a transient Kerberos problem cannot put the password on the wire. AD CS certificate-database rows parse into a vocabulary that keeps PENDING distinct from FAILED and from DENIED (a pending request rendered as failed makes an operator re-submit instead of going to get it approved; a human decided a denial), an unrecognized disposition code maps to unknown rather than failed so a code Microsoft adds does not report healthy certificates as broken, serials normalize so the same certificate from two exports is not two inventory rows, and an unparseable NotAfter is counted as a VISIBILITY GAP rather than treated as an expiry in year zero. Scope, stated exactly: this is the transport and parsing layer only. NOTHING YET CALLS THE DATABASE INGESTION — there is no scheduled or served ingest that pulls a CA database into inventory, so issued/pending/revoked state is not reconciled by C4/XREC; no Kerberos or NTLM Authenticator IMPLEMENTATION ships (the seam exists, the relay-side implementation does not); the ACME/EST front-end that would issue a leaf through an AD CS template under a trstctl policy gate is not built; and there is no "modernize this template" console action or AD CS issuer health view.

  • Staged agent upgrades with an automatic canary halt (A5): GET/POST /api/v1/agents/upgrade-campaign, /pause, /resume, and POST /api/v1/agents/upgrade-ring (trstctl agents upgrade-campaign show|start|pause|resume, trstctl agents upgrade-ring) run a rollout through canary, early, then broad rings, and a leader-only sweep advances it. The halt is the product: ONE unhealthy agent halts, with no percentage tolerance, because a tolerance on a deliberately small canary means the ring can never stop anything — which is the entire reason the ring exists. SILENCE halts too; an agent that took an upgrade and stopped answering is the most likely shape of a bad build, so it is never scored as a success. An EMPTY ring halts as well: an unassigned canary proves nothing, and advancing through it would skip the stage whose failure is supposed to stop the rollout, on exactly the fleet nobody has triaged. Resume restarts AT the ring that halted, never past it — skipping ahead would leave the agents whose failure stopped the rollout on the broken build while the campaign reported success — and a halted campaign never un-halts on its own, so a flapping agent cannot resume a rollout nobody re-approved. Pause GATES DISPATCH (the sweep returns early and fleet.Advance refuses independently), not merely the button; a pause that grayed out UI while jobs kept flowing would be worse than none because the operator believes they stopped it. upgrade_ring empty means UNASSIGNED and is never read as broad, and the console counts unassigned separately. Halted and paused are distinct states: one is the machine's finding, the other a person's decision. A campaign started WITH per-platform artifacts (artifacts: [{os, arch, url, sha256}]) DISPATCHES: the sweep hands every agent in the active ring its own agent.upgrade job through the A1 ledger, each row narrowed to that one agent by required_agent_id (the control-plane dispatcher structurally refuses such rows — only the named agent's claim can reach them). The agent — with the -self-upgrade opt-in, which is the MACHINE operator's consent and is off by default — downloads its platform's artifact, verifies the pinned sha256 (a mismatch is refused and reported; the artifact host is a mirror, not a trusted party), swaps its binary keeping the previous one as .old, reports a SIGNED receipt, and restarts (exec on Unix; on Windows the process exits and relies on the service's recovery action — an agent that stays down reads as silence and halts the ring, which is the correct verdict). The ring is then scored against the DISPATCH LEDGER: a signed failed receipt halts immediately with no grace; an agent observed running the target version verifies (the executed receipt alone means "staged" — the reconnected version report is the proof the new build runs); an agent that neither failed nor arrived within the 10-minute grace is silent, and silence halts. Resume re-dispatches the halted ring as a NEW round, so the failed round's receipts stop counting against the retry. Scope, stated exactly: a campaign with NO artifacts is OBSERVE-ONLY — the pre-dispatch behavior, kept for fleets an external mechanism upgrades, where verification is "the agent reports the target version and has been seen since" and nothing is pushed; the console says which mode a campaign is in. The artifact sha256 is operator-supplied — the platform guarantees the fleet installs exactly those bytes, not that those bytes are good. An artifact may also carry its publisher's Ed25519 signature over that sha256: a control plane that pins the publisher key (agent_channel.upgrade_publisher_key_file) refuses to open a campaign whose artifacts it does not verify, and an agent started with --upgrade-publisher-key refuses to stage an artifact that is the pinned bytes but not the pinned publisher's. With neither key pinned the digest is the only integrity control, which is the default. Ring assignment is by hand per agent (POST /api/v1/agents/upgrade-ring) or by one deterministic placement (POST /api/v1/agents/upgrade-rings/auto-assign, trstctl agents upgrade-rings auto): unassigned in-service agents only, heartbeating ones first, then longest enrolled, then by name; the first canary_count (default one) become the canary, the next early_percent (default ten) the early ring, the rest broad; every ring keeps at least one agent, an operator's own placement is never moved, and a fleet with fewer than three unassigned agents is refused rather than planned into an empty ring that would halt the campaign. The rule is served with the result so it can be disagreed with before the campaign starts. A campaign opened with soak_seconds holds the next ring until that long after the current ring was dispatched (for a campaign that dispatches nothing, after it entered the ring), and only while the verified ring stays healthy (the reason says until when; an agent that stops heartbeating during the soak is silence and halts). Recording the soak neither restarts the window nor dispatches the ring again. Zero, the default, advances a verified ring at once, so the 10-minute grace is the only wait unless an operator asks for a soak. Verification is more than a health signal: the first sweep that sees an agent on the target build queues it an agent.self_check job pinned to that agent (enabled together with agent.upgrade), and the new binary claims it, exercises what an upgrade can break without stopping the process — that it is the build the campaign targets, that a host executor's exec profile still names roots and commands that exist on the machine, that the relay's plugin runtime loaded — and reports a signed receipt naming each check. The agent verifies only once that receipt says every check passed; a failed check halts the ring with its own reason, and a check unanswered past the grace is silence. The campaign view and console tally the live round's checks. The check is bounded on purpose: it proves the agent still works as a job executor, not that every connector it carries deploys correctly on this estate.

  • Provider tenant templates: /provider/v1/templates records versioned templates of customer objects, /provider/v1/tenants/{id}/template onboards a customer from one (registering its core tenant when needed), .../template-drift reads its drift, and a rollout applies a new version across customers after a preview whose fingerprint it must match. What a template can say is what core holds: "roles" are role grants to named members, built-in or a custom role the tenant has already defined (a template does not define custom roles); a CA binding is an issuer record (a changed chain binds a new issuer and the old one stays for what it issued); connector defaults are deployment targets. A template's report schedule can carry dispatch, announcing each run on the customer's channel of that type. The Provider console does not show templates yet; they are served on the Provider API. A console view is what comes next.

  • Provider console (L3 + AUD-58): the web app has a /provider route — a console for the provider's own staff, separate from the tenant plane. It lists customer tenants with their lifecycle state and drives the /provider/v1 API the plane already served but no web client consumed: provision a customer, suspend, offboard, each mutation confirmed because one click changes a whole customer's world. The plane's operator is authenticated by the provider IdP (L1), so the console carries an operator bearer rather than a tenant session, and holds it IN MEMORY ONLY — never web storage — per the SPA's XSS posture (SURFACE-I01): the token lives in the tab's JS heap, is lost on reload, and an auth refusal returns the operator to the sign-in gate rather than showing an error. Each customer row expands to show its quota (max agents, certificates, secrets), reading /provider/v1/tenants/{id}/quota; an UNSET limit renders as "unlimited", never zero, because a missing cap is the absence of a limit, not a limit of nothing. Scope, stated exactly: this is the tenant-lifecycle console with per-customer quota MANAGEMENT — each customer row expands to view and EDIT its limits (max agents, certificates, secrets), a blank field saved as unlimited never zero — and per-customer WHITE-LABEL BRANDING: a brand editor (product name, custom domain, login message) that reads the current delegated brand and writes through /provider/v1/tenants/{id}/brand with If-Match, a route wired to the tenant-scoped provider authority event receiver and its white-label read projection, behind the same per-customer delegation gate (an operator can brand only a customer they are delegated, because a custom domain is a claim on a host and branding another's customer could seize it). A duplicate custom domain is refused before event append, and replay retains a rejected outcome for older conflicts. Customer lifecycle, quota, brand, delegation, and break-glass state now rebuild exactly from one immutable provider authority history; the six PostgreSQL views, now including provider_operators, expose no production mutator. Every provider mutation requires a key bound to operator + method + path + body, and the console sends one; identical sequential or concurrent retries return the original HTTP bytes, while changed commands return 409. The console also triggers the served siloed-isolation drill and shows its checks. Its Authority activity panel reads GET /provider/v1/activity and renders newest-first event id/sequence/type/time/customer/actor evidence derived from the same immutable authority history. Current delegation filters customer events before serving, global drill evidence is admin-only, and the response deliberately omits command bindings, authority payloads, and break-glass snapshots. AUD-58 adds the Provider-admin/MFA access panel. It lists SCIM operator lifecycle/source/role and exact customer-operation grants with expiry, last use, and retained revocation; it grants, revokes, and changes the Provider role through idempotent event-backed routes. SAML login is discovered from /provider/v1/auth/methods; the browser uses the separate HttpOnly Provider session and sends only the non-credential CSRF value from JavaScript. OIDC bearer input remains as a memory-only option for operator tooling. The local trstctl provider-grant command remains the install-time bootstrap path; AUD-59 adds invoice evidence in the same console: an operator selects a delegated billing customer and closed period, pulls the signed JSON without impersonating a tenant, sees billable/reconciliation/digest plus independent Signature verified posture, and downloads either the signed JSON or the finance CSV whose every row retains customer, period, verdict, and digest. AUD-60 adds Customer health beside that Invoice evidence for the same selected customer. GET /provider/v1/tenants/{id}/health requires the exact customer's read delegation before DirectTenantSnapshot opens the customer's forced-RLS certificate transaction. It reports lifecycle-derived health and the active-certificate count; a missing customer is 404 and an unavailable snapshot is an explicit 503. Before a pull, or after a failed read, the console says health is unknown/unavailable — it never turns missing telemetry into a healthy zero. Another customer's active certificates cannot enter the count because the query carries and is confined by the selected customer's tenant identity. The break-glass request, consent, and result-use APIs are exposed in this console (A93): the exact-customer emergency panel lets a requester open a bounded request without offering self-approval, offers consent only to a distinct approver on the selected customer, shows the narrow result only to the requester after two approvals, names a requester withdrawal apart from an approver denial, and shows the queue without emergency mutation controls in license read-only mode.

  • AD CS certificate-database lifecycle visibility (F4, PARTIAL): POST /api/v1/adcs/ca-database/ingest and GET /api/v1/adcs/ca-database (plus trstctl adcs ca-database ingest|list and a Posture console panel) turn certutil rows a domain-joined relay collected into a per-CA breakdown by disposition — issued, PENDING a CA manager's approval, revoked, denied, failed. Pending is kept distinct from failed and denied throughout, because the fix for a pending request is approval, not resubmission; an unrecognized disposition code is unknown, never folded into failed; an issued row whose expiry could not be read is counted unparsed; and a row carrying no request id is counted rejected, never dropped — so a collection problem cannot read as an empty or healthy CA. This wires adcs.ParseDBRow and adcs.Summarize, which existed with NO production caller — parsing that nothing ran and a summary nobody saw. It is deliberately VISIBILITY, not control: trstctl reads the CA database, it does not approve or revoke through this surface. Scope, stated exactly: the control-plane ingestion, projection, serving and console are built and proven end-to-end (a relay posts rows, the per-CA summary reads back, a re-sweep replaces rather than accumulates); the relay-side COLLECTION is not — the CA database is read by certutil on the domain-joined Windows relay, which this repository does not yet drive, so the rows arrive over the ingest endpoint but nothing yet runs certutil to produce them. The rest of F4 — in-domain Kerberos/NTLM authentication (an Authenticator interface, not a Kerberos implementation) and fronting AD CS templates with trstctl ACME/EST — remains unbuilt.

  • PQC readiness reports are signed and offline-verifiable (M1, PARTIAL): a cohort's handshake evidence — every targeted client's outcome WITH its handshake size and latency — is turned into a verdict by Assess and exported as a readiness report signed inside the isolated signer under its own pqc-readiness-report key, verifiable offline from the report alone. The signer key refuses every other artifact kind, so being in the shared signer chain grants it no authority over XREC or VDEC artifacts; the signature covers the whole body, so an edited verdict cannot pass verification, and the report carries the FULL evidence so a reader re-derives the verdict rather than trusting the sentence. Cost travels in the signed body on purpose — a cohort that negotiated but tripled handshake size is a different recommendation, and a report omitting cost would recommend an outage. Scope, stated exactly: this is the EXPORT and the verdict rule, not the lab. Parallel classical/PQ pilot hierarchies are served (A26/A27): a root or intermediate ceremony whose signature_algorithm is ml-dsa-44, ml-dsa-65 or ml-dsa-87 mints that key in the isolated signer, the authority issues classical and pure ML-DSA leaves, and its CRL and OCSP responses carry its ML-DSA signature, all beside the classical hierarchy. Pilot waves with a canary halt are served (A28/A29): POST /api/v1/pqc/pilots records an ordered plan of endpoint targets bound to the exact leaf each must present, dispatches the canary wave as one relay endpoint-verification sweep through the durable outbox, classifies the relay's signed transcripts (negotiated, rejected, unreachable, with handshake size and latency), takes the compat-lab verdict over everything probed so far, and halts the instant a canary refuses or cannot be reached — later waves are never dispatched — or dispatches the next wave; a finished pilot's evidence is signed inside the isolated signer as the readiness report, verifiable offline. The console's Posture page carries the lab view (A30): start a pilot, follow each wave's outcomes with handshake latency and chain size, the verdict that halted or advanced it, the skipped waves, and the signed report. The acceptance's end-to-end pilot on the lab is not yet demonstrated.

  • Per-tenant silo isolation identifiers all key on the tenant ID (L4): a siloed tenant's Postgres schema, JetStream subject lane, and object-key prefix are each derived from the tenant's unique, immutable ID. The event LANE used to be the exception — it was derived from the operator-supplied slug, which has no charset validation and normalizes lossily, so acme-corp, acme_corp and acme.corp all collapsed to one lane and two distinct tenants with those slugs would have SHARED a JetStream stream — one customer's events landing in another's, a cross-tenant isolation breach in the sovereignty feature. The lane now carries the slug only as a readable prefix with the ID as the uniqueness suffix, and a structural assurance test proves distinct tenants stay disjoint on all three axes (schema, lane, object prefix — with no prefix a prefix of another's) even when their slugs collide. Scope, stated exactly: this hardens the derivation and adds the assurance. The per-tenant audit stream is served (A94): a siloed tenant's lane gets a JetStream stream of its own, TRSTCTL_AUDIT_<lane>, that sources exactly events.<lane>.> from the live events stream (it accepts no publishes of its own, so it cannot diverge from the log), with its own name, retention, and deletion; the silo provisioner ensures and drops it with the lane, and the isolation drill's event_audit_stream check proves on live JetStream that each probe tenant's stream received its own probe event and none of its neighbour's, then removes the probe streams (event_audit_stream_cleanup). Siloed placement itself is still exercised by the drill rather than by customer provisioning. Tenant-scoped break-glass with dual consent is served (A95): a customer sets PUT /api/v1/provider-access/policy (trstctl provider-access policy set) to require its own consent and how many distinct approvals it takes; the Provider's emergency requests appear under the customer at GET /api/v1/provider-access/requests (projected from the Provider's own provider.breakglass_request event recorded under that tenant); each customer subject with access:write approves or denies a grant once through POST /api/v1/provider-access/consents/{grant_id} (a denial needs a reason), recorded as immutable tenant.provider_access.* events in the customer's own history. After both Provider custodians approve, the Provider's POST /provider/v1/breakglass/{id}/results is refused with tenant_consent_pending until the customer's approvals are in and with tenant_denied after a customer denial; the Provider queue shows the customer's standing. The provider assurance readout is served (A96): GET /provider/v1/tenants/{id}/isolation returns, under the customer's read delegation, the isolation the deployment actually serves that customer (the model, PostgreSQL schema, JetStream lane and object prefix from the live tenancy router, and the customer's audit stream with its event count as JetStream reports it, or a note saying why there is none); the Provider console shows it in each customer's setup view beside health, and the isolation drill with its lane and audit-stream checks sits on the same page. Per-tenant HSM custody is served (A123): a tenant can migrate its key domain to a pkcs11 wrapper the isolated signer holds. The signer creates the tenant's domain KEK, keeps it wrapped by an AES-256 key the operator provisions on an HSM and reaches over PKCS#11, and wraps and unwraps the tenant's data keys itself; the control plane persists only the wrapped KEK and never holds the KEK. Each data-key wrap or unwrap is one signer call that opens the KEK from its wrapped form inside the signer and destroys it after. Seal, unseal, rotation (including onto another HSM key) and the history rewrap work as they do for a local wrapper, and a lost HSM key fails only that tenant closed (wrapper_unavailable), never falling back to the deployment KEK. It runs in the cgo signer (the signer-hsm image) under the BYOK license and is proven against SoftHSM in a container; proof against a hardware HSM needs hardware and has not been run. Stated limits: the data keys themselves return to the control plane, which decrypts tenant data with them; the PKCS#11 binding hands the decrypted KEK back through a C buffer it frees without zeroing (the signer is non-dumpable and refuses ptrace, and the Go copy is wiped); and every tenant-crypto callback for such a tenant makes signer calls, each an HSM decrypt, inside the signer's in-flight bound. What comes next: one non-extractable HSM key per tenant, so the tenant's KEK is itself an HSM object and data-key wraps run inside the HSM. The product ships an S3-compatible object-store client that keeps each tenant's objects under the object-key prefix the tenancy router resolves (tenant/<id>/, or silo/<id>/ for an isolated tenant), and the doctor's operator-run isolation suite proves that confinement against the deployment's configured store (OBJ-1 to OBJ-3 in docs/operations/doctor.md; OBJ-LOCK checks the bucket has Object Lock when the deployment writes under it). The silo provisioner itself is not wired into the served binary: its planes, the object-prefix one included, have only in-memory implementations used by tests, so moving a tenant between isolation models neither creates nor clears anything in the store, and objects keep the full name they were written under.

  • XREC reconciliation rounds are configurable (AUD-1, C4): a reconcile block in the config file supplies the schedules the rounds worker needs. Before this, roundSchedules was declared and never assigned, so the worker — registered, licensed, and visible in the runtime roster — hit its len(Schedules)==0 guard on the first tick and blocked for the life of the process. Zero anti-entropy rounds ever ran, no xrec.witness.recorded event was ever appended, and GET /api/v1/reconcile/agreement answered "0 open witnesses" forever. A licensed operator watching a healthy worker would reasonably conclude reconciliation was running while nothing was ever compared. A schedule naming FEWER THAN TWO authorities is dropped rather than scheduled: comparing an authority to itself produces no witness and would make the report claim collecting=true on a deployment that still compares nothing — the same illusion in a new place. An unparseable cadence or liveness takes a sane default rather than zero, because a zero cadence busy-loops the scheduler against a customer's authorities and a zero liveness marks every authority instantly stale. With C4's adapters, two authorities are DURABLE: trstctl-self reads the certificate inventory and trstctl-ca reads the internal CA issuance ledger — independently written state under RLS — both projected onto the shared assertion vocabulary (issuer+serial identity, active-vs-revoked standing, scoped to the internal-CA jurisdiction so a DigiCert certificate in inventory is not reported as drift the ledger never claimed to know about). A round whose digests disagree hands each differing pair to the runtime's witness emitter: witness built over the exact sets the digests committed to, signed in the isolated signer, recorded with both signed digests on the ledger event, quarantine admission updated, and the rounds worker now survives a failed round (logged, retried next tick) instead of dying on the first transient error. The claim-1 acceptance is demonstrated end-to-end on the production assembly in internal/reconcile/conformance/storebacked_e2e_test.go: seeded divergence, a real scheduled round, a witness naming exactly the differing subset, offline verification from the recorded event alone, and remediation authorized only after in-signer plan verification. Boundaries, stated exactly: a Vault or OpenBao authority is observed only when its schedule entry carries a source block (address, token file, KV/PKI/Transit mounts); it asserts KV metadata, PKI-issued certificates and Transit key metadata, never secret data, and pairs meaningfully only with an authority observing the same universe. A KMIP server is observed the same way when its entry carries a kmip source block: KMIP 1.4 over mutual TLS, reading object metadata and certificate values and never key values, proven against an in-process double and against a PyKMIP server (run with TRSTCTL_PYKMIP_PROOF=1); a pre-active KMIP object reduces to an unknown status. An EJBCA or step-ca CA joins with an ejbca or stepca source block, and an inventory source block makes an authority of the certificate inventory scoped to the same CAs. A round then compares a CA's issuance ledger with what the inventory holds. Both sides read only the CAs in issuer_certificates_file and only certificates unexpired at the observation, and assert issuer and serial (read from each certificate's DER) and whether it is revoked. EJBCA is read with its REST v2 certificate search. The search is frozen at the observation time, and its pages must add up to the count read first; otherwise the observation fails rather than report part of the ledger. step-ca's PostgreSQL tables x509_certs and revoked_x509_certs are read in one read-only, repeatable-read transaction; its file and MySQL backends are not read. Proven in containers against EJBCA 9.6.3 Community and step-ca 0.30.2 (run with TRSTCTL_XREC_CA_PROOF=1). A round per CA witnessed each kind of drift: a revocation only the CA knew, a certificate the CA issued that the inventory lacked, and one the inventory held that the CA never recorded. EJBCA issued that one while its certificate storage was off; for step-ca it was signed offline with the intermediate key. AD CS ledgers need the Windows image and an x86 machine, and commercial CAs need accounts; those come next. The cloud-kms reducer still has a nil source — scheduling it fails observation closed and logs, it does not fabricate an authority — and inventory rows with no captured DER cannot participate in the comparison (no issuer bytes to derive the shared identity from) until their bytes are recovered. POST /api/v1/certificates/{id}/der-recovery (CLI certificates recover-der, the certificate detail's Certificate bytes actions) looks for them in a mint of the tenant's own CAs that retained the certificate, found by the row's serial (ca_ledger), or reads the certificate the row's host:port location serves from the control plane under the usual egress rules (endpoint; a private address needs egress:private and its CIDR). Bytes are attached only when they hash to the row's fingerprint, only to a row that holds none, and with where they came from (certificate.der.recovered); the row then joins the comparison. An address only a relay reaches gets its bytes from a sweep with agent_channel.probe_scanned_certificates on, and fetching by serial from external CAs is what comes next.

  • CA-key retirement (H4): GET /api/v1/ca/keys/{id}/retirement and trstctl ca keys retirement list the dependents standing between a CA key and destruction. An operator with keys:write can submit the exact final dependency epoch through POST /api/v1/ca/keys/{id}/retirement, the explicit-confirmation lifecycle control, or trstctl ca keys retire <id> --force -f <request.json>. The wire request contains only final_epoch and confirm_irreversible: the server fixes the policy class to ca-signing-key and derives the sole approval identity from the authenticated principal, so a caller cannot name a weaker class or impersonate another approver. The request only appends a frozen evidence command and its same-transaction outbox intent. The outbox worker replays that frozen event prefix and calls GatedDestroy over the signer UDS; a frozen public event segment that exceeds the signer's 1 MiB unary receive limit uses 128 KiB streamed frames. The signer keeps that segment in an unlinked, local spool with a 512 MiB total cap, hashes its exact bytes into the durable operation identity, and folds only the target key's dependency state. The API process never receives a private key or a key-backend destruction handle. Incomplete evidence produces a signed refusal and leaves the key alive. Complete evidence destroys the signer-local handle first, then mints one full public record bound to the command ID, tenant, stable key ID, final epoch, required/completion/revocation digests, quorum and audit head. A signed refusal is fsynced under the exact frozen command hash before the signer replies, so a lost response or restart returns the same refusal bytes without another signature. Both outcomes are immutable events and rebuild the tenant-RLS projection after restart. The console displays signed refusals and downloads the complete destruction record for offline verification. VDEC attaches in every build and remains fail-closed on missing evidence. The checklist now derives its CA-key dependents from the original immutable managed-leaf, PKI-secret, child-CA, re-key and public cross-sign events. It does not depend on a later best-effort registration append after signing. The indexed tenant read model is rebuilt from those same events on restart; the signer's frozen event-set check uses the same derivation, including issuance after the frozen cut. A serial-only historical issuance remains a conservative serial: dependent rather than an invented certificate inventory row. For a host-held certificate, the effect-free POST /api/v1/ca/keys/{id}/reprotect/plans/preview and CLI preview derive the exact public plan without recording it. POST /api/v1/ca/keys/{id}/reprotect/plans and trstctl-cli ca keys reprotect plan recheck current state and record an immutable public plan only after checking the tenant's outstanding dependent, the exact certificate and its predecessor CA signature, one managed identity bound to that certificate and target, an active direct successor CA, and the enabled host target's current revision and assigned agent. Ambiguous identity bindings refuse. The target supplies the probe address and name; callers cannot redirect them in the plan request. GET /api/v1/ca/keys/{id}/reprotect/plans/{job_id} and the CLI plan readback recover that immutable public selection after a lost response. A plan binds the selected certificate-policy name. Preview and every later execution stage resolve its current active revision, using the configured default only when the identity has no selected or retained policy. Policy narrowing refuses before queue or signature; a legacy plan without a recorded policy name is recovered under the identity's current selection, not an unrelated default. The plan receipt proves that review was recorded, not that issuance, deployment or served-state cutover happened. The host worker dispatches the bound job to its assigned agent. The agent uses the installed subject key to make one encrypted, restart-recoverable CSR, pins the exact predecessor, installs the successor, probes the served leaf, and reports signed custody. Only that verified report can atomically supersede the old inventory certificate and change the identity's selected CA. Its source event carries the signed readback, and separate first-writer supersession and completion events account for the old CA dependent. This is an administrative continuation of the same deployed subject key and identifiers, not a new API enrollment. The identity-selected certificate profile still checks the CSR's names, key algorithm, EKU and requested lifetime. Its allowed_protocols list governs new enrollment and does not exclude this CA re-protection path; the profile audit labels the decision ca-reprotect. Ordinary API enrollment remains subject to allowed_protocols. The host agent cannot ask for a new name or key through this exception because the control plane rechecks the immutable plan and exact predecessor SPKI before signing. The CA hierarchy Lifecycle console now exposes the host-held certificate, direct successor, enabled target and lifetime selection. Its effect-free preview shows the server-derived agent, key, target revision and listener; it records the reviewed plan, independently reads it back, and can recover a lost response by job ID before queueing. The console never calls a queued job complete. The path is qualified on the evaluation stack against a live host target: a host-executed leaf under a predecessor intermediate was re-issued under its rotated successor by the enrolled agent with the same subject key, read back by a stock TLS client, accounted on the retirement checklist, and the predecessor key was then destroyed with a signed destruction record that the API, CLI and console all read back after a control-plane restart and a cold restart. A host-executed first issuance also resolves the identity's ownership readiness at the deploy call: an unattested owner is refused with 409 before any key, signature or host job exists, and a refusal that still happens after the host finished is recorded on the identity's delivery timeline as lifecycle_transition_refused with its cause. Retirement stays blocked until its completion evidence is present. An expired requester-held managed leaf with no managed deployment can be dispositioned through release_managed_ca_leaf after two distinct custodians approve its exact public certificate digest, immutable mint event and bounded external-use risk decision. The operator previews and starts the ceremony in the API or console, then calls POST /api/v1/ca/keys/{id}/dependencies/release or trstctl-cli ca keys dependent release. A durable first-writer fence survives restart and backup; its event and approval evidence rebuild the checklist. This deliberately does not claim that expiry proves the leaf was unused outside trstctl, and serial-only historical rows are not eligible. A revoked, unserved host-held leaf can use a separate bounded reason only when the exact local-issuer revocation fence is completed and the immutable revocation event matches its mint and public certificate. The same two-custodian, first-writer release flow records this risk decision. A local revocation record cannot prove that every external relying party rejected the leaf; a deployed or observed host leaf remains ineligible. If the tenant has no recorded dependency state for the requested key, the checklist returns HTTP 404 with an explanation to verify the key ID and collect dependency evidence. This does not establish whether the key itself exists; another tenant's dependency state is never disclosed. A recorded history whose dependents are all accounted for still returns HTTP 200 with its actual counts. A missing dependency report is not an empty outstanding list: an empty list reads as "this key has no dependents", which must never authorize an irreversible act on evidence nobody gathered. Served on the CA hierarchy console's lifecycle tab, where retirement belongs beside rotation. Its selector is populated only from served, signer-backed CA authorities already marked superseded or revoked; it sends that authority's exact ID, while the API repeats the same precondition check. A failed checklist read renders an explicit unavailable state and no destruction control, never a reassuring zero. A failed vdec.retirement.destroy outbox row can be recovered through POST /api/v1/ca/keys/{id}/retirement/retry, trstctl-cli ca keys retirement retry, or the lifecycle console. The checklist includes the frozen command ID, delivery state and attempt count. The operator must bind a reason and explicit irreversible-action confirmation to the observed command ID and attempt count. The event grants exactly one more outbox attempt on the original row; it cannot mint a replacement command or change the frozen dependency evidence. A repeated idempotency key recovers the same grant after a lost response. The isolated signer returns its durable signed result for the same command if it completed before the response was lost. A changed dependent, command, payload, actor, tenant, or attempt count refuses the grant; a cold replay cannot refund an already consumed attempt.

  • Incident fleet-reissue batch gates (H3): each batch's health gate is recomputed on read from THAT BATCH's own replacement identities, through the same verification summary D6's canary uses. It previously round-robined the run's gate list across batches (gates[(index-1)%len(gates)]), so a batch's gate described some other batch — worse than showing nothing, because during an incident it reads as per-batch evidence and the operator deciding whether to continue is exactly who would act on it. The verdict rules are unchanged and deliberately refuse to round up: any failed replacement fails the gate, and a single UNVERIFIED replacement keeps it not_evaluated rather than passed — a run that is 99% verified is a run with an endpoint nobody looked at. Only receipts paired with an accepted signed agent-job receipt count. Batches that are planned, queued, or halted keep not_evaluated, because a batch that never ran has nothing to verify and a halted one changed nothing; a read failure leaves the gate as it stands rather than inventing a verdict in either direction.

  • Migration waves (H2): internal/migration is the generic ordered-cohort engine behind served CA rollover. POST /api/v1/migrations/assess remains a separate, read-only operation and reports UNKNOWNS separately from findings: a member with no observed trust store is not a member confirmed to have an empty one. A reviewed manifest then names one active signer-backed CA authority, ordered waves, exact deployed identities, exact enrolled host agents, and exact public trust-anchor paths. The server derives the public anchor from that authority; it never accepts an operator-supplied key or silently changes authorities after review.

    Execution is event-sourced and restart-safe. Each wave runs distribute public trust → signed readback → issue from the manifest authority against a host-generated CSR → deploy → signed live-listener handshake → advance. Both observations are gates, not progress labels: one missing receipt licenses no next action, one failed receipt stops forward publication, and later waves never enter the outbox. The failure automatically begins the newest-first inverse, including every member effect already published for the failed gate; signed receipts that arrive late are retained without restarting forward motion and their serialized inverse still runs. Pause stops new work from being published and recalls the work this run has already published: a job not yet claimed stops being claimable, a job an agent holds refuses its next lease extension and its result (the agent stops before it installs), and a signed result that still arrives is retained as a late receipt rather than applied. Resume lifts the recall and drops the cut lease, so the exact required agent claims the job afresh and the idempotent intent converges; resume continues the same gate. Rollback restores and reverifies predecessor leaves newest wave first, projects the predecessor active and successor superseded in the same event transaction, then removes successor trust. A rollback failure halts on a durable attempt cursor and a manual retry gets fresh idempotent outbox identities. The exact-agent jobs, immutable aggregate, outbox effects, receipt evidence, and rollback cursor all survive PostgreSQL projection loss and event replay.

    Served through /api/v1/migrations/runs plus /{id}/pause, /{id}/resume, and /{id}/rollback, the matching trstctl migrations commands, SDKs, and the /migration console. Scope: executable membership is deliberately narrower than assessment today — one active predecessor per identity that carries any mix of DNS names, IP addresses, rfc822Names and URIs, an enabled host-agent connector with verify_address, and an active host-role agent. The member binding freezes each kind of identifier on its own half (subject_dns_names, subject_ip_addresses, subject_email_addresses, subject_uris) exactly as the predecessor spelled it; the host-generated successor CSR asserts exactly those identifiers and the control plane refuses any other name, address, email or URI by name, so a client that connects by IP, matches an rfc822Name or pins a SPIFFE URI keeps working after the migration. The same binding shape serves host-generated renewal and CA re-protection, so such leaves renew and re-protect on the host path too. The predecessor is the leaf the identity's last completed deploy, verified host renewal or rollback receipt names on its target, whichever authority issued it: a leaf a public or partner CA issued through the external registry migrates onto the new private authority exactly like a locally issued one, the member binding records its inventory source and the issuing external CA id (predecessor_source, predecessor_external_ca_id), and the successor supersedes it in inventory. An identity with no such receipt falls back to its one active internally issued leaf; a receipt naming a leaf the inventory does not hold, or one no longer active, is refused at start by name. An incident run still revokes through a served internal authority, so its cohorts are the compromised internal CA's leaves, never externally issued ones. A network appliance (an f5, kemp, netscaler, a10, fortigate or paloalto target the connector catalog marks executes_rollback) joins a run as an appliance member: the operator names the network relay the target is assigned to (required_agent_id), and the member binding records execution_model: appliance. An appliance has no trust stage of its own — a relay cannot write a trust store on an appliance and the appliance has none this control plane manages — so its trust verdict reads not_applicable rather than verified, it takes no trust_anchor_path, and the hosts whose clients must trust the new anchor are members of their own; a cohort of only appliances opens at the leaf gate. The control plane then issues the successor under the new authority with the predecessor's exact identifiers, records it as the predecessor's successor, and queues the sealed deploy for that relay alone; the relay installs it and its signed post-deploy handshake is the live gate. Rollback re-binds the predecessor object through the same relay (the A46 re-bind families) and removes no anchor. An appliance family without an executable re-bind, a host agent named for an appliance, or a target not yet assigned to the relay is refused at start by name. Built and proven against appliance doubles; proof on a real appliance needs the operator's device. Cloud-store migrations are refused at start rather than represented as executable. An imported offline root may be named as the new authority: its key never enters the signer, so successors are issued by the root's one active signer-backed intermediate (imported through the offline-intermediate ceremony) while the root certificate is the anchor hosts receive; the member binding records that intermediate as the issuing authority. An offline root with no such intermediate is refused at start with the ceremony to run, and one with several is refused until the operator names the issuing intermediate itself, which then makes that intermediate the distributed anchor. A host-generated successor that fails its live gate keeps its sealed key on the host for the same job to resume; rolling that successor back now frees the slot, so the next host-generated job for the target generates its own key. Members whose certificate profile requires a distinct issuance approval (or a deployment with require_approval) no longer refuse the run: the start is the approved operation. The first POST /api/v1/migrations/runs records an immutable approval request (resource_kind: migration_run, the plan id, the requester, the canonical request digest, the new authority and each gated member's frozen profile revision and lifetime) and answers 202 with status: pending_approval, the request id and intent digest; reviewers with certs:issue approve it through /api/v1/approval-requests, and the same start posted again under a new Idempotency-Key records the run with the consumed approval on it (approval in the run). Incident fleet reissuance still refuses such members; it has no start approval of its own yet.

  • Trust in the graph (H1): trust-store anchors agents collect are promoted from flat discovery findings into relationships — a trust-store node kind, TRUSTS (store → issuer, oriented the way impact travels) and HOSTS (resource → store). GET /api/v1/graph/trust-stores/{id}, trstctl graph trust-stores, and the Graph console's issuer detail answer "trusted by N stores across M hosts"; hosts are counted distinctly, because one machine running both an OS store and a JVM cacerts is two stores and one machine to visit. A store is its own node rather than a host attribute on purpose: a machine routinely carries several with DIFFERENT contents, so "is this CA trusted on host X" has no single answer. Scope, stated exactly: an authoritative store → issuer edge requires either the same certificate SHA-256 fingerprint or the same SPKI SHA-256 public-key identity. The SPKI path deliberately recognizes a cross-signed copy of the same CA key. A subject-name-only match is exposed separately as an unverified candidate and is excluded from authoritative store/host counts, blast-radius traversal, and automation; two roots can share a subject while holding different keys. Stores nobody has scanned do not appear, which is the honest answer rather than a reassuring one; a CA showing zero trusting stores means none have been observed, not that none exist.

  • Third-party secret scanning: CI/CD log, container-registry, Slack, and Jira artifact scanning is served through /api/v1/secrets/scans/third-party/{provider}/ingest and the matching CLI. The contract intentionally accepts an operator-owned artifact_path; raw logs, registry metadata, chat exports, and issue exports stay outside trstctl storage — discovery stores only redacted rule/file/line/provider metadata. The console queues an artifact, reopens the exact discovery run by ID after a restart, shows its terminal state and redacted findings, and links to the full filtered Discovery triage. This is an operator workflow over the local export; it does not collect native provider events. Provider signature verification is served: an ingest may carry signature_path (the provider's detached RSA or ECDSA signature over the artifact bytes, raw or base64) and signer_public_key_pem; the signature is checked before the scanner opens the artifact, a failing signature fails the run without scanning, require_signature refuses unsigned artifacts at ingest, and every finding records signature_verified and signer_key_sha256. Artifact retention is automated at both ends: delete_artifact_after_scan removes the operator's export file once the scan is terminal (one regular file; directory exports stay), and the privacy retention worker deletes redacted third-party findings older than TRSTCTL_PRIVACY_RETENTION_THIRD_PARTY_SCANS (default 90 days) while the run record stays. Native provider API polling and provider-native annotations remain architecture shortfalls.

  • React console scale work: cursor-aware inventory pages consume next_cursor and accumulate additional pages on explicit operator action. Every DataGrid-backed large table switches to a bounded, overscanned DOM window after 100 loaded rows; the certificate inventory has a focused multi-page acceptance test proving the second cursor is sent and that scrolling replaces, rather than appends, rendered row nodes.

The React web console: served by the binary

The React 18 + Vite + shadcn/ui single-page app (F12) is the real embedded artifact the running binary serves:

  • The shipped binary serves the real console. The release pipeline builds the SPA into the binary's embedded asset bundle; the built bundle is committed, so even a plain control-plane build serves the real console at / — hashed /assets/index-*.{js,css} and an index.html that references them, not the old "not built" placeholder. A test boots the served handler over the real embedded bundle and fails if it ever regresses to the placeholder, and a release gate (TRSTCTL_REQUIRE_BUILT_UI=1) blocks a release that would embed the placeholder.
  • Generated frontend↔backend contract. The frontend's API types are generated from the served OpenAPI contract, not hand-duplicated: the generator emits TypeScript types from the served API spec, and the API client re-exports those types so a backend field add/rename/remove that isn't regenerated fails the TypeScript build. A CI regenerate-and-diff gate fails the build on drift, so a frontend/backend status mismatch can no longer recur silently.
  • Operational console routes. First-class routes, nav entries, typed API wrappers, and route-test coverage cover the GA operator slice: Profiles (/profiles), Graph (/graph, inventory + blast-radius query), Audit (/audit, event list + evidence export in JWS/NDJSON/CSV/Splunk-HEC/Sentinel), dual-control approvals from the identity table, core incident execution (/incidents — replacement issue/deploy, fleet reissuance, revocation queue, connector receipt, rollback evidence, remediation playbooks, response dispatch, sealed audit bundle), and Product help (/assistant) with grounded questions, cause investigation, and on-demand read-only MCP tools. The bounded break-glass reconciliation workflow has its own panel on the Incidents page. Deliberately API-only surfaces stay labeled until they get their own UI — a UI boundary, not a claim that the underlying API is library-only.
  • Console UX hardening. A destructive-transition confirmation (revoke/retire require an explicit, credential-named confirm dialog) and 429/Retry-After handling (a concrete "retry in Ns" hint) are served and tested. Cursor-based pagination and bounded list virtualization are also served: the client carries next_cursor, accumulates rows, and the shared DataGrid renders only the visible window plus overscan for large result sets.

Interactive OIDC, SAML, and LDAP / Active Directory browser login & sessions: served by the binary

The OIDC authorization-code login, SAML 2.0 Service Provider login, and LDAP / Active Directory bind login + sessions are served by the running binary (behind auth.oidc.enabled, auth.saml.enabled, and auth.ldap.enabled). OIDC remains in the Free core; tenant SAML and LDAP require Enterprise SSO, included in Enterprise and inherited by Provider. An enabled tenant SAML, LDAP or SCIM configuration without that implementation refuses startup with SAML, LDAP and SCIM login require an Enterprise licence. Core-only builds cannot attach it. The separate Provider operator authentication plane is unchanged. OIDC mounts /auth/login and /auth/callback; SAML mounts /auth/saml/login, /auth/saml/acs, and /auth/saml/metadata; LDAP mounts POST /auth/ldap/login; all three share /auth/me and /auth/logout. OIDC verifies the id_token's signature, issuer, audience, nonce, and temporal claims (exp/nbf/iat). SAML verifies signed POST-binding assertions against configured IdP metadata through the same isolated cryptography boundary. LDAP binds the user, then performs a configured group search; production directories should use ldaps:// — plaintext ldap:// is accepted only for loopback development fixtures. All paths set an HttpOnly + SameSite=Strict session cookie (marked Secure whenever the control plane serves TLS) plus a double-submit CSRF token, authorizing API calls under the same RBAC and per-tenant database-isolation scoping as an API token; mutations on the cookie path require the CSRF header. When browser sign-on is disabled the binary authenticates with scoped API tokens only, exactly as before; an enabled-but-misconfigured OIDC, SAML, or LDAP block fails closed at startup.

  • Per-user → tenant mapping is served. Each authenticated user is mapped to its real tenant at session issue — by a configurable OIDC claim or SAML attribute (auth.oidc.tenant_claim / auth.saml.tenant_claim), by an IdP or LDAP group → tenant table, or by an explicit subject/claim/group → tenant mapping (auth.*.tenant_mappings) — instead of collapsing every browser user to one tenant. A user that maps to no tenant is rejected (the login fails closed, never minting a session in a fallback tenant unless an operator explicitly opts into allow_default_tenant). Per-tenant database isolation then confines each session to its mapped tenant, so two SSO users in different tenants see only their own data via the served API. The legacy single default tenant is retained only as that opt-in fallback. This is the served half of the defense against cross-tenant leakage; a freshly logged-in user still cannot self-issue (issuance stays behind the served RA/policy gate and the requester scope excludes certs:issue).

SCIM 2.0 provisioning: served by the binary

The SCIM 2.0 provisioning surface requires Enterprise SSO and is served by the running binary behind auth.scim.enabled. It mounts GET /scim/v2/ServiceProviderConfig, /scim/v2/Users, and /scim/v2/Groups; bearer tokens are loaded from configured token files, hashed, and bound to one tenant before a request body is trusted. SCIM user create/update/PATCH writes the same tenant-member event path used by RBAC. SCIM active:false, DELETE, or group removal changes the projected tenant-member roles, so the next browser-session API request sees the new authorization result.

SCIM Bulk (RFC 7644 §3.7) is served at POST /scim/v2/Bulk (A72) and advertised in ServiceProviderConfig (maxOperations 100, maxPayloadSize 1 MiB): the operations run in order through the same per-operation logic as the single routes, a POST's bulkId names its created user or group for the operations after it (in a path or in any string value of the data), each operation reports its own status and body, failOnErrors stops the request after that many failures, and the whole request is one idempotent mutation — a replay under the same Idempotency-Key returns the recorded response. Current SCIM limits are deliberate and fail closed: password management and password-change flows are not implemented; SCIM groups never create roles. A group grants a role only when its name is a configured RBAC role (such as admin, operator, viewer, auditor, or ra-officer) or the token's group_roles maps it to a configured RBAC role or a custom role the tenant defined; any other group is refused and grants nothing. Inbound token rotation is rewriting the token file: the new hash loads within two seconds without a restart, and a rewrite that cannot be used keeps the last good token. The writeback bearer is still read only at startup.

Directory writeback (A73) is served when a token also names the directory's own SCIM 2.0 endpoint (auth.scim.tokens[].writeback_endpoint and writeback_token_file; the bearer stays in locked memory). A change an operator makes in trstctl to a member that token provisioned — a role grant or removal through PUT /api/v1/access/members/{subject}, or an offboard through POST /api/v1/access/members/{subject}/offboard — queues a scim.writeback outbox row in the same transaction as the member event. The worker finds the directory's User by the same attribute the token binds the login subject to (userName, or externalId when mapped), PATCHes active and the trstctl roles as the User's roles, reads the User back, and records scim.writeback.recorded with the verdict; a directory answer that disagrees with what was written is recorded as refused, a transport failure retries through the outbox, and the directory's own words never enter the event log. A change the directory itself pushed is never echoed back, and a member with no SCIM identity is never written. The directory must serve SCIM 2.0 for its own users (an identity broker, a SCIM-fronted directory, or an IdP whose SCIM API accepts writes); an IdP that only pushes outbound SCIM cannot be written to, and without a writeback endpoint local changes to directory-owned members still stand until the directory's next push, as before.

Unified NHI inventory

Unified NHI inventory (CAP-NHI-02) is served as metadata, not credential material. GET /api/v1/nhi/inventory requires nhi:read and merges tenant-scoped identities, certificate inventory, API-token metadata, enrolled agents, and discovery findings into one normalized inventory across certificates, SSH keys, secrets, API keys, OAuth apps, tokens/PATs, service accounts, IAM roles, webhooks, workload IDs, and agents. It does not return secret values, private keys, raw API tokens, client secrets, or other credential bytes; use the specific governed secret/issuance endpoints for those flows.

Malicious / abused OAuth-grant detection (CAP-ITDR-03) is served from metadata exports. oauth_grant Discovery sources emit normal oauth_grant inventory findings and, when the export contains concrete abuse evidence, additional oauth_grant_abuse findings tagged CAP-ITDR-03. The detector stores provider threat signals, reason codes, evidence refs, and source event ids only; OAuth client secrets, access tokens, and refresh tokens are rejected. Live IdP/SaaS grant revocation and provider-side enforcement remain connector/remediation work rather than being hidden inside discovery.

AI, RCA, and MCP surface

The AI surface — model adapter (F76), grounded RCA / NL query (F75/F77), and the guarded MCP tool server (F78) — is served, mounted under /api/v1/ai/* and /api/v1/mcp/* (off by default — ai.enable_api — and fail-closed when off, so an upgrade does not silently expose it):

  • POST /api/v1/ai/query reads typed, tenant-scoped evidence from owners, certificates, the graph, CBOM, and the event log (F75). A question planner reads certificate questions that combine expiry windows, expired, status, issuer, key algorithm, subject or SAN name, owner and no-owner clauses (as a list or an exact count), “who owns” a certificate, crypto-asset questions by algorithm, key size, quantum vulnerability or policy, and credential-graph inventories. Each becomes a typed query whose values are bound parameters, and the answer states its interpretation with dated citations. The planner is a fixed grammar, not a language model: a question outside it, such as one about event history or causes, returns scoped evidence marked insufficient, not a claimed complete answer. A bounded result also declares when more matches may exist.
  • POST /api/v1/ai/rca answers a grounded root-cause / NL question from cited real records gathered through the tenant-then-RBAC scoping seam, preferring "insufficient evidence" to a guess (F77). A cause question needs a cited failure reason; event types alone are marked insufficient. The allowlisted renewal-failure reason is redacted before it enters the answer.
  • GET /api/v1/mcp/tools + POST /api/v1/mcp/tools/{tool} expose the tenant-scoped MCP tools an external AI agent can list and invoke (F78). Investigation tools are read-only by default; guarded write tools (issue_certificate, rotate_certificate) appear only when TRSTCTL_AI_MCP_WRITE_TOOLS=true, each still requiring certs:issue, an Idempotency-Key, and emitting mcp.tool.write. They act on a lifecycle identity through the identity transition route, so the policy gate and dual control apply: under dual control an issuance is proposed (mcp.tool.proposed) and runs only when repeated after approval, and a rotation is refused. MCP has no route to direct CA issuance or to an approval decision. Evidence tools (posture_findings, weak_key_exposure, distrust_exposure, assurance_record, revocation_status, crypto_inventory) read the served routes with the caller's token, and every tool result carries findings under the published result schema. Exposure covers what discovery and the inventory have seen: hosts and trust stores no agent has scanned are not covered, and a certificate known only by its issuer name is matched by name (confidence inferred). A result lists at most 200 findings. The rated certificate inventory behind weak_key_exposure and distrust_exposure reads at most 500 certificates and 500 discovery findings; the rogue-certificate posture reports read_limited when it reached that, and the tool answer then says it may be incomplete.

Every route is auth-gated (API token or session, graph:read), tenant-scoped (the tenant is the authenticated principal's, never a request field), rate limited, and injection-inert (a hostile string in a record is inert, cited data, and cannot by itself trigger a write). The AI model is air-gapped / opt-in by default — no model is configured, so grounding and citations work and nothing phones home; when an operator opts into a cloud/local model, every prompt crosses a redactor plus a residual-entropy refuse-gate before any egress, so no key/secret material leaves to a model (secret bytes live only in wipeable, zeroed memory and never reach a prompt). Proven end-to-end by acceptance tests (grounded NL-query/RCA citing real records, cross-tenant denial, injection-inert + secret-redacted, MCP list+invoke).

Secrets and identity frameworks

Transit keys are sealed at rest and shared by every replica. Each tenant's transit (encryption-as-a-service) keyring is one sealed row in PostgreSQL: AEAD, HMAC, and signing key material is sealed under the tenant's key domain (the deployment KEK for a tenant that has not migrated), and re-locked into protected memory on load (AN-8). Moving the tenant into its own key domain, or rotating that domain, re-wraps the row with the tenant's other secrets; only its data key changes. A create or rotate is saved before it is reported, over the revision the replica last read, so two replicas changing one tenant's keys at once cannot overwrite each other: the replica that loses the save catches up and applies its change again. A failed save fails the mutation rather than handing back a key that disappears later: the unsaved key (or new version) is rolled back out of the in-memory ring and its material wiped, so nothing can encrypt under it and a retry succeeds. A replica loads a tenant's row the first time the tenant is used and reloads it when an operation names a key or version it does not hold, so a key created or rotated on one replica works on every other.

Three properties bound what this gives you:

  • Persistence requires key protection. With no deployment KEK the keyring stays in memory only and keys do not survive a restart. Key material is sealed at rest or it is not written at all; there is no plaintext fallback.
  • transit.keyring_dir is now an import source. A keyring an earlier version checkpointed there is copied into the shared keyring at startup for each tenant that has no row yet; the file is left as the operator's copy, and a file that cannot be opened still refuses the start.
  • A keyring that cannot be opened fails the operation. A row the tenant's key domain cannot open (a sealed tenant, a lost KEK) fails each transit operation for that tenant instead of minting fresh keys over it. The key bytes are deliberately not event-sourced, so the table restores from the PostgreSQL backup.
  • Create and rotate are event-sourced facts (A126). Each is a transit.key.created or transit.key.rotated event (schema 2: key, kind, version) under a fact ID saved with that version's sealed material, appended only after the material is saved, and projected into transit_key_facts, which rebuilds from the log. GET /api/v1/transit/keys/{name}/versions reports each version's fact_id and whether the log records it. A fact whose append fails leaves the key usable and is recorded by the leader's reconcile pass (at start and every ten minutes), which also gives fact IDs to versions saved before facts existed; until then the version reads recorded: false. Facts recorded before this change are schema-1 audit records that name only the key and project nothing.

Private-egress allowlist entries must be exact network prefixes. Every allow_private_cidrs / private_egress_cidrs surface (connectors, Rekor code-signing, incident notifications, managed keys, secret integrations, external CAs, ServiceNow bindings) now refuses an entry whose host bits are set (for example 10.1.2.3/8) or that covers every address (0.0.0.0/0, ::/0) at configuration load, naming the offending value. Such entries used to pass validation on most surfaces and were then silently ignored at dial time, so private egress failed as SSRF-blocked with nothing explaining why. Migration: operators whose entries currently carry host bits should write the network address the entry already effectively meant (for 10.1.2.3/8, write 10.0.0.0/8 — or, if a single host was intended, 10.1.2.3/32). The dial-time guard remains as defense in depth and now logs and counts any entry it skips.

CMP binds every CSR to the authenticated protection identity by default. A PKIMessage's protection identity must chain to the operator-configured anchors (protocols.cmp_client_trust_anchor_file), and — new — the CSR's subject and SANs must be asserted by that identity, so a device credential can renew only itself and a stolen credential stays one device in blast radius. RFC 4210 registration-authority enrollment on behalf of third parties remains available as an explicit opt-in: protocols.cmp_allow_ra_enrollment in the config file or TRSTCTL_PROTOCOLS_CMP_ALLOW_RA_ENROLLMENT=true in the environment (echoed by -check-config as protocols.cmp.allow_ra_enrollment when CMP is enabled). Cross-identity requests under the default are refused with a distinct audit reason (csr not bound to protection identity), so an operator can tell an impersonation attempt from a malformed message. Deployments that relied on the previous unbound behavior must set the RA opt-in deliberately on upgrade.

Six of six secrets/identity frameworks are mounted on the running binary under /api/v1/secrets/* (off by default — secrets.enable_api — fail-closed when off, requiring a KEK when on):

  • Auth-method framework (F58) backs POST /api/v1/secrets/login — a machine presents a token, Kubernetes SAT, AWS IAM signed GetCallerIdentity request, GCP identity JWT, Azure workload JWT, generic OIDC token, or generic JWT and receives a scoped, tenant-scoped session (distinct from the human OIDC SSO bridge). Token credentials MAC-bind tenant, audience, principal, and expiry; JWT methods require a tenant claim or tenant-pinned config; AWS IAM is tenant-pinned through allowed account/ARN config. X-Tenant-ID is only a lookup hint — mismatched tenant headers are rejected.
  • Application secrets SDK (F64) backs the secret store POST/GET/PUT/DELETE /api/v1/secrets/store/... (create, read, rotate, delete), GET /api/v1/secrets/store/{name}?resolve=true (${secret.path} reference expansion with cycle rejection), GET /api/v1/secrets/store/history/{name}?version=N (read one prior sealed version), and POST /api/v1/secrets/store/recover/{name} (point-in-time recovery). Values are sealed at rest under the KEK. trstctl-cli run --secret ENV=path -- <cmd> wraps the same read path, injecting values only into the child process environment. Bulk import uses one event, per-name reservations, and an atomic sealed-row/receipt projection for 1–100 new names. It is served through API, CLI and console; existing or reserved names reject the whole batch.
  • The Vault/OpenBao compatibility shim backs the common migration paths GET /v1/auth/token/lookup-self, KV mount-discovery preflight for secret/, POST/PUT/GET /v1/secret/data/{path}, and POST/PUT /v1/pki/sign/{role} for requester-generated CSRs, and the deprecated POST/PUT /v1/pki/issue/{role} server-keygen path for stock vault CLI token lookup, KV v2 put/get, mount and ACL policy lifecycle, and PKI issue — deliberately a subset over the native secret store and dynamic PKI secret. Tenant-authored mount management, ACL policies, and common transit paths are also served, as are response wrapping (/v1/sys/wrapping/wrap|unwrap|lookup over the native one-time share rows, single-use trstw_ tokens with a bounded TTL) and the per-token cubbyhole (/v1/cubbyhole/{path}, sealed in the tenant key domain, keyed by the presented token, expiring with it). The native dynamic leases are served at Vault's engine paths: database/creds/<role> (username and password from a PostgreSQL, MySQL, MongoDB or Redis provider), aws/creds/<role> and aws/sts/<role> (an STS session), kubernetes/creds/<role> (a ServiceAccount token), or <provider>/creds/<role> when several providers offer the role, with sys/leases/lookup, renew and revoke. Each read opens a native lease (an explicit Idempotency-Key replays it). Database leases renew up to the provider's maximum TTL; AWS STS and Kubernetes leases are not renewable on these paths because their native renewal mints a replacement credential. A synchronous revoke returns once the provider confirms removal; a Kubernetes API server can still accept a token it authenticated recently for its authentication cache TTL (10 seconds by default) after its ServiceAccount is gone. AWS cannot end one STS session early, so revoking an STS lease closes the lease and reports the session valid until its native expiry. Vault's engine configuration paths (database/config, database/roles, aws/roles, kubernetes/roles), static database credentials, revoke-prefix and lease listing are not served: providers stay operator-configured. The stock Vault CLI's KV put and PKI issue/sign commands derive a token-keyed idempotency key when no header is sent. Mount and ACL mutations also derive a token-keyed key with the exact mount or policy event generation, allowing stock CLI retries and later delete/recreate cycles. KV v2 metadata (GET /v1/secret/metadata/{path}: retained versions and write times), folder listing (LIST or GET ?list=true on /v1/secret/metadata/{folder}: direct children, sub-folders suffixed /, 404 for an empty folder), object deletion (DELETE on data or metadata, the native delete with no soft-delete state), POST /v1/sys/capabilities-self (from the token's permissions and ACL policies) and POST /v1/auth/token/renew-self (refused: API tokens are not renewable leases) are served. A re-enabled tenant KV mount starts an empty namespace; older sealed values remain under native retention and mount disable is not a physical-erasure receipt. Transit mutations require an explicit Idempotency-Key.
  • Dynamic secrets (F65) are served when tenant providers are configured: POST /api/v1/secrets/leases, GET /api/v1/secrets/leases/{lease_id}, POST /api/v1/secrets/leases/{lease_id}/renew, and POST /api/v1/secrets/leases/{lease_id}/revoke — issue returns the backend credential once, later reads return metadata only, renew extends an active lease except for immutable Kubernetes TokenRequest and AWS STS credentials (renewal issues and reveals a replacement lease, then closes the predecessor), and revoke closes it; the leaseworker expires leases through an outbox-backed backend revocation queue. buildRunDeps constructs a tenant-bound registry for postgresql, mysql, mongodb, aws-sts (aws-iam alias), gcp-iam, azure-entra, kubernetes, and redis. Issuance is outbox-only: the pending event and sealed command commit first, provider retries reuse one stable lease identity where the provider has a native replay/lookup path; AWS STS fences an indeterminate call instead. Only the authorized issue response opens the sealed credential. Local acceptance proof logs in with each generated PostgreSQL, MySQL, MongoDB, Redis, and Kubernetes credential, rotates it, revokes both copies, and verifies rejection afterward. AWS STS is a contract-tested adapter, not a completed cloud journey: no AWS account is available in this local-only run for stock-client authentication, CloudTrail reconciliation, or rejection after native expiry. The local query double does not qualify AWS IAM/STS behavior. aws-sts and its aws-iam alias issue expiring AssumeRole sessions; they no longer create permanent IAM users or keys. STS cannot delete one session early. A trstctl revoke closes its lease, but AWS may accept the session until native_expires_at; the status remains pending until then. A lost AssumeRole reply leaves an indeterminate, fenced operation rather than risking a second session. Existing aws-iam startup attachments must migrate their policy ARN role bindings to full assumable IAM role ARNs and set an explicit 15-minute to 12-hour max_ttl. New Kubernetes leases record the UIDs of the ServiceAccount, bound Secret, and RoleBinding and delete with UID preconditions. Leases issued by older builds have name-only provider references; revoking one now inspects every same-name object first, leaves any object that carries the lease-owner label to the newer lease that reused the name, and deletes the unlabelled legacy objects under the UIDs seen at inspection, so generated names can be reused without draining older leases first.
  • Honeytokens include the native trst_ bait (F39 extension) and an operator-attached AWS IAM bait-key adapter (F80). The AWS path is a local IAM/ CloudTrail contract test until a real account runs it: the pinned LocalStack profile does not emulate IAM, CloudTrail delivery, account quotas, or cross-account event visibility faithfully. No owner AWS account was available in this local-only campaign. AWS-native use detection, live-account IAM deletion, and a stock AWS client rejection after deletion remain unverified. CloudTrail LookupEvents covers only management events in configured Regions for the configured account, with delayed delivery and a 90-day lookup limit. An outbox retry/dead-letter or an expired CloudTrail reader session is a monitoring outage; it must be investigated and requeued before asserting coverage. The AWS key is deliberately longer lived bait with explicit deny, unlike the short-lived STS credentials in ordinary dynamic-secret leases.
  • Secret rotation (F37) backs POST /api/v1/secrets/rotations for worker-queued connector:<target> secret-sync handoffs and for durable static-provider commands: a configured PostgreSQL, MySQL or AWS IAM dynamic-secret provider with allow_static_rotation runs stage, cutover, verify and retire as outbox-driven phases recorded by immutable events, with rollback to the sealed predecessor on a failed verification; GET /api/v1/secrets/rotations/{id} follows a command. Boundaries: the secret name must equal the provider role; verification is a connection test for databases and not_available for AWS IAM access keys; a rollback_failed or retire_pending command needs an operator to retire the leftover credential at the provider (no retry route yet); approval-gated tenants must approve the worker's cutover write. dynamic-lease:<provider> runs the same durable command on a configured PostgreSQL, MySQL or Redis provider: stage issues a new lease of ttl_seconds (required, at most the provider's max_ttl) through the durable lease lifecycle, cutover publishes its credential, verify opens a database session or authenticates to Redis and runs PING, and retire revokes the old_ref lease, completing only once the provider confirms the removal; old_ref is none or an active lease of that provider for the role the secret names. Rotating a MongoDB, cloud IAM or Kubernetes lease is what comes next (503 today); those leases issue, renew and revoke through the lease endpoints. A connector or static request that supplies ttl_seconds returns 400. New schedules accept connector:<target> and static providers; scheduled dynamic-lease rotation is what comes next (creation returns 503). Historical dynamic-lease schedules, and static schedules whose provider no longer allows static rotation, are marked unsupported, disabled, and make zero provider calls. Scheduled ticks bind each exact due edge to one durable, deterministic command before effects, execute at most 50 rotations while scanning at most 500 due rows, report every approval_pending, command_in_flight, command_claimed, or config_revision_unanchored deferral, and continue past row-local poison. A shared store/event/custody/integrity failure returns a cached 503 partial envelope; the same idempotency key replays it byte-for-byte without child execution, while a new key reconciles retained terminal evidence and continues. delivery_failed advances a connector schedule only after its canonical local version committed; generic failed never promotes new_ref. Verified terminal command rows are purged only after the schedule advanced, the exact event remains retained, and a newer command exists; the newest lineage fence and all claimed or ambiguous rows remain.
  • PKI-as-a-secret / dynamic certificate leasing (F67) backs effect-free POST /api/v1/secrets/pki/preview and mutating POST /api/v1/secrets/pki. Preview and execution share the same subject, SAN, public-key, profile, and overflow-safe TTL validator. Preview signs nothing, writes nothing, makes no external call, returns no credential material, and reports structured CA/custody/revocation/audit prerequisites plus exact effects, recovery, verification, CLI/Vault parity, and a server-keyed request fingerprint. Supplying that fingerprint on execution makes a changed plan fail closed. The recommended csr_pem mode signs a requester-generated PKCS#10 request and returns only the certificate. The mutually exclusive common_name mode still returns a usable certificate and private key for compatibility, but is deprecated and fails before key generation unless its issuance.server_side_keygen receipt is durable. Both modes use the issuing CA in the out-of-process signer. New issuance events project the exact public leaf, subject-key custody, and responder serial together, returning the inventory certificate_id for normal exact revocation. A historical version-2 PKI-secret event projected only its responder serial but retained the public leaf: after an upgrade, the first lifecycle sweep checks that leaf against the issuing CA's certificate and its serial against the CA ledger, then appends one ca.certificate.inventory_recovered event, which projects the inventory row a version-3 mint would have had, with subject-key custody recorded as unknown (it is never guessed). The leaf is then revoked through the normal inventory workflow. A leaf the issuing CA did not sign, or one already in inventory, is left as it is. Version-1 events retain only a serial: an operator who still holds such a leaf's public certificate submits it to POST /api/v1/certificates/pki-secret-recoveries (trstctl certificates inventory-pki-secret -f leaf.pem), which accepts it only when the issuing CA signed it and a PKI-secret mint recorded its serial, and then gives it the same recovered inventory row. A version-1 leaf whose certificate nobody kept remains serial evidence.
  • Secret sharing (F60) backs effect-free, value-free POST /api/v1/secrets/shares/preview, reviewed idempotent creation at POST /api/v1/secrets/shares, and atomic .../redeem. The server-keyed review fingerprint rejects a changed lifetime; an exact same-key retry returns the original one-time bearer rather than creating a duplicate. Redemption succeeds exactly once, and neither value nor bearer token enters the audit/event log.

Every served route is auth-gated (API token or session, secrets:read / secrets:write) and tenant-scoped. Every mutation is idempotent (deduplicated by Idempotency-Key) and recorded as immutable events; effect-free previews intentionally create neither. Secret values are held in wipeable, zeroed memory (never as a string), never logged, and never returned beyond their design. Proven end-to-end by acceptance tests.

Secret sync, Transit, and KMIP

Secret sync external stores (F68) — served when configured. The running binary mounts POST /api/v1/secrets/syncs and trstctl-cli secrets syncs run. A request reads one stored secret, writes a sealed tenant-scoped outbox row before any external write, records immutable sync intent, and returns metadata only. buildRunDeps constructs tenant-bound targets for AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, GitHub Actions, GitLab CI/CD variables, Vercel project environment variables, generic CI JSON endpoints, and Kubernetes Secrets, plus first-class Terraform Cloud/OpenTofu Variables API and Vault KV v2 targets (Terraform writes are workspace-scoped with category/sensitive metadata; Vault writes use the KV v2 data/metadata paths and preserve version/CAS semantics). Only the outbox dispatcher resolves the target credential and writes externally. The acceptance proof enforces exact target authentication, decrypts GitHub's X25519 sealed box in a separate receiver process, and independently reads every destination value back. GET /api/v1/secrets/syncs/targets shows the catalog and this installation's configured targets. A sync whose receiver result was lost is reconciled by reading the destination back where the target can (receiver-reconcile in the catalog), and a later version never waits forever behind it: a target without readable state holds the lane until an operator records what they saw through POST /api/v1/secrets/syncs/{id}/reconcile; GET /api/v1/secrets/syncs/{id} names that blocking predecessor.

AWS, GCP, and Azure targets independently opt into OIDC workload identity instead of their static target credential. The matching aws_workload_identity, gcp_workload_identity, or azure_workload_identity switch is false by default, and workload-identity mode rejects the provider's static credential fields rather than keeping a quiet fallback. The served /api/v1/secrets/syncs/workload-identity-sources API, CLI, and Secrets console bind a tenant JWT/JWKS trust source, exact audience/subject, provider target, and remote-key scope. Three thin, hand-written REST exchanges use one shared internal/cloudauth minter: AWS calls AssumeRoleWithWebIdentity; GCP calls the RFC 8693 STS endpoint and may then impersonate one service account; Azure calls the Entra v2 token endpoint with the configured application client ID and Key Vault scope. Azure's federated-credential binding accepts the already validated OIDC proof, which the form sends unchanged in its JWT-bearer client_assertion field. trstctl does not create another assertion, sign one with a certificate, or take custody of an Entra certificate/private key.

Proof resolution, signature and exact-claim validation, exchange, locked short-lived credential caching, and refresh happen only inside the bounded secret-sync outbox worker. Air-gapped mode records offline_disabled before either the token endpoint or target can receive a request, fails that delivery once, and does not retry forever. The resulting GCP or Azure bearer is passed to the existing hand-written internal/secretsync pusher; no vendor SDK or provider-specific cache is added. For Azure this replaces the azure-key-vault sync target's static token_ref. It does not replace TRSTCTL_MANAGED_KEYS_AZURE_BEARER_TOKEN(_FILE): those settings and internal/kms/azurekv belong to the separate managed-key child-signer path. TestServedAzureFederatedOutboxTenantIsolationTokenRedaction and TestServedAzureFederatedAirGapIsTerminalWithoutNetwork prove the served Azure exchange, target readback, tenant isolation, token redaction, and zero-network air-gap result.

Related read-only posture routes: GET /api/v1/secrets/cloud-secret-managers (CAP-SEC-04 — read-only cloud_secret discovery for AWS/GCP/Azure/Vault, plus sealed-outbox sync for AWS/GCP/Azure); GET /api/v1/secrets/kubernetes-operator (CAP-SECR-04 — the TrstctlSecretSync controller reconciles secret references into Kubernetes Secret.data and patches pod-template annotations for reload); GET /api/v1/secrets/workload-injection (CAP-SECR-05 — TrstctlSecretInjection patches Deployment/StatefulSet/DaemonSet pod templates with the shipped trstctl-agent --secret-inject sidecar, a memory-backed shared volume, app mounts, optional valueFrom.secretKeyRef entries, and status/content-hash annotations; the operator reads Secret metadata and key names but does not retain values; secret bytes never appear in API/status/audit/pod-template metadata); and GET /api/v1/secrets/unvaulted (CAP-SECR-07 — combines configured repository/third-party scan sources, redacted leaked_secret findings, AWS/GCP/Azure/Vault discovery visibility, and vault-augmentation sync targets). A webhook target delivers templated JSON signed with HMAC-SHA256 over a timestamp and the body, so a receiver can verify it and refuse a changed or replayed delivery; generic-ci-json stays the bearer-token-only push with no payload signature. aws-parameter-store writes SecureString parameters (proven against LocalStack, not a live AWS account), and netlify sets secret environment variables through Netlify's API, proven against a double built from Netlify's API reference; a live Netlify account proof is what comes next. Terraform/OpenTofu and Vault no longer depend on the generic JSON target. If a target is not configured, the route returns 503 rather than attempting an external call.

Transit/KMIP (F66) — served, with a bounded OASIS KMIP 1.4 profile. The running binary mounts /api/v1/transit/* and the trstctl-cli transit command group for tenant-scoped key create/rotate, complete metadata-only version history, effect-free sealed-restore/KMIP posture, encrypt/decrypt, rewrap, HMAC, sign, and verify. The Secrets console exposes the same workflow, filtered immutable receipts, and explicit no-auto-retry recovery. Transit keys never leave the process as exportable material, request plaintext uses wipeable []byte buffers, keyrings are zeroized on shutdown, and mutating operations emit immutable transit.* audit events. The binary also mounts an opt-in raw KMIP mTLS listener when protocols.kmip.enabled is true and protocols.kmip.tenant_id, protocols.kmip.cert_file, protocols.kmip.key_file, and protocols.kmip.client_ca_file are configured. That first served KMIP profile is intentionally bounded: it accepts verified client certificates, decodes TTLV with frame-size, field-count, and nesting-depth caps, serves Query and DiscoverVersions, AES-256 SymmetricKey Create/Register/Get (including AES-GCM wrapped Get/Register), plus Locate/Revoke/Destroy over the wire for stock clients, records kmip.object.created, kmip.object.revoke, and kmip.object.destroyed, and zeroizes in-memory key material on rekey/destroy/shutdown. It also serves RSA (2048/3072/4096) and ECDSA (P-256/P-384) key pairs: Create Key Pair (both halves Pre-Active and linked), Activate, single-part Sign and Signature Verify (PKCS#1 v1.5, PSS, ECDSA; SHA-256/384/512), Get of the public key, and Get Attributes, Get Attribute List and Add/Modify/Delete Attribute for names, contact information and x- text attributes. A generated private key is not extractable unless the client created the pair with Extractable set, so by default it never leaves the server; KMIP 1.4 would default Extractable to true, and trstctl deliberately does not. Not served: Register of asymmetric keys, certificates, Re-key Key Pair, Encrypt/Decrypt, digested or streamed signing, Sensitive objects, templates, and the ID Placeholder (each request names its object). PyKMIP 0.10 drives the key-pair path in an opt-in proof (TRSTCTL_PYKMIP_PROOF=1 go test ./internal/kmip -run PyKMIP -v). With protocols.kmip.client_registrations, one listener serves every tenant's self-service clients: a tenant registers a client (/api/v1/kmip/clients, trstctl-cli kmip clients, the Transit page) and gets a one-time enrollment, and the client's first KMIP request presents its own TLS certificate with that username and password, which binds the certificate to the tenant. The handshake proves the client holds the certificate's key and the secret proves its tenant issued it, so no tenant can claim another tenant's client by registering a certificate it has only seen; a removed registration is refused at its next request. A two-tenant proof drives this with the listener's own clients. Appliance-specific templates (a vendor's key naming and attribute conventions) are not shipped: each needs the appliance itself to prove it. Production Transit durability remains conditional on the deployment KEK; the KMIP listener additionally remains conditional on its licensed runtime, a tenant binding or client registrations, and configured mTLS trust.

Authorization policy gates and ABAC overlays: served by the binary

The RBAC guard, ABAC deny overlay, OPA/Rego default-deny policy gate, RA scope split, and dual-control approval are enforced by the running binary, not just in library code. The RBAC guard runs on every guarded API route. When auth.abac.enabled is set, the ABAC deny overlay runs after RBAC on guarded routes using request, actor, environment, and time attributes; on the served lifecycle transition (POST /api/v1/identities/{id}/transitions) for issue, deploy, and revoke, trstctl adds identity resource tags before the deny check. The OPA/Rego lifecycle policy gate then gates issue/deploy/revoke fail-closed before the orchestrator records the transition or enqueues the mint/revoke effect. The gate is tenant-scoped under per-tenant database isolation, recorded as immutable events, and runs on its own bounded worker lane so a policy flood fast-rejects rather than starving other subsystems.

Registration-authority (RA) separation and dual-control approval are served. The gate enforces the RA scope split: a privileged issue/revoke transition requires the certs:issue authority, so a certs:request-only requester (the ra-officer) cannot self-issue. When dual control is enabled (ca.policy.require_approval), a privileged action is denied until a distinct approver records an issue, rotate, or revoke approval via POST /api/v1/identities/{id}/approvals (itself requiring certs:issue); self-approval is rejected, backed by tenant-isolated approval-request and approval records. This is the served half of the "loaded gun" defense — the bootstrap token already withholds certs:issue; the served mint now enforces the RA split and dual control too. The /request and /approvals console pair is served for profile-bound certificate requests: it records requester/profile/purpose metadata, keeps the request in requested, blocks requester self-issue, accepts a distinct approval, and mints through the signer-backed outbox. The request is a first-class object (/api/v1/issuance-requests, trstctl issuance-requests) with its own lifecycle: requested, then approved, denied (by a distinct approver, with a reason), cancelled (by the requester), expired (seven days after it opened, recorded with no decider because nobody decided), or issued; the list keeps closed requests for audit (A117, proven on the lab: open, withdraw, deny with a reason, refused self-approval, approve).

The OPA/Rego policy gate is default-deny on issue/deploy/revoke. With ca.policy.enabled set, the served binary invokes the embedded policy engine on every issue/deploy/revoke transition: the request is denied unless the deployed Rego policy explicitly allows it (default-deny, fail-closed). The policy input carries the action, tenant_id, the actor, and the bound profile name, so an operator can enforce a real Rego document at runtime. A non-compiling policy module is a hard startup error, an evaluation error denies, and a saturated policy pool sheds with a 503 (never an allow). The built-in base policy is default-deny, permits revocation, and requires a bound certificate profile to issue/deploy (composing with the profile enforcement below). Enforcement is off by default (ca.policy.enabled=false) so an in-place upgrade does not silently start denying; the RA scope split is enforced for privileged transitions regardless of this flag. Live policy authoring, activation, listing, and rollback are served through POST /api/v1/policy/versions, GET /api/v1/policy/versions, POST /api/v1/policy/versions/{id}/activate, and POST /api/v1/policy/versions/{id}/rollback; activation compiles the module, records policy.version.activated, and only then installs it into the running mutation gate.

The ABAC deny overlay is served. With auth.abac.enabled set, the served binary compiles a package trstctl.abac Rego module at startup and evaluates it after RBAC. It is deny-only: it cannot grant access that RBAC refused. Every guarded route carries input.permission, input.resource.request.method, input.resource.request.path, actor roles, input.env, and UTC time fields; issue/deploy/revoke transitions also carry identity metadata and flattened identity attributes such as input.resource.env and input.resource.tags.service — supporting controls like "prod certs may issue only during a change window." Bad Rego is a startup error, evaluation errors deny with 403, saturated policy workers return 503, and decisions are recorded as policy.abac.decision events. Candidate policy dry-runs are tenant-facing through /api/v1/policy/dry-run and the /policy workbench.

Independently of the policy flag, when a default certificate profile is bound (ca.default_profile) the served mint validates the request against the active profile version and rejects an out-of-profile request before signing (an issuance.profile_evaluated deny event) — so the served mint is profile-gated, not ungated.

Regulated CA governance mode is one coherent posture switch. Previously the policy gate, four-eyes dual control, the bound default profile, revocation publication, and FIPS were each enabled independently, with no single mode that refused to start unless they were all coherently present — a compliance deployment could half-enable the posture and silently drop a control. With ca.governance_mode=regulated the running binary fails startup unless all of the OPA policy gate (ca.policy.enabled), distinct-approver four-eyes dual control (ca.policy.require_approval with a >= 2 threshold), a bound default certificate profile (ca.default_profile), revocation publication (ca.crl_distribution_points and/or ca.ocsp_servers), and — when ca.require_fips is declared — an active FIPS 140-3 module are present together, each with an actionable error. A complete regulated config boots; the default (standard) posture imposes no coupling. The switch is enforced in the served startup/config validation path, where the FIPS power-on self-test already asserts the module when required. See configuration → regulated CA governance mode.

Plugin isolation: first-party in-process, third-party sandboxed

This is a deliberate, documented trust boundary, not an accident.

  • Shipped connector families run in the WASM sandbox by default, on the control plane (connectors.isolation: wasm) and on enrolled host agents and network relays (trstctl-agent --connector-isolation): each binary embeds one digest-pinned first-party connector module, and every connector operation runs in a fresh wazero instance of it whose only effects are the sandbox operations its grant permits, performed by the host; the instance's memory is wiped afterwards. Every family is proven to behave identically in and out of the sandbox (same effects, same errors, same optional operations). in_process remains available. An agent does not report its choice, so the connector catalog claims isolation only for families the control plane runs. The shipped CA integrations still run as trusted in-process Go code. Their blast radius if one is defective is the executing process — on the control plane, the database connection pool (confined to the tenant by per-tenant database isolation) and the signer client handle (it can request signatures), but not the CA private key, which stays in the separate signer process. They are mitigated by code review, the conformance suite, the connector SDK's capability-scoped sandbox facade, and per-subsystem bounded worker lanes.
  • The WASM plugin host (wazero) is real and is the isolation boundary for third-party plugins. A loaded plugin has no ambient capabilities and only the host functions its grant permits; the host holds no database pool or signer handle; and a deliberately misbehaving plugin is proven contained by test. The shipped connector families now run on it (above); moving the CA integrations onto it is what comes next. See the plugin trust model.
  • Plugin extensibility is served by the binary. The WASM plugin host is wired into the served control plane: when plugins.enabled the running binary loads operator-supplied CA plugins from plugins.ca_dir and connector plugins from plugins.connector_dir (or the legacy connector alias plugins.dir). A signed CA plugin is listed under GET /api/v1/external-cas and issues through POST /api/v1/external-cas/{id}/issue; a signed connector plugin routes served connector.deploy work through the plugin's capability sandbox (the same capability-grant model the connector SDK uses) — tenant-scoped under per-tenant database isolation, recorded as immutable events, on the plugin's own bounded worker lane. The plugin runs in its own wazero runtime with no database pool or signer handle, an operation outside its grant is denied at runtime, and the surface is off by default. The built-in connector families run in the same kind of sandbox (see above); the CA integrations still run in process.
  • Served plugins are signature/provenance-verified. The served loader admits a .wasm module only after its detached Ed25519 signature verifies (through the single isolated cryptography path) against the operator-configured trusted-key set (plugins.trusted_key_files), with an optional content-digest pin (plugins.pinned_digests). An unsigned, wrong-key, byte-tampered, or unpinned module is refused and the binary fails closed at startup — it never instantiates an unverified plugin. The raw Host.Load, Conformance and ConformanceUnderGrant methods exist only in builds explicitly tagged trstctl_conformance and in package tests. Normal and Core production builds cannot compile a call to them. An architecture guard confines the private loader to verified admission and conformance helpers. Author-side conformance executes guest code under its grant; it checks compatibility, not publisher trust. The served surface runs the provenance gate before guest start code and keeps the wazero sandbox as defense-in-depth.

Protocols

  • ACME server with ARI: all three domain-validation challenges are validated for real, each failing closed — HTTP-01 (RFC 8555 §8.3), DNS-01 (§8.4, the _acme-challenge TXT digest), and TLS-ALPN-01 (RFC 8737, the acme-tls/1 id-pe-acmeIdentifier handshake) — behind a multiplexer with an automatic method selector (wildcards → DNS-01, no inbound :80 → TLS-ALPN-01, else HTTP-01). The prior accept-everything validator has been removed from the production build (it survives only in the test binary). A DNS-01 solver with a reference provider and conformance harness ships for the publish side. A real RFC 8555 client conformance suite exercises HTTP-01 end to end (the production validator fetches the published key authorization, multi-SAN issuance, a wrong key authorization fails closed), and the same protocol-conformance routine runs as a differential against Pebble (the reference test ACME CA) in CI — so a divergence from the reference surfaces as a failure. Hosted DNS provider coverage is served through the DNS-01 provider catalog (GET /api/v1/acme/dns-01/providers): Route 53, Cloudflare, Google Cloud DNS, Azure DNS, RFC 2136, webhook, NS1, Akamai, UltraDNS, and acme-dns; the catalog exposes secret-reference fields and capability grants, not raw provider tokens. Tenant DNS-01 provider configs are served through POST/GET/PUT/DELETE /api/v1/acme/dns-01/provider-configs, and POST /api/v1/acme/dns-01/preflight evaluates delegation, TXT propagation (read from live DNS unless the caller supplies its own observation, which the response labels), live CAA, method, and wildcard policy before issuance. Served ACME DNS-01 challenge acceptance checks live CAA before any DNS write, then publishes and cleans through acme.dns01.present / acme.dns01.cleanup outbox rows using tenant provider configs and secret-reference-backed credentials. Wildcard X.509 identity issuance requires an explicit blast-radius acknowledgment and validation_method=dns-01; deployed wildcard identities renew through the lifecycle scheduler's ca.renew path with rotation evidence. The ACME server is served by the running binary: it is mounted on the control-plane TLS listener at /directory + /acme/... and brokers issuance through the orchestrator-backed path — signed in the isolated signer (so the CA key never enters the API process), tenant-scoped, recorded as immutable events, idempotent (deduplicated by Idempotency-Key), and profile-gated. A stock golang.org/x/crypto/acme client with an ECDSA account key drives the served handler end to end (new-account → new-order → http-01 → finalize) and downloads a real, signer-issued certificate; a served acceptance test asserts the cert verifies and a certificate.recorded event exists, then revokes via ACME revokeCert and asserts the served OCSP responder returns revoked. The directory advertises the mandatory revokeCert and keyChange resources, and the server accepts ECDSA and Ed25519 account keys (not only RSA). Enable it with protocols.acme.enabled plus protocols.acme.tenant_id; it activates only when an issuing CA is provisioned and fails closed otherwise. The Protocols console now exposes the tenant-scoped, read-only ARI publication and scheduler-consumption posture. For certificates issued by a configured external ACME CA, the control-plane outbox worker fetches that CA's renewalInfo through the selected issuer's egress policy. The scheduler and posture use the retained per-certificate window; the console labels its source and fetch state. A discovered external leaf has no authenticated issuer, so it is matched to an authority by its Authority Key Identifier against that authority's operator-declared ari_issuer_key_ids (the Subject Key Identifiers of its issuing intermediates); a match gets the same upstream poll, the request and observation events, the row and the posture carry provenance: issuer_key_id so the two bindings are never confused, and the worker re-matches the key id before every GET. A discovered leaf whose issuer no authority declares is listed with a local estimate and nothing is polled. Local Pebble evidence qualifies the supported client contract, not a public CA's production availability or network behavior. The upstream client presents an RFC 8555 §7.3.4 external account binding when the letsencrypt-type authority sets eab_key_id and eab_hmac_key_ref together (CertiNext/InCommon, HARICA/GÉANT TCS and other EAB-gated CAs); the MAC key is a credential reference held as locked, zeroed bytes, never logged or served, and one field without the other refuses to start. The binding is proven against a local EAB-requiring ACME double (registration without a binding or with the wrong key is refused, with it issuance and revocation run); a live proof against CertiNext or HARICA needs a design partner's credentials. The ACME admin console is served (A52): GET /api/v1/acme/accounts lists the mount's accounts (status, admitting EAB credential, order and certificate counts), GET /api/v1/acme/accounts/{id}/orders drills one account down to each order's authorizations and challenges and names the issued leaf's serial, fingerprint and inventory record id, and POST /api/v1/acme/accounts/{id}/deactivate retires an account on the operator's authority (the same acme.account.upserted event a client's own deactivation writes: new requests under the key are refused, pending orders are cancelled, issued certificates stay). Revocation from that console runs through POST /api/v1/certificates/bulk-revoke with the named record id, so it carries the certificate path's approvals and OCSP/CRL effects rather than a second revocation path. CLI acme accounts list|orders|deactivate. An account is named by the public RFC 7638 thumbprint of its key; no served field carries key material, and a token for another tenant sees an unserved list. Client setup stays the readiness panel's credential-free command.

  • External account bindings are authorizations, not door keys. RFC 8555 §7.3.4 EAB proves an ACME account key was pre-authorized out of band. The server used to verify that proof and discard the key id, which made every admitted account identical: nothing recorded which credential let it in, so nothing could scope what it asked for next, count what it had taken, or stop one credential without stopping all of them. An account now remembers its kid — in its state event, so it survives a replay — and every order under that account is checked against that credential's policy. A credential may carry allowed identifiers (exact names or *.suffix, which covers the apex too), an order quota, and a validity window; an out-of-scope identifier, an exhausted quota, or a closed window refuses the order fail-closed, names the cause, and records an acme.eab.order_denied event. A single out-of-scope identifier refuses the whole order — issuance is granted or it is not. GET /api/v1/acme/eab-credentials serves each credential's scope, quota, window, and live accounts-bound / orders-created / orders-denied counters, and POST /api/v1/acme/eab-credentials/{kid}/disable (and /enable) stops or resumes new accounts and orders under one credential at runtime. Disabling is a closed tap, not a revocation: certificates already issued under the credential stay valid. The Protocols console shows the same list and carries the disable action. The policy is served as well as configured (A53): PUT /api/v1/acme/eab-credentials/{kid}/policy (CLI acme eab policy) replaces a credential's allowed identifiers, order quota, validity window and pinned profile at runtime; the change is recorded as an acme.eab.policy_updated state event and replayed after a restart, so a protocols.acme_eab.keys[] entry is the starting policy, not the only one. Rotation is served without minting a secret (A50/A54): the operator writes the successor MAC key to the tenant secret store first, then POST /api/v1/acme/eab-credentials/{kid}/rotate (CLI acme eab rotate) with successor_key_id and a secret:// hmac_key_ref registers the successor under the predecessor's policy. The key is read through the secret store into locked memory, the acme.eab.credential_rotated event carries only the reference, and replay resolves the reference again; a reference that no longer resolves leaves the successor listed as unavailable, which admits nothing, rather than dropping it or starting the mount without it. The predecessor keeps working until the operator disables it. trstctl does not mint external account credentials over the API and accepts no key in a request body, because that would mean a shared MAC secret in an HTTP body; the HMAC key stays byte-backed in locked memory and appears in no served response, in any encoding. Per-credential profile selection is served: keys[].profile pins the tenant profile every order under that credential is issued with (the server reads that profile's authentication mode and signs under it), and without a pin the tenant's ca.default_profile applies. The served disable verb cannot re-enable a credential that configuration disables; config is the floor.

  • EST (RFC 7030), SCEP (RFC 8894), CMP (RFC 4210/6712), the SPIFFE Workload API, and the SSH CA issuance servers are served end-to-end by the running binary, each behind the same issuance seam as the API mint: signed in the isolated signer, tenant-scoped, recorded as immutable events, idempotent, and profile-gated:

    • EST at /.well-known/est/... (Bearer-API-token authenticated on top of TLS). The challenge is emitted by the configured authenticator: the served tenant route advertises Bearer plus certs:request and returns bounded RFC 6750 invalid_request, invalid_token, or insufficient_scope results, while BasicAuthenticator continues to advertise Basic. The conventional uppercase header spelling and a strict libest token wrapper keep the pinned challenge-driven reference client interoperable without changing ordinary raw Bearer behavior. SCEP at /scep, CMP at /cmp — mounted on the control-plane mux and exercised by served round-trip acceptance tests (a stock base64-PKCS#10 EST enroll, a CMS-enveloped SCEP PKIOperation, a CMP p10cr) that each download a real, signer-issued certificate verifying against the served CA and assert a certificate.recorded event in the tamper-evident log. SCEP/CMP use a sealed RSA transport identity at protocols.ra_key_file for CMS (deliberately not the CA key, which stays in the isolated signer); keep that file on shared persistent storage in HA so cached clients survive restarts and rolling deploys.
    • The SPIFFE Workload API is served as a gRPC service on a Unix domain socket (protocols.spiffe.enabled), so a spiffe-helper/go-spiffe/Envoy-SDS client dials the socket and fetches X.509-SVIDs, JWT-SVIDs, X.509 bundles, and JWT bundles. X.509-SVIDs are signed through the isolated signer; JWT-SVIDs use the signer-backed JWT handle and the served ValidateJWTSVID RPC validates them against the served JWT bundle. A served acceptance test drives the SPIFFE Workload API wire protocol (with the mandatory workload.spiffe.io metadata) over the socket and validates both SVID families. A required CI job also runs stock go-spiffe and stock spiffe-helper against that served socket; go-spiffe is a test-only dependency so the served binary does not take a new runtime dependency for the proof.
    • Host-served Workload API (B3): a host agent started with --workload-api-socket serves the same Workload API on its OWN machine, for the workloads that run there. It attests each caller from the kernel's record of the connecting process (SO_PEERCRED on Linux; LOCAL_PEERCRED, LOCAL_PEERTOKEN and LOCAL_PEERPID on darwin) and renders SPIRE-shaped unix:uid:/unix:gid:/unix:path: selectors; a platform with no peer-credential mechanism REFUSES to serve rather than issuing without attestation. The SVID key is generated on that host and only its public half travels: the control plane signs an SVID for a key it has never seen, over the FetchWorkloadSVID node API on the agent channel. Authorization is per node, not per agent-fleet: a registration entry carries a ParentID naming the node permitted to deliver it, taken from the certificate the agent authenticated with, so an agent that misreports selectors reaches only the workloads on its own machine. An entry with NO ParentID is deliverable by no agent at all and stays on the control plane's socket — so a forgotten field refuses rather than widening who may impersonate a workload, and scoping an entry moves it rather than duplicating it. On darwin the path comes from the connecting process's audit token (LOCAL_PEERTOKEN): its pid and pid version, read at connect time. The path is resolved only while that pid version still owns the pid, checked before and after the read, and LOCAL_PEERPID must agree, so a pid reused by a later process yields no path selector rather than the new process's; an entry requiring unix:path: then does not match. As on Linux, the path is the executable's at attestation time: a process that execs another binary after connecting is attested as that binary. The control plane's own socket no longer issues SVIDs to workloads unless protocols.spiffe.serve_local_workloads: true is set: FetchX509SVID and FetchJWTSVID on it are refused with a message naming the host agent's --workload-api-socket, while trust bundles and JWT validation stay served for validators. Each refusal, and each issuance when the switch is on, emits the spiffe.workload_api.local_socket_used audit event, so an operator can see which workloads have not yet moved rather than assuming they all have; the SPIFFE readiness check reports local_workloads_served. The eval profile keeps the switch on, being one machine with no host agent. The Workloads console reports, per host, whether it serves the socket, how many SVIDs it has issued since its agent started (a counter that resets on restart, and says so), and when it last reported — with "not reported" kept distinct from "not serving", because the first is fixed by upgrading an agent and the second by changing a flag.
    • Kubernetes TrustBundle distribution is served for public CA-bundle propagation: the agent reconciles cluster-scoped TrustBundle.trstctl.com resources, validates that spec.caBundlePEM contains only PEM CERTIFICATE blocks, writes namespace ConfigMaps, and marks status with target count, bundleSHA256, and Ready=True. The posture route GET /api/v1/kubernetes/trust-bundles, CLI command trstctl-cli kubernetes trust-bundles, and Workloads console disclose the CRD, RBAC, ConfigMap target, and residuals: the controller is event-driven — it watches its TrustBundles and the ConfigMaps it wrote, converges a change or a deleted copy within about a second, and writes nothing when a copy already holds the bundle. One bundle can be rolled out from the control plane to every reporting cluster, canary first (/api/v1/kubernetes/trust-rollouts, trstctl-cli kubernetes trust-rollouts, Workloads): each cluster's agent receives it in the answer to its posture report and applies it as a TrustBundle it owns, and a cluster reads verified only when its own controller reports the bundle distributed with the rollout's hash. A TrustBundle can also name a Secret in each target namespace and a ClusterTrustBundle that pods mount through a projected clusterTrustBundle volume, and the opt-in CSI node plugin (deploy/kubernetes/csi-trust.yaml, driver trust.csi.trstctl.com) serves a TrustBundle to a pod as an inline volume; focused kind tests prove each target end to end, including a changed bundle reaching a running pod. Their limits: Secret targets outside the agent's namespace need the opt-in grant in deploy/kubernetes/rbac-trust-secrets.yaml, which lets the agent read any Secret by name; ClusterTrustBundle needs a cluster serving certificates.k8s.io/v1beta1 (off by default through Kubernetes 1.35) and the kubelet projection feature gate; the CSI plugin runs as root without capabilities to write into the kubelet's pod directories, and a changed bundle reaches pods on the kubelet's next republish. A bundle that cannot be distributed is marked Ready=False with the cause.
    • The SSH CA is served at /ssh/... (protocols.ssh.enabled): cert issuance plus the OpenSSH binary KRL at /ssh/krl (sshd's RevokedKeys consumes it); a served acceptance test issues a user cert (verified with ssh-keygen -L), revokes it, and confirms the served KRL is the binary format. The SSH workflow API and CLI also serve status, explicit-confirmation trust-rollout evidence, attestation-gated user cert issuance, KRL revocation, and host retirement handoff. The SSH CA key lives in the isolated signer under its own handle constrained to SSH-cert signing.
    • The RFC 3161 TSA is served at /tsa (protocols.tsa.enabled): clients POST application/timestamp-query TimeStampReq bodies and receive application/timestamp-reply TimeStampResp bodies. The timestamping key lives in the signer under its own stable handle, the TSA certificate is persisted at protocols.tsa_cert_file, and the certificate carries the critical timeStamping EKU that stock OpenSSL enforces. The authenticated POST /api/v1/protocols/tsa/qualification surface and Protocols console panel review the exact tenant-bound mount, certificate, isolated signer, audit, and bulkhead posture without reading files, calling the signer/network, issuing a timestamp, or writing state. They name a secure repair for each failed gate and keep retry available. This is readiness only; stock OpenSSL over /tsa remains the wire and signature-verification proof.
    • The code-signing service is served by the running binary at POST /api/v1/code-signing/sign and POST /api/v1/code-signing/keyless when code_signing.enabled is configured. Tenant-scoped persistent keys and one-use keyless keys stay in the isolated signer process. GitHub OIDC identity is verified from tenant-pinned JWKS before the Fulcio SAN/issuer is derived. Rekor publication uses the transactional outbox and acknowledges only an exact HashedRekord receipt whose signed-entry timestamp verifies under the operator-pinned log key.

    Each protocol surface is gated by protocols.<name>.enabled and binds a tenant via protocols.<name>.tenant_id. All protocol toggles default off until an operator explicitly binds the served endpoint to a tenant; if a protocol is enabled without a tenant, startup validation fails before the route is exposed (per-tenant isolation forbids minting evidence into a blank tenant). All protocols activate only when an issuing CA is provisioned. HTTP protocol namespaces stay machine-only even while a toggle is off: /directory + /acme/, /.well-known/est/, /scep, /cmp, /ssh/, and /tsa return a stable 404 application/problem+json response instead of falling through to the React console. Unknown /directory/, /cmp/, and /tsa/ children are reserved too. The Protocols register does not infer availability from HTTP 200 alone; it verifies each responder's protocol-specific status, media type, and public discovery payload, so console HTML cannot masquerade as ACME, EST, SCEP, CMP, SSH CA, or TSA readiness.

    • Reference-implementation differentials: cross-checked against an independent implementation in addition to our own parser. ACME: a differential against Pebble (the reference test ACME CA) as a dedicated CI job, plus a stock certbot CI transcript (certbot manual DNS-01 issues, renews, and revokes through the served /directory endpoint while CI archives public challenge records, client logs, and issued certificates). EST: a differential against the OpenSSL pkcs7 parser/verifier on every make test (so /cacerts and /simpleenroll output is validated by code we did not write), plus a dedicated CI job that builds a checksum-pinned libest estclient from source, removes its verbose access-token print before archiving transcripts, and requires it to follow the Bearer challenge and perform simpleenroll against the served EST endpoint. SPIFFE Workload API: a served stock-client differential — the real go-spiffe workloadapi client fetches an X.509-SVID, a JWT-SVID, and JWT bundles, and validates the JWT-SVID over the served UDS; stock spiffe-helper writes the served X.509-SVID, key, and trust bundle to disk. CMP: a dedicated stock-client CI transcript — OpenSSL cmp -cmd p10cr creates the request, enrolls through the served /cmp endpoint, accepts the protected response, and uploads the request/response/cert/log artifacts. SCEP: a dedicated stock-client CI transcript — a SHA-256-pinned sscep v0.10.0 build fetches the served CA-plus-RA bundle, classifies its numbered public files by exact issuing certificate and signing constraints rather than suffix order, enrolls through /scep/pkiclient.exe, and uploads both CA/RA files plus the captured PKIOperation request/response and client logs. TSA: a dedicated stock-client CI transcript — OpenSSL ts -query creates a DER TimeStampReq, CI POSTs it to the served /tsa endpoint, OpenSSL ts -verify validates the returned TimeStampResp, and public request/response/log artifacts are uploaded.
    • SSH KRL distribution format. The SSH CA's key-revocation list is emitted in the OpenSSH binary KRL format, the artifact sshd's RevokedKeys and ssh-keygen -Q -f consume — verified end-to-end by a test that has stock ssh-keygen report a revoked certificate as revoked using trstctl's KRL (and a non-revoked one as valid). A legacy JSON revocation snapshot is retained for programmatic callers.
    • Public-CA profile linter. Issued certificates are checked by a built-in structural RFC 5280 / CA-Browser-Forum profile linter in the issuance test suite — version, serial bounds, validity ordering/length, basicConstraints, key usage, SAN presence, SKI/AKI presence, weak-signature and minimum-key-strength checks — and the suite is red on a deliberately-broken profile. The CI gate also generates a PEM corpus for every emitted X.509 profile shape (served leaves, mTLS agent certificates, SPIFFE X.509-SVID, TSA, and the issuing CA), runs pinned zlint over the served CA plus that corpus, and uploads the generated fixtures and JSON lint transcripts as artifacts. This is a private-CA assurance gate (for your own internal PKI), not a claim that trstctl operates as a WebPKI public CA.
  • SPIFFE transport (Workload API): the X.509-SVID document is spec-shaped (a single spiffe:// URI SAN, correct key usage), and the Workload API is served as a gRPC service on a Unix domain socket (protocols.spiffe.enabled). A spiffe-helper/go-spiffe/Envoy-SDS workload dials the socket for FetchX509SVID, FetchX509Bundles, FetchJWTSVID, FetchJWTBundles, and ValidateJWTSVID. The X.509-SVID workload key is minted server-side and returned in the response (per the spec); the X.509-SVID CA is the served issuing CA in the signer and the JWT-SVID signing key has its own signer handle. The Workload-API gRPC/protobuf contract is vendored verbatim from go-spiffe so the wire format is byte-identical without a build-time go-spiffe dependency.

  • SPIRE upstream authority: the trstctl-spire-upstream-authority plugin puts the served CA hierarchy behind SPIRE as its X.509 upstream: SPIRE sends its local CA CSR to /api/v1/ca/authorities/{id}/intermediates/csr. The served route signs only under an approved issue_intermediate_csr ceremony for that exact CSR and the plugin sends no ceremony ID, so every SPIRE CA rotation needs custodians to approve a ceremony for its new CSR; there is no standing approval for an upstream authority. CI's SPIRE container test signs with a stand-in for trstctl; a served container test opens and approves each ceremony through the API. The plugin binary is built by make build and published from every release tag as trstctl-spire-upstream-authority-linux-{amd64,arm64} GitHub Release assets with a SHA-256 manifest and SLSA provenance. SPIRE loads it from its own host filesystem (plugin_cmd), so it ships as a standalone binary and is not part of the trstctl container image. The plugin also publishes each SPIRE server's JWT signing key to /api/v1/spiffe/upstream-jwt-keys and returns every unexpired key the tenant's SPIRE servers published, with updates on the open subscription, so SPIRE servers sharing a trust domain through trstctl verify each other's JWT-SVIDs (two SPIRE server containers, each verifying the other's JWT-SVID with its own bundle). trstctl keeps the public keys only; its own SPIFFE JWT authority is not added to that set, and SPIRE's optional SubscribeToLocalBundle is not implemented. Against a trstctl that does not serve that route (an older trstctl's 404, or a 405 or 501), the plugin reports JWT key publication as unsupported, so SPIRE keeps its JWT keys in its own bundle and still starts; the next JWT key rotation publishes again.

  • Attested issuance transport (REST): POST /api/v1/workloads/attested-issuance is the served proof-before-trust mint for workloads that already have their own key pair. The request carries the attestation method, base64 proof payload, public key PEM, and requested TTL; the response carries the signer-issued X.509-SVID PEM, credential id, verified subject, expiry, and attestation metadata. The SPIFFE ID is derived from the verified attestation subject, not caller-supplied text. Acceptance coverage exercises a Kubernetes projected service-account token, an AWS instance-identity document with an emulated trusted root, idempotent replay, and a forged AWS document rejection.

  • AI-agent broker issuance (REST): POST /api/v1/broker/agent-identities is served when the agent broker is configured with attestors, a policy module, trust domain, and signer-backed issuing CA. The route requires certs:issue and an Idempotency-Key; it verifies the agent proof, evaluates policy before signing, mints a short-lived X.509-SVID, records certificate.recorded, emits agent.identity.issued or agent.identity.refused, and projects the agent-to-credential edge into the graph. The Workloads page now has an effect-free server preview, explicit issuance, unchanged-request retry after uncertain delivery, and durable broker history with search, state/method filters and detail readback. History requires certs:read independently of issuance. It shows original public issuance facts and projection freshness, never raw proof, task text or internal recovery bindings. Missing or privacy-erased facts remain explicitly unavailable. Clearing or reloading the form is not cancellation: inspect history and audit before choosing a new recovery key. Revocation uses shared certificate controls. Fresh-image API/CLI/browser, negative-security and preserved-volume restart qualification is still required before this source workflow can be called a complete F61 vertical slice.

  • Agent key sets for authorization servers: GET /agent-keys/{tenant}/{agent}/jwks.json is public, like the CRL and OCSP routes, and lists the public keys of the agent's valid broker certificates (an unknown agent gets the same empty set); a revoked certificate's key leaves it at once. An authorization server that reads it can authenticate the agent with private_key_jwt, which is how an agent with a trstctl identity obtains tokens under MCP authorization (guide); a lab test proves it against Keycloak 26.8 with token exchange. How long a revoked identity can still obtain tokens is set by the authorization server's key cache (Keycloak: the keys cache lifespan), and tokens already issued live until they expire. What comes next: the same proof with a production identity provider.

  • Ephemeral / JIT issuance: effect-free POST /api/v1/ephemeral/preview and mutating POST /api/v1/ephemeral are served when ephemeral issuance is enabled with approval TTL/threshold, trust domain, and signer-backed issuing CA. Each tenant supplies public verification material through an enabled workload attester trust source; no process-wide trust source silently applies to every tenant. A requester with certs:request presents a proof and public key; trstctl verifies the proof, opens an approval request, and enqueues the approval notification intent in the same tenant transaction. A distinct approver with certs:issue records approval at POST /api/v1/ephemeral/{id}/approvals, where {id} is the genuine queue approval_request_id and the JSON body carries action: issue, the same UUID as request_id, and its matching intent_digest; the requester then calls /api/v1/ephemeral with a fresh Idempotency-Key to mint the short-TTL credential. The Workloads page now provides the same exact-preview, send-for-approval, approval-queue handoff, safe-retry, and public-certificate collection workflow. It retains only digests and public metadata after issue; the proof is cleared and private-key bytes never enter trstctl. Ephemeral API keys are served separately at POST /api/v1/ephemeral/api-keys and trstctl-cli ephemeral api-keys issue: callers provide subject, scopes, and ttl_seconds, the raw token is returned once, and the leaseworker emits api_token.revoked at expiry. Headless operators can use trstctl-cli ephemeral preview, ephemeral issue, and ephemeral approve; the approval command requires the genuine approval_request_id in both the path and body plus its matching intent_digest. The certificate inventory retains issued JIT leaves and their public custody metadata. The health dashboard separates their intended short expiry into expected_ephemeral; it does not send ordinary renewal alerts for them or put them in the expiry action queue. Operators request a new attested credential from Workloads after expiry. This classification follows the retained issuance command even if a discovery scan later changes the source label.

  • Agent ↔ control-plane mTLS gRPC channel: the agent steady-state channel is served by the running binary when agent_channel.enabled (off by default — an upgrade does not silently open an agent port). The control plane mounts an agent-facing gRPC listener (default :9443) over mutual TLS, and an enrolled agent connects to it to (a) heartbeat its inventory/status — the server records the agent tenant-scoped and emits an agent.heartbeat event in the tamper-evident log; (b) renew its own certificate before expiry — a fresh cert is minted through the signer-custodied agent CA, idempotently on the presented serial (deduplicated so a retry does not mint twice), recorded as an agent.cert.renewed event; and (c) report local inventory as metadata-only discovery findings, including public OS/Java/NSS/browser/ Windows trust-store anchors and private-key-material locations/ classifications from configured roots. Inventory reports create a tenant-scoped discovery source, run, finding rows, discovery.* events, and credential-graph nodes; they do not carry private keys, PEM/DER key bytes, or secret values, and secret-looking inline metadata keys are rejected before projection. The tenant is derived from the agent's verified client-certificate SPIFFE SAN, never a request field. The GET /api/v1/agents response also publishes the served agent.mtls.ReportInventory path and the source kinds the shipped agent binary can actually collect — filesystem, trust-store, k8s-secret, windows-store, private-key, and ssh — each with the flags that switch it on, so a capability that is listed but unconfigured is not read as coverage that is running. pkcs11 ships only in a cgo-enabled agent build, because it dlopens a vendor module and the default agent binary is deliberately statically linked; the census is build-dependent so a cgo-free binary neither carries the reader nor advertises the kind. All three formerly-unbuilt kinds were previously listed here and on the API, which read as a Windows estate, token store, and Kubernetes Secrets being inventoried when nothing was collecting them. Advertised capability is derived from the agent package's own record of what it ships, and docs/agent_advertised_capability_test.go fails the build if a kind is advertised without a constructor the agent binary calls. k8s-secret left the unbuilt list when its enumerator was actually wired: with --inventory-k8s-secrets the agent enumerates the TLS Secrets in its own namespace through the in-cluster service account, reading tls.crt and never tls.key, and reports metadata-only findings over the same mTLS inventory path as every other source. It needs list access to Secrets in that namespace. --inventory-k8s-namespaces all or a comma-separated list widens one agent's inventory across namespaces (all needs cluster-scoped list on namespaces and Secrets, shipped opt-in as deploy/kubernetes/rbac-inventory-namespaces.yaml); a namespace the service account cannot list is reported as an error for that namespace after the readable ones, never as an empty result. windows-store left the unbuilt list when its crypt32 reader shipped: with --inventory-windows-stores the agent opens the named machine or user stores read-only, walks each certificate context, and reports metadata only — it never asks the platform to export a private key, so a key held in a TPM or on a smart card is untouched and irrelevant. MY, WEBHOSTING, CA, ROOT and TRUSTEDPUBLISHER are supported; WEBHOSTING is where IIS keeps site certificates on current Windows Server, which is the population most likely to expire unowned. An unreadable store fails the report rather than contributing nothing, and on a non-Windows build the source returns an error, never an empty inventory — telling a Linux operator their Windows estate is clean would be worse than the original defect, because it would arrive with the authority of a scan that never happened. pkcs11 completed the set: a cgo build opens the configured module, walks each matching token over a read-only, non-read-write session, and reads CKA_VALUE from objects of class CKO_CERTIFICATE. It never searches for CKO_PRIVATE_KEY, never calls C_Sign, and never asks a token to export anything — a PKCS#11 key is normally CKA_EXTRACTABLE=false and could not leave regardless, but the code does not ask. Login is optional: public certificate objects are readable without one, and the PIN, when a token needs it, comes from a file rather than a flag, because process arguments are readable by anyone who can list processes — the same rule the bootstrap token follows. A cgo-free build returns an error rather than an empty token inventory, and does not advertise the kind at all. The channel is behind its own bounded agent worker lane and per-connection gRPC stream cap, so a heartbeat or renewal storm sheds with ResourceExhausted rather than starving API, protocol, outbox, or signer capacity. Agents announce an explicit protocol/capability handshake and schedule heartbeats from the server hint with bounded jitter, so rolling upgrades and fleet restarts do not synchronize a thundering beat. The agent CA key lives in the isolated signer under a stable handle, so it does not regenerate per boot — an agent's pinned CA survives a control-plane restart (the earlier in-process/per-boot stand-in is replaced when the channel is enabled, and the same signer-custodied agent CA also signs the bootstrap enrollment, so a bootstrap-enrolled agent is accepted on the steady-state channel). The shipped chart exposes the channel: when agentChannel.enabled, the control-plane Service publishes the agent port 9443 (agent-grpc), the container exposes it, and the NetworkPolicy admits it (from the configured agentChannel.allowedCIDRs plus the in-cluster peers the API admits) — so the fleet manifests (deploy/kubernetes/daemonset.yaml, the Windows MSI) that point agents at :9443 reach a served port. This is distinct from the isolated signer's :9443 (a signer-only Service under signer.mode=isolated, which admits only the control plane). An untrusted/unpinned agent client is rejected at the mutual-TLS handshake (fail-closed). Proven end-to-end by acceptance tests (real signer + embedded Postgres: enroll → heartbeat → endpoint inventory report → served API capability readback → Discovery findings → graph node, plus renew → idempotent retry → reject untrusted) and rendered-chart assertions.

  • Embedded HTTP enrollment renewal: POST /enroll/bootstrap is mounted on the served control-plane HTTPS listener. When agent_channel.enabled, the binary also mounts POST /enroll/renewal on a dedicated agent-CA mTLS HTTPS listener (agent_channel.http_renewal_addr, default :9444) that uses tls.RequireAndVerifyClientCert semantics against the signer-custodied agent CA. Bootstrap consumes a one-time token. Renewal requires the current verified agent client certificate, rejects missing or untrusted peers at the live TLS boundary, rejects expired verified peers in the handler, and deduplicates on the presented certificate fingerprint plus CSR when the served API has idempotency storage. The renewed certificate is tenant-attributed from the verified peer, never from a request header or CSR field.

Revocation

Revoking a credential through the running binary is real and recorded, not a no-op. Transitioning an identity to revoked drives the served outbox handler to mark the issued certificate revoked in the inventory — via a projected certificate.revoked event, so the status is reconstructable from the log on a read-model rebuild, and the certificate API returns status / revoked_at / revocation_reason so the revocation is visible on the served surface (a revoked cert reads "revoked", not silently "active") — and project the certificate's serial into the revocation read model from the same event, so OCSP/CRL state is rebuilt from the log instead of from a side write.

The online revocation-distribution surface is served: the running binary mounts an RFC 6960 OCSP responder at /ocsp/{tenant} (GET base64-in-path and POST application/ocsp-request), an RFC 5280 full CRL endpoint at /crl/{tenant}, a manifest at /crl/{tenant}/manifest.json, partitioned shard CRLs at /crl/{tenant}/shards/{index}, and RFC 5280 delta CRLs at /crl/{tenant}/delta/{base}. The freshness scheduler regenerates each tenant's CRL set ahead of nextUpdate. Trusted issue, renewal, revocation, protocol-enrollment, and scheduler paths publish CRLs; public CRL reads are read-only and return 404 until artifacts are already published for a tenant that has issued certificates.

When a local revocation has been accepted but its event has not projected, OCSP refuses a Good answer and CRL readers refuse the predecessor artifact. After projection, CRL readers continue to refuse an older signed artifact until its revoked count covers the current issuer ledger. The replacement is published by the outbox worker; clients configured to soft-fail may still accept a leaf during this unavailable interval, so relay health and stock-client rejection remain separate evidence. Both are proven on the running product: a network relay reports the managed CA's CRL fresh and signature-verified with the revocation counted, and its OCSP answer signature-verified; stock OpenSSL 4 refuses the revoked leaf with verify -crl_check (error 23), reads it as revoked over OCSP, and refuses a TLS connection made with -crl_check, while a TLS client that does not check revocation still connects.

A query for a revoked serial returns revoked over OCSP and the serial appears on the full CRL, its shard, and any applicable delta CRL within the freshness window; a query for an issued-but-not-revoked serial returns good; an unknown serial returns a signed unknown. The shard plan is 4-1024 partitions targeting roughly 100k revoked serials per shard, so 10-100M-row estates use bounded shard/delta fetches while retaining the compatibility full CRL. These endpoints are public by RFC design (relying parties check status without credentials) but run on the API worker lane, so an OCSP/CRL flood sheds rather than starving the rest of the control plane.

OCSP responses and CRLs are signed through the out-of-process signer: the signing op crosses the single isolated cryptography path using the same signer-held CA key the leaf path uses, so the CA private key never materializes in the control plane — only the digest crosses (for an ML-DSA authority, the TBS bytes the pure signature covers). Every query is tenant-scoped. Each published CRL emits a ca.crl.published event that carries the CRL DER artifact metadata, parent/base CRL number, revoked count, and validity window, so the published-CRL read model is rebuilt from the event log. This is exercised end to end in the local acceptance suite: issue, revoke, assert OCSP returns revoked (and good before revocation), assert the full/sharded/delta CRLs list the right serials within the freshness window, and verify the signatures against the issuing CA over real HTTP against the assembled binary and the real out-of-process signer.

The CDP/AIA pointers stamped on issued leaves are operator-configured (ca.crl_distribution_points / ca.ocsp_servers) because the externally reachable URL is deployment-specific; point them at the binary's /ocsp/{tenant}, /crl/{tenant}, and, where clients support it, the shard/delta distribution URLs (behind your ingress) so relying parties discover and fetch revocation status automatically. Existing leaf certificates keep the URLs they were issued with until reissued. trstctl revocation is both authoritative in the product's own inventory/records and publishable to external relying parties over served OCSP/CRL.

Signer-backed CA hierarchy leaves use their own tenant-and-authority status routes (/crl/{tenant}/authorities/{ca-id} and /ocsp/{tenant}/authorities/{ca-id}), including after that authority is superseded. New leaves are inventoried with their signed public bytes; their first CRL publication is queued in the same tenant transaction as the issuance projection. Exact-certificate revocation checks the immutable issuance event, exact leaf signature and serial ledger before it records the revocation and queues that authority's CRL. The authenticated /api/v1/revocation/crls list returns the issuer-specific URL. Managed-CA CDP/AIA pointers require the operator's ca.hierarchy_revocation_base_url; without it, standard-mode leaves have no automatic status pointer, while regulated mode refuses the managed-CA issue request. Existing leaves retain their original extensions, so reissue them to add discoverable pointers.

Older managed-CA hierarchy version-2 issuance events include the public leaf and are replayed into inventory on upgrade after exact issuer-signature verification. Same-version responder-only events (an issuance surface recorded the leaf for OCSP and CRL, usually beside another event that inventoried it) stay serial evidence in the projection; when no event inventoried that leaf, the first lifecycle sweep after an upgrade checks it against the managed CA's stored certificate and its serial against the CA ledger, then appends one ca.certificate.inventory_recovered event that gives it its own inventory row (custody unknown). The inventory workflow then revokes it against that managed CA. The inventory identity stays stable across rebuilds and the original issuance event remains its revocation authority. Version-1 events retain only a serial; trstctl cannot reconstruct their missing public certificate or infer key custody from that serial. An operator who still holds such a leaf's public certificate submits it to POST /api/v1/certificates/managed-ca-recoveries (trstctl certificates inventory-managed-ca -f leaf.pem), which accepts it only when one of the tenant's managed CAs signed it, by that CA's stored certificate, and a version-1 mint of that CA recorded its serial; the leaf then gets its own inventory row once (custody unknown) and is revoked against that managed CA through the inventory workflow. A leaf another CA signed, or one no mint recorded, is refused and records nothing. A version-1 leaf whose certificate nobody kept remains serial evidence.

CT log submission is served as an outbound side effect, not as an inline API call: the API validates public certificate PEM and CT log URLs, records ct.submit outbox rows in the tenant transaction, and the worker posts RFC 6962 add-pre-chain / add-chain requests. Public HTTPS logs are required by default; allow_private_endpoint additionally requires egress:private and private_egress_cidrs destination grants. A request may name each log's public key in log_public_keys (base64 DER or a PEM block). trstctl keeps every SCT a log returns (ct.sct.recorded): one from a log with a key is verified against it over the exact RFC 6962 signed entry — the certificate, or a precertificate's issuer key hash and TBSCertificate without its poison — and the log id that key implies, and is recorded verified or invalid with the reason. One from a log without a key, or a precertificate SCT the submitted chain cannot check (no issuer, the wrong issuer, or a precertificate signing certificate), is kept unverified, and the response names the logs without a key. For each verified SCT the leader fetches the log's get-sth, checks the tree head's signature with the same key, asks get-proof-by-hash for the entry in that tree and verifies the audit path to the signed root (ct.inclusion.recorded): an entry not yet merged stays pending; a tree head the log signed more than 24 hours after the SCT that still lacks the entry records it not included; a proof or tree head that does not verify records a failed proof. The 24 hours is the maximum merge delay RFC 6962 logs publish; a log that publishes a longer one is not read from its metadata. Logs that serve only the static CT API (tiles instead of get-sth) are submitted to and their SCTs verified, but their inclusion is recorded not checked; proving it from tiles is what comes next. GET /api/v1/certificates/{id} and the certificate detail show each log's answer.

Single sign-on

trstctl's interactive sign-on is served for OIDC, SAML 2.0, and LDAP / Active Directory. OIDC is core; tenant SAML and LDAP require Enterprise SSO. OIDC supports the authorization-code flow against Microsoft Entra ID / Azure AD, Okta, Ping, Google, Auth0, Keycloak, and similar providers. SAML serves a Service Provider with SP-initiated login (/auth/saml/login), IdP-initiated login through the ACS (/auth/saml/acs), and SP metadata (/auth/saml/metadata). SAML assertion verification requires configured IdP metadata and accepts signed HTTP-POST binding responses, and the HTTP-Artifact binding (auth.saml.response_binding: artifact, also provider.saml.response_binding): the browser then carries only an opaque artifact to the ACS and the SP resolves it over a back-channel SOAP call to the IdP's artifact resolution service, dialed under the same endpoint rule as every configured URL (https, or plain http on a loopback host), so the signed response never transits the browser; an artifact arriving at an SP that did not ask for the binding is refused, and an IdP whose metadata declares no SOAP artifact resolution service fails startup. Encrypted assertions are served (auth.saml.sp_key_file, provider.saml.sp_key_file): the SP keeps an RSA identity sealed at rest under the deployment KEK (generated on first start, shared by replicas), advertises its certificate in the SP metadata with use="encryption", and decrypts EncryptedAssertion elements inside the cryptography boundary; an IdP that honors the metadata then never sends the assertion in the clear. Without the file, assertions are accepted in plaintext as before. Single logout is served (auth.saml.single_logout with auth.saml.slo_url): a sign-out from trstctl revokes the local session and then hands the browser a signed LogoutRequest for the IdP's SingleLogoutService (the console follows the logout_url that POST /auth/logout returns); the IdP's LogoutResponse at /auth/saml/slo must answer that browser's own request. An IdP-initiated LogoutRequest at the same route is verified against the IdP metadata (signature, destination, issuer, freshness), every session of that NameID in the tenant the SAML mapping resolves is revoked, and the IdP receives a Success LogoutResponse; a NameID the mapping cannot place in a tenant signs nothing out and is refused, and a tenant SAML login that overrides the subject with subject_attribute is matched by that subject, so an IdP that names only the NameID then ends nothing. The provider plane's SAML sign-in has no single logout yet. LDAP / Active Directory serves username/password bind at POST /auth/ldap/login, supports direct-bind or service-account user search plus group search, and maps directory groups to tenant roles. Password-change flows are served (auth.ldap.password_change.enabled): POST /auth/ldap/password proves the current password by binding as the user, then changes it through the LDAP Password Modify extended operation (mode exop, the OpenLDAP / 389-ds / FreeIPA path) or, for Active Directory (mode active_directory, ldaps:// required), by replacing unicodePwd as the user; the directory's own refusal of a new password (history, complexity, minimum age) comes back as HTTP 422 with its reason, and a wrong current password changes nothing. A sign-in whose password the directory reports expired or "must change at next logon" (OpenLDAP ppolicy passwordExpired / changeAfterReset, Active Directory data 773 / data 532) is answered with problem.auth.password_change_required instead of a session, and the console opens the change form on it. Nested groups map roles when auth.ldap.nested_groups.enabled is on: mode recursive walks parent groups through group_filter ({group_dn}, default (member={group_dn})) up to max_depth (default 5, ceiling 20) with a seen-set so a membership cycle cannot loop, and mode in_chain asks Active Directory for the transitive set in one search (LDAP_MATCHING_RULE_IN_CHAIN); a mapping on a grandparent group then signs the user in. Directory writeback is served (auth.ldap.writeback): a role change or offboarding an operator makes in trstctl to a member whose subject is a DN under base_dn is queued in the same transaction (outbox, AN-6) and applied by the licensed worker through the service account to the directory's role groups (role_groups maps a trstctl role to a group DN; the member DN is added to the groups its roles map to and removed from the mapped groups they no longer cover, idempotently), each application recorded as ldap.writeback.recorded with the groups added and removed; a directory refusal (missing group, insufficient access, policy) is recorded as refused and not retried, a transport failure is retried, and a member outside the base is never written back. API/CI access still uses scoped API tokens. Kerberos/GSSAPI browser login is served (auth.kerberos, Enterprise SSO): GET /auth/kerberos/login answers with a Negotiate challenge, the SPNEGO ticket the browser presents is verified against the service keytab (keytab_file, optionally pinned to service_principal and realm) inside the cryptography boundary, and the principal (user@REALM) maps to a tenant through tenant_mappings by subject, or by Active Directory PAC group SID when pac_groups is on (leave it off for an MIT KDC, whose PAC carries no logon info); a ticket under another key, an expired or replayed one, or a principal that maps to no tenant is refused. The console offers it as "Continue with Kerberos"; browsers negotiate only for hosts they trust for integrated authentication, so the console host must be in that zone. Built and proven against a local MIT KDC. NTLM browser login is served the way non-Windows servers serve it (auth.ntlm, Enterprise SSO): trstctl holds no NT hashes and validates no NTLM response itself; GET /auth/ntlm/login runs the NTLM challenge sequence over the client's connection and relays each message to a domain-joined helper speaking the squid-2.5-ntlmssp line protocol (Samba's ntlm_auth, reached as a local helper_command or over a helper_address socket), and only an account the helper reports as authenticated (DOMAIN\user, normalized to user@DOMAIN, optionally pinned to domain) maps to a tenant through tenant_mappings. A handshake is bound to the connection that started it, expires after thirty seconds, and a refused or completed one cannot be replayed. Built and proven against a Samba Active Directory domain controller; a Windows forest is the remaining proof.

CA key custody

The assembled issuing CA's key is persisted, sealed at rest in the signer's key store: a signer restart preserves the CA instead of silently rotating it, and the key survives across restarts. Root/intermediate m-of-n ceremonies and signer-backed leaf issuance are served. The release also publishes a cgo-enabled HSM signer artifact that links the native PKCS#11 binding; the default static control-plane artifact does not load native modules. The conditional managed-key path has required launched-binary receipts for all six advertised providers, while the sealed local signer key store remains the default when no provider is configured. Helm externalKMS separately wraps signer key-store DEKs through the shipped trstctl-kms-wrap adapter (AWS KMS, Cloud KMS, Azure Key Vault or Managed HSM, and, in the HSM signer image, PKCS#11) or an operator's own adapter instead of mounting the local signer KEK. A key store the local KEK already sealed is not re-wrapped onto the adapter; external custody starts from an empty key store. The chart runs the standard image, so a PKCS#11 HSM behind the adapter needs the HSM signer image outside the chart today. Online break-glass issue, signer-backed CA rotation with bidirectional overlap cross-certificates, and target-CA cross-signing are production-assembled when the online block is configured. Approvals come from authenticated immutable ceremony events, not caller-supplied names. Break-glass bundle reconciliation remains served at POST /api/v1/breakglass/reconcile. The credential-store key-encryption key is a local file by default; a served rotation moves it to a key under an operator wrapper file or an HSM key the signer holds (see the deployment KEK passage below). See the key-ceremony runbook, incident response, and disaster recovery.

In-memory custody of the reference-path CA keys: the served CA-hierarchy path does not use these in-process reference keys — it binds each served root/intermediate to an isolated-signer handle. The library reference manager still holds live ECDSA signing keys in locked, wipeable secret buffers (mlock + MADV_DONTDUMP) rather than as a bare unprotected key on the garbage-collected heap for the lifetime of the in-process CA; the key is reconstructed only for the instant of each signature and the transiently parsed copy is best-effort zeroized afterward (the same hardening the isolated signer uses). This narrows — but, given Go's runtime, does not eliminate — the window in which an unprotected key sits in dumpable heap; it is complemented process-wide by RLIMIT_CORE=0 / PR_SET_DUMPABLE=0.

BYOK / HSM key lifecycle: the conditional Enterprise surface supports AWS KMS, Azure Key Vault / Managed HSM, GCP Cloud KMS, PKCS#11, TPM 2.0, and YubiHSM 2. The required wiring census reports six of six backends served through the shipped control-plane plus cgo HSM signer artifact; package reachability or a registry built only by a test is not used as evidence.

With an active BYOK license and managed_keys.enabled, the control-plane process does not construct a provider or receive provider credentials. Its tagged EE attach seam installs the tenant-scoped event/projection factory and durable PostgreSQL outbox handler. A separate dispatcher delivers the lifecycle command over the authenticated signer transport. The isolated signer constructs every listed provider under the ee/ fence and stores provider ownership, operation outcome, and consumed sign-authorization nonces in its fsync-backed journal. An executing operation is not abandoned after restart: every shipped receiver finds or reconciles the same durable operation identity. AWS stamps it atomically in CreateKey; Azure and GCP use deterministic provider resource names; PKCS#11 and YubiHSM use deterministic CKA_ID; TPM enumerates persistent objects and accepts only a full SHA-256 operation tag from immutable TPM Public.AuthPolicy, probing past foreign handles without adopting or overwriting them. Revoke and zeroize read provider/device state before and after the accepted transition, so a lost response does not repeat the effect. For cloud KMS, the product reports deletion_pending while the provider may retain material; the local emulator and current QA window cannot prove the later provider retention deadline or physical erasure. A stock provider client must check that deadline, and a later provider readback is needed to close the destruction claim. Historical cloud zeroized events remain immutable and appear as deletion_pending in the API and console. The swtpm gate also pre-occupies the first deterministic handle with a same-algorithm foreign object and proves that object remains untouched.

The handlers at POST /api/v1/managed-keys, its dedicated JSON approval route, and its verify-custody, rotate, revoke, and zeroize companions return only opaque handles, public DER, algorithm, non-extractable state, and lifecycle state. Verification uses a fresh signer-generated challenge signature and records its last result and time independently of the lifecycle. This is a point-in-time proof, not a promise the provider will retain the key afterward. The default outbox retry budget can leave custody pending after the API's short synchronous wait returns 202 Accepted with an operation ID and status URL. Poll GET /api/v1/managed-keys/operations/{operation_id} or watch the console's operation panel until the durable outcome is completed or failed; refresh the exact key to read verified or unavailable. A caller whose own deadline expires before an accepted receipt arrives should read inventory and audit before retrying with the same idempotency key. New terminal failure audit events carry the key ID when one existed before the action; failed generation has no minted key ID. Historical v1 failures carry only the operation ID. Every mutation requires Idempotency-Key; immutable events build the tenant/RLS projection, and the provider call comes only from the sealed outbox. Managed-key signing additionally requires a short-lived, request-bound token from the configured content-authority command. The signer consumes its random nonce durably before calling the provider, so token replay fails both in-process and after restart. Provider credentials must be file-backed, are copied into locked byte buffers, and are wiped on shutdown; the one unavoidable PKCS#11 C_Login conversion exists only at the upstream string-only ABI edge.

Provider Production binding in the shipped HSM signer Gate substrate and lifecycle proof
AWS KMS Official AWS SDK v2 asymmetric KMS client SigV4-checking emulator; atomic operation tag, ambiguous-create recovery, disable/deletion readback
Azure Key Vault / Managed HSM Keys data-plane client with bearer-token file Managed-HSM emulator; deterministic key identity, ambiguous-create recovery, revoke/delete readback
GCP Cloud KMS Cloud KMS REST data plane with bearer-token file Deterministic-resource emulator; ambiguous-create recovery, disable/destroy readback
PKCS#11 cgo module session with operation-derived CKA_ID handles SoftHSM token; restart find-or-create plus independent pkcs11-tool state readback
TPM 2.0 google/go-tpm device or swtpm socket with tagged persistent handles swtpm plus tpm2-tools; full operation-tag readback, forced foreign-handle collision, restart and eviction
YubiHSM 2 Yubico yubihsm_pkcs11 ABI through the PKCS#11 connector Vendor-ABI emulator with deterministic CKA_ID; independent sign/revoke/deletion readback

The cloud receipts are high-fidelity protocol emulation, not a claim that this test ran in a customer's live cloud account. Likewise, SoftHSM proves the PKCS#11 ABI and swtpm proves TPM command/lifecycle behavior; an operator still validates its exact device firmware, module certificate, network policy, and cloud IAM. The LocalStack demo is not the acceptance proof's receipt source. Its pinned Community 3.8.1 image keeps KMS keys in memory; the named volume and PERSISTENCE=1 do not make them durable. A key may remain in trstctl's event-projected inventory after this emulator loses the external key. Cold recovery of the AWS KMS demo lane is blocked on a durable, faithful KMS service (for example, a licensed LocalStack edition or a real isolated cloud account). SoftHSM and swtpm can prove their own provider recovery, but cannot stand in for native AWS KMS. Static no-cgo signer builds fail closed for native PKCS#11/YubiHSM selection; use the published HSM signer artifact. Several providers run at once (managed_keys.providers): a key's provider is chosen per key by the generate request or per tenant by a managed_keys.tenant_providers pin, and every key reports its provider. The provider list is startup-static: this is ordinary Go interface injection, not a runtime crypto plugin engine or a policy-selected algorithm marketplace.

The managed-key runtime proof qualifies each released platform on a native runner of that platform. make dod-gate proves the linux/amd64 control plane and cgo HSM signer on an amd64 runner. make dod-gate-arm64 proves the linux/arm64 builds of the same two release artifacts on a native arm64 runner, against the same custody substrate and the same seven managed-key entries (tools/dodcensus/manifest-arm64.json, which a census test keeps derived from the shipped manifest); CI runs it on GitHub's ubuntu-24.04-arm runner. The arm64 target refuses a Docker host that is not arm64: an emulated runner cannot qualify either platform's artifact, even when its SoftHSM and swtpm component tests pass. Neither a component test nor a locally rebuilt signer is a substitute for that production-assembly proof. The rest of the wiring census is proved on linux/amd64; extending the arm64 profile beyond managed-key custody is what comes next.

The signer-held CA/issuing key's lifecycle is served through ceremony-gated verbs in the isolated signer: generate (root, intermediate, or a key and CSR for an external issuer), rotate and re-key, revoke (POST /api/v1/ca/authorities/{id}/revoke after a revoke_ca ceremony: the signer refuses every signature with the key, the authority stops issuing, and its certificate is revoked on a managed parent's CRL), import (an import_ca_key ceremony takes a CA's existing private key into the signer, sealed on the operator's machine to a one-time HPKE MLKEM768-P256 import key, so the control plane relays only sealed bytes), and destroy through signed retirement. An imported key may still exist where it came from; trstctl cannot prove other copies were destroyed. Each tenant's secrets KEK, its tenant key domain, has served lifecycle verbs too: generate (migrate into the domain), rotate (POST /api/v1/platform/tenant-key-domain/rotate moves the domain to a new generation under the same or another configured wrapper, re-wrapping every stored data key and the tenant's event history), and a reversible revoke as seal and unseal. Rotation re-wraps data keys, not payloads, so backups, archives, and exports taken before it still open with the previous generation. Destroying a tenant's key (crypto-shred) and importing an operator-supplied tenant KEK are what comes next. The deployment KEK, which protects every tenant that has not migrated and the deployment's sealed files, rotates too (POST /api/v1/platform/deployment-kek/rotate, only by the tenant secrets.kek_operator_tenant_id names): a new generation under a local wrapper file or a signer-held HSM key, from which every new data key is wrapped, then a leader re-wraps every stored data key (each tenant's stored ciphertext, the history of every tenant that has not migrated, and the sealed files) in bounded steps while requests are served, and a verify pass reads them all back before the previous generation is retired. The history pass copies the event stream once per tenant it re-wraps, so a deployment with many unmigrated tenants and a long history takes correspondingly long; it inherits the tenant rotation's refusal of a certificate-dependent history record, which stops the rotation for an explicit migration. The key file remains the evidence-MAC and backup-HMAC key and the signer key store's default seal. Wrapping the deployment KEK under a cloud KMS key is what comes next: the signer's KMS wrap adapter now seals the signer key store, but the deployment KEK's wrappers are still a local file or a PKCS#11 HSM key (proven against SoftHSM). Break-glass issue/rotation/cross-sign and offline-root public re-key/cross-sign import have served, ceremony-gated verbs; reconciliation remains served at POST /api/v1/breakglass/reconcile. The signer's at-rest CA key is still sealed under a local key-encryption file by default. See the key-ceremony runbook, incident response, and disaster recovery. The remaining external residual is the product NIST CMVP certificate (see compliance → FIPS), a lab process software cannot perform. The validated-module path itself is served: GET /api/v1/editions and the Platform page expose the live FIPS POST booleans, make fips-build build target, fips-capable build (GOFIPS140) CI gate, and internal/crypto boundary as the CAP-KEY-03 operator posture.

Signer UDS peer-uid is Linux-only: the signing service's Unix-domain-socket listener authenticates the connecting process's uid via SO_PEERCRED, which exists only on Linux — the supported production target (Docker/Helm). On non-Linux hosts, trstctl-signer fails closed when process hardening, locked memory, or UDS peer credentials are unavailable. Local developers can opt into the filesystem-permissions-only fallback with the explicit --allow-insecure-dev-nonlinux flag (or TRSTCTL_SIGNER_ALLOW_INSECURE_DEV_NONLINUX=true for child signer mode), but this is not a production control. Production deployments without reliable UDS peer credentials should use the signer's fail-closed mTLS transport with pinned peer certificates.

Post-quantum cryptography (issuance algorithms)

trstctl's cryptography sits behind one isolated path, and the post-quantum support lives there — ML-DSA, ML-KEM, the hybrid scheme, and SLH-DSA — all built on Cloudflare's CIRCL. What is available today:

  • ML-DSA (FIPS 204; mldsa44 / mldsa65 / mldsa87) — the NIST-standard lattice signature.
  • ML-KEM (FIPS 203; mlkem512 / 768 / 1024) — the NIST-standard key encapsulation. trstctl can generate ML-KEM keys, encapsulate to an ML-KEM public key, and decapsulate the resulting ciphertext; all three parameter sets are checked against FIPS 203 known-answer vectors. The served HTTPS and mTLS listeners prefer X25519MLKEM768 for TLS 1.3 hybrid key exchange when a peer supports it, with classical TLS 1.3 groups retained for compatibility (restored in B13-8: from July until then the listeners offered only classical groups, and stock OpenSSL offering only the hybrid group was refused). Pure ML-KEM groups are not offered: Go's TLS stack implements the hybrid group. With stock OpenSSL 4.0.3, a client offering only X25519MLKEM768 negotiates it on the HTTPS API and on the agent mTLS channel, a classical X25519 offer still connects, and a client offering only pure MLKEM768 is refused for want of a shared group.
  • SLH-DSA / SPHINCS+ (FIPS 205; SLH-DSA-SHA2-128s / 128f / 192s / 256s) — the NIST-standard stateless hash-based signature. Its security rests only on the hash function, so it is the conservative choice for long-lived roots where you want assumptions independent of the lattice schemes; the trade-off is much larger signatures. Here it is a signer and algorithm-succession primitive: trstctl issues no SLH-DSA X.509 certificate and accepts no SLH-DSA CSR (the ML-DSA certificate profile of RFC 9881 is served; the SLH-DSA profile of RFC 9909 is what comes next), so stock-client SLH-DSA interoperation is not claimed even though stock OpenSSL 4 ships the algorithm. A certificate profile may list an SLH-DSA-SHA2-* label among its allowed key algorithms, but no enrollment path accepts an SLH-DSA subject key yet.
  • A hybrid signature (HybridEd25519Dilithium3) — classical Ed25519 paired with ML-DSA, so breaking either component alone does not forge a signature.

Private key material is held in locked, zeroized buffers and parsed only for the moment of each operation, exactly like classical keys. The isolated signer can generate and use signer-held ML-DSA and SLH-DSA keys over its UDS or mTLS gRPC channel, and those keys are sealed in the signer key store so a restart does not silently rotate them. ML-KEM is not exposed as a signer key because it is encapsulation, not a signature; use it as the key-establishment primitive for protocol wiring rather than as an issuing CA key.

The served CA can mint a hybrid transition leaf: the certificate remains a normal ECDSA P-256 leaf for stock TLS clients, while a signed ML-DSA-44 + ECDSA-P256 composite binding is carried inside the certificate for PQ-aware verifiers — deployable without forcing every client to understand draft composite public keys on day one. The ACME, EST, SCEP, and CMP served enrollment paths all run through that same issuer, and a CSR carrying the hybrid proof (a classical ECDSA-P256 CSR with the composite-binding extension) issues through all four. Pure ML-DSA CSRs are narrower today: EST accepts and issues them (the PQC PKCS#10 parser sits behind EST's verifier seam, proven against a stock OpenSSL 3.5 client), ACME hands the CSR bytes to the same PQC issuer without parsing them first, and CMP p10cr carries them under a classically protected PKIMessage: the carried CSR is verified and its names are inspected through the same PQC-aware seam, so the binding between the CSR and the protection identity is evaluated on the parsed names rather than by re-parsing the CSR with the core parser (proven with a stock openssl cmp client carrying an openssl req ML-DSA-65 request). SCEP verifies and inspects the enveloped CSR through the same seam, so a pure ML-DSA PKCS#10 enrolls when the client's CMS transport identity (the self-signed certificate that signs the pkiMessage and receives the CertRep envelope) is RSA; that transport key is separate from the subject key in the CSR, and SCEP still cannot deliver its CMS-enveloped reply to a signature-only transport key, a protocol limit rather than a code gap. Stock SCEP clients (sscep) do not produce ML-DSA requests, so the SCEP proof is a Go-built pkiMessage against the served parser, not a stock client. The platform (broker) issuing CA key remains classical ECDSA-P256; a signer-backed hierarchy authority can be post-quantum: signature_algorithm ml-dsa-44, ml-dsa-65 or ml-dsa-87 on a root or intermediate ceremony mints that key in the isolated signer (HASH_NONE carries the TBS for the pure signature), the authority issues classical and pure ML-DSA leaves under RFC 9881, signs its CRLs and OCSP responses with the same key, and OpenSSL 3.5+ verifies all three; a ceremony reviewed for ecdsa-p256 cannot mint it, and a re-key stays in the predecessor's key family. Certificate-profile allowed_key_algorithms labels accept the post-quantum and hybrid names in every build — a pure label matches its exact inspected CSR algorithm, while a hybrid enrollment carries a classical subject key and stays governed by its classical family label. The selected profile still controls which algorithms an enrollment may use.

PQC certificate-key migration limits: New host-bound runs require an exact CBOM leaf fingerprint, a requested identity whose subject_key_algorithm is the run's target_algorithm (ML-DSA-44, ML-DSA-65 or ML-DSA-87; the console offers the choice and defaults to ML-DSA-65), an active profile that allows that subject key and DNS name, a signer-backed platform or managed private CA, and an enabled host connector with an enrolled agent whose installed certificate and key the agent can read back exactly. The exact-predecessor reader covers Apache, NGINX, Caddy, Elasticsearch, MySQL, PostgreSQL, RabbitMQ, Tomcat and Traefik targets (separate certificate and key files), HAProxy (the certificate+key bundle at crt_path, which must hold exactly one private key) and Postfix/Dovecot (both pairs must serve the same leaf, since one predecessor is restored to both). Java keystores (the store password is not available at adoption time), IIS (the key lives in the Windows store) and Envoy (the key sits behind SDS) still have no exact predecessor reader and are refused for host-bound certificate migration and CA re-protection. A native OpenSSL 3.5 probe and a server capable of serving ML-DSA certificates are required for signed TLS readback; classical-only clients cannot be presented as PQC verification. Under the platform CA the leaf has an ML-DSA subject key over a classical chain; a requested identity that pins an ML-DSA hierarchy authority (issuing_authority_source: private, issuing_authority_id) gets a fully post-quantum chain. Historical issuance-only runs have no retained host key or predecessor and still report rollback_unverified. A profile requiring distinct issuance approval no longer refuses this path: the start is the approved operation. The first POST /api/v1/pqc/migrations records an immutable approval request bound to the canonical selection, the target parameter set and every gated reissue's frozen profile revision and lifetime, and answers 202 with status: pending_approval, the request id and its intent digest; reviewers holding certs:issue approve it at /api/v1/approval-requests/{id}/approvals, and the same start posted again under a new Idempotency-Key consumes that single-use authority in the transaction that records the run. A changed selection or profile edit is a new request; a consumed, denied, expired or superseded one answers 409.

TLS posture rollout is a separate bound-target action. The Envoy PQC posture worker queues an agent job through the outbox to one pinned agent: the exact enrolled host assigned to a co-resident Envoy, or, for a remote Envoy that declares vantage: network, the network relay its required_agent_id names (A104). That agent stores the observed predecessor encrypted before a native Envoy change, then signs a report that binds the active configuration, its own TLS handshake with the listener (loopback on a host, from the relay's segment for a remote Envoy), the leaf fingerprint, and its identity. A network target that names no relay, or names an agent without the network role, is refused before anything is queued. Rollback requests return to that agent and restore its exact predecessor; the relay must run with --host-rollback-dir in persistent storage. The local partner-lab Envoy xDS fixture is documented in deploy/demo/lab/README.md. Previously queued pre-agent intents still refuse in-process execution, and posture changed by an older build has no host-retained predecessor to restore. These local fixtures do not qualify an external appliance or a multi-host deployment.

The core CBOM posture recognizes ML-DSA, ML-KEM, and SLH-DSA / SPHINCS+ (and hybrid labels) when it finds them in your estate. Algorithm recognition and posture classification attach in every build without a commercial license. Because all cryptography enters through one isolated path, each scheme is a contained boundary implementation (a CIRCL scheme plus known-answer tests), with no ripple into the rest of the system. The served CBOM inventory exposes this posture through GET /api/v1/cbom/assets: classical signing algorithms are mapped to ML-DSA-65/FIPS 204 targets, key-establishment findings (TLS protocols and ciphers) to ML-KEM-768/FIPS 203, deprecated DSA to SLH-DSA/FIPS 205, and migration_progress shows how much of the observed estate is already post-quantum-ready — pure post-quantum assets count as future-ready, while hybrids stay migration-required until they shed their classical component.

The core attach serves two of the former end-to-end residuals in every build: a stock OpenSSL 3.5 client creates an RFC 9881 ML-DSA-65 CSR, enrolls it through EST, and verifies the returned pure ML-DSA-65 subject leaf; the stock SPIFFE Workload API returns a two-entry response for one SPIFFE ID (the normal classical SVID and an ML-DSA-65 SVID with its matching private key). The shipped-binary CBOM TLS finding rollout against local Envoy was replayed through the host agent and outbox: stock OpenSSL observed the served TLS posture before migration, after migration, and after rollback to the exact predecessor; a signed agent readback recorded each applied state. This proves the local Envoy target, not an external appliance or every TLS client. Older runs created before the host-agent path existed remain historical evidence, not rollout proof.

Those proofs define the compatibility boundary: they do not claim every legacy TLS client or every connector understands ML-DSA. A hybrid-to-pure cutover for an existing hybrid certificate is served behind the relying-party acknowledgment gate: a migration run that targets pure ML-DSA for a certificate whose current leaf is the hybrid composite plans that reissue as a cutover and holds its host job (finding status awaiting_acknowledgment) until a quorum of signed relying-party acknowledgments for the identity's PCAS retirement policy is recorded inside the policy's window. That is the PCAS-10 rule that gates retiring a predecessor key, and a policy PCAS has already retired on that quorum still counts. A leader worker then queues the held job unchanged and records pqc.migration.cutover_released with the acknowledging relying parties and the ack-set digest; the host generates the ML-DSA key and returns only its CSR, and the finding reaches applied only on the host's signed readback. Without a policy for the identity, or without the quorum, the cutover waits and the served leaf stays hybrid. Direct pure ML-DSA enrollment is served through EST (and as the SPIFFE Workload API's second SVID), and CMP consults the same PQC parser for its carried CSR (PKIMessage protection stays classically verified). SCEP cannot by protocol. The direct identity API now accepts a caller-supplied CSR — supply subject_csr_pem on the transition to issued and trstctl signs that request rather than generating a subject key — so CSR-based enrollment, including post-quantum subject algorithms, is no longer confined to the enrollment protocols. See Lifecycle & PQC for operator flow and compatibility limits.

Key custody, stated per credential kind. Whose process created a private key and whose disk holds it is answered in one CI-checked table at Key custody, not in prose scattered across pages. The short version: every enrollment protocol, and the identity API when given a CSR, generate keys in your environment and the control plane never sees them; CA keys are created inside the isolated signer and never leave it; and every remaining path that can generate a subject key in the control plane is named there with what replaces it. The identity API without a CSR is one of those three: it still works for one release train and records an issuance.server_side_keygen event every time it runs, so you can find which of your flows still rely on it. Each tenant can turn it off with PUT /api/v1/tenant/key-generation, and a tenant registered since that switch starts with it off: an issuance that would generate the key in the control plane is then refused before anything is recorded, naming a CSR or an agent-executed target. The switch governs PKI-as-a-secret by common name (and the Vault-compatible pki/issue) the same way, naming the CSR route, and the compatibility Workload API socket's X.509-SVIDs, naming the host agent's socket; see Key custody. An identity issued from your own CSR cannot be deployed by a control-plane connector — the key that deployment needs is on your side, which is the correct consequence. Its renewal honors the same CSR, and an endpoint whose target is agent-executed renews on its host (A118, below).

And stated per credential, not only per kind. The table above is the right level for a design review and the wrong level for an audit, because an auditor is not asking about a kind — they are asking about the certificate in front of them, and a kind-level table cannot tell them whether that one took the modern path or the deprecated one. So custody is also recorded on each certificate at issuance, from what the issuing code did rather than from what the table says it should: an identity issued from your CSR records that the control plane never held the key; one issued through the deprecated server-keygen path records that it did. The certificate API returns it and the console shows it on the certificate.

Two honest gaps. Certificates issued before this shipped have no custody recorded, and so does every certificate found by discovery — trstctl did not witness their issuance and has no basis for a claim about it. Both read as not recorded, which is deliberately a different value from any custody claim rather than a default that quietly resembles the good one. And what is recorded is the control plane's own account of what it did; it is evidence, not an attestation. A certificate issued through ACME device-attest-01 can also carry a claim the hardware signed: when the device's TPM certifies its key as one it generated and will not release (sensitiveDataOrigin, fixedTPM, fixedParent), the certificate records custody device / device_bound / non_exportable and keeps the attestation and its binding, and anyone can re-check it offline against TPM roots they choose (trstctl-cli certificates key-attestation then verify-key-attestation; see Key custody). It is proven against fixture attestations and swtpm's TPM2_Certify; proof on a hardware TPM needs one. Host agents that generate endpoint keys in their own TPM, so agent-executed targets carry the same claim, are what comes next.

Recording it per certificate also made a fourth control-plane keygen path visible that the kind-level table had not named: automated renewal of an identity with no CSR, on a target the control plane deploys to, builds the successor's CSR itself, so that successor reads control_plane. The key is not lost (A119): it is retained in a tenant-bound sealed recovery record (key_storage sealed_store) and travels sealed in the connector's deploy intent, so the endpoint serves the renewed certificate; an identity with no deployment target has no endpoint to serve it, and its key stays sealed. Host-executed renewal (A118, B2) removes the control-plane key for an agent-executed target: the renew-before-expiry pass and rotate queue endpoint.renew before anything is minted, the host agent generates the key, sends only a CSR, installs the pair, verifies the endpoint, and signs the custody receipt (key_origin host_agent, the agent named as generator). The batch-12 lab replay ran five such renewals on its host agent, each recorded as host_agent/file/lab-host-1. Key custody states this in full; the count of successors reading control_plane measures what still relies on the deprecated fallback.

Served TLS floor and stock enrollment clients

The control plane's HTTPS listener negotiates TLS 1.3 only by default. Stock device enrollment clients that cap at TLS 1.2 (cisco libest estclient, some SCEP agents) cannot complete the handshake against that default. The libest conformance job proves both sides of that policy against the control plane's own listener code (mtls.ServerCert.ServeHTTPS, the path serveControlPlane runs): with the TLS 1.3-only default the pinned estclient fails its handshake and writes no certificate, and with TRSTCTL_SERVER_TLS_MIN_VERSION=1.2 (AEAD suites only), the documented opt-in for such fleets, the same client enrolls. The earlier proof against a Go test server that silently allowed TLS 1.2 is retired. EST /simplereenroll on the main listener still authenticates with the bootstrap token; with TRSTCTL_PROTOCOLS_EST_MTLS_ADDR set, a second listener requires a client certificate from the enrollment CA and authenticates renewals by that certificate, binding the CSR to its identifiers, so devices keep no token for renewals (DP2-037 closed; proven with a Go client and with libest estclient -r). Found by the cold design-partner run 20260906t143500z (DP2-036, DP2-037).

ACME certificate profiles

Both directions of draft-ietf-acme-profiles are served. trstctl's ACME client (the letsencrypt external CA type) requests the profile named by the authority's acme_profile configuration on every newOrder, and only when the authority's directory advertises it in meta.profiles: an unadvertised or misspelled name fails before an order exists. With no acme_profile the authority applies its own default, as before. trstctl's ACME server advertises its offered profiles in meta.profiles (the mount's default plus every credential-pinned profile), accepts a client's profile member when it names an advertised profile, and refuses an unknown one with invalidProfile; a protocols.acme_eab.keys[].profile pins the profile every order under that credential is issued with, and a client asking for a different one is refused. The selected profile is recorded on the order and the served issuance signs under it. A live profile request against Let's Encrypt's production directory is not part of local evidence.

Upstream domain validation

trstctl is an ACME server and an ACME client, and until now only one of those could do DNS-01.

What was wrong. As a client to a public CA, the driver looked only for http-01 and errored if the authority offered none, so DNS-01 was never attempted and wildcards were impossible. Worse, the production constructor wired a solver whose present and cleanup did nothing. The upstream path therefore worked only against orders the authority had already authorized out of band. Every test passed, because the fixture returned orders as pre-authorized — the same shape of defect as a feature whose executor is never asked for work.

What is served now. The client negotiates challenge type from what the authority actually offers, and solves DNS-01 through the same publish path, the same providers and the same credentials the server direction already uses. A missing solver fails closed with a named reason rather than silently validating nothing. The record is retracted on every exit path, including when the context expired — which is exactly when a validation token is most likely to be left live in public DNS.

Consent is explicit, and it is two decisions, not one. The authority must be configured with upstream_dns01, and each DNS-01 provider config must set allow_upstream_dv. Both default to false, including on configs that already existed. Credentials an operator supplied so that trstctl could verify a challenge somebody else published are not consent for trstctl to publish into that zone whenever an external authority asks — and as validation-reuse windows compress, that publishing becomes frequent and unattended. A migration cannot grant that permission; a content harness asserts no existing config comes out consenting.

CAA is checked against the right issuer. The server-side check uses the config's own CAA identifier, which is correct when trstctl is the issuer. Upstream the certificate comes from someone else, so the check uses the external CA's identifier, and refuses to run at all if none is configured — a CAA check against an empty issuer authorizes everything while appearing to check. Configuration validation requires caa_issuer_domain whenever upstream_dns01 is on, so that state is unreachable rather than merely handled. The identifier is bound per authority: two configured ACME CAs do not share one, because a CAA check that authorizes the wrong CA passes while being wrong, which is harder to notice than one that authorizes everyone.

Reused authorizations are recorded, and served. An authority that already considers an identifier authorized issues without a challenge. That is normal and it is also the thing worth watching: automating validation removes the human from the cycle, and with them the human who used to notice when validation broke. GET /api/v1/acme/dns-01/upstream-authorizations and the console's Upstream authorization freshness panel report, per identifier and per authority, when control was last actually proved — not when a certificate was last issued. Those two diverge silently, and the gap is the warning. An identifier that has never been validated by this deployment is called out by name: every issuance for it so far rode a reuse this install did not earn and cannot repeat, and when the window closes they fail together rather than one at a time. A reuse never overwrites the last real validation date; that is asserted by a test, because the convenient single "last seen" column would erase the only signal here.

What is not served. trstctl does not wait for DNS propagation before telling the authority to validate. The record is published through the provider's API and the challenge is accepted as soon as that call returns, so a zone whose nameservers are slow to converge can have its authorization marked invalid and the order retried rather than waiting. The operator-run preflight (POST /api/v1/acme/dns-01/preflight) checks propagation for a domain, but it evaluates TXT values the caller supplies rather than querying DNS itself, so it cannot be reused as an automatic gate. Closing this needs authoritative-nameserver TXT verification that does not exist in the tree yet; until it does, a slow zone costs a retry, not a wrong answer.

Also not served: http-01 upstream — it would require an inbound listener on the validated host, which this architecture does not have and will not grow. The challenge census says so rather than claiming a type that would be selected and then fail. The publish is bounded, but by whichever deadline is tighter: the caller's context if it carries one, and otherwise the DNS-01 automation's own 30-second outbox wait. There is no separate, configurable propagation budget, and the 30 seconds is a floor for callers that set no deadline rather than a cap that always applies.

Issued, delivered, verified

Three claims about the same certificate, with three different lifetimes, and only the third is what an operator actually wanted:

  • Issued — a CA produced it. Recorded in the certificate inventory.
  • Delivered — a connector applied it to a target. Recorded on a connector.deploy delivery receipt. This is trstctl's own account of what it did, and it is where every deployment surface stopped before D3.
  • Verified — a TLS handshake observed the endpoint serving it. Recorded on a second delivery receipt and, separately, as current endpoint state.

Delivered and verified are counted separately and never summed. A renewal can succeed at the CA, be delivered by a connector, and never reach the listener, with every delivery record staying truthfully green. A health surface that counted deliveries would be reporting intentions; GET /api/v1/platform/system counts by verified state for exactly that reason, and verified_percent is of DELIVERED targets — "of what we have deployed, how much is confirmed live".

Verification writes a second receipt rather than editing the delivery one. Both facts belong in the evidence chain and they have different lifetimes: "a connector applied the credential" is true forever once it happens, and "the endpoint was serving it" is true of the moment it was observed. A certificate verified in June whose listener silently reverted in August shows a June receipt still reading verified and an endpoint state reading diverged. Overwriting the first would destroy the record that delivery succeeded, which is what an operator needs to tell a pipeline problem from a listener problem.

unverified is the honest middle. A delivered target nobody has probed is neither a failure nor a pass — it means nothing has looked. Folding it into either direction would be an overclaim, and on a fresh install every target sits here, which is the correct starting picture rather than a discouraging one.

Renewal windows, canaries and SLOs

Maintenance windows defer, they never drop. A renewal deploys to a listener and reloads a service, and there are hours in every organization's week when nobody wants that unattended. Before this the only control was switching renewal off, which trades an outage risk for an expiry risk. A window that closes now holds the sweep and records why, naming when it reopens — because a change freeze that quietly stopped renewals looks exactly like a scheduler working correctly, right up until certificates expire, and expiry is the more expensive failure by a wide margin.

An empty window list means unrestricted, never "never": an operator who configured no windows has not asked for a freeze, and defaulting to closed would turn an upgrade into a fleet-wide expiry event. A malformed window refuses to start rather than being ignored — an operator who wrote a freeze this could not read would believe production was protected while the scheduler renewed through it. Windows are expressed in named timezones so they follow daylight saving the way the person who wrote them expects, and a window whose end precedes its start wraps midnight, which is the shape most operators actually want.

Fleet gates are computed, not asserted. Until D2/D3 nothing re-read an endpoint, so every gate trstctl filled in itself was not_evaluated — correctly, because a verdict nobody computed is not a pass. The replacement-deployment gate is now derived from verification receipts under two rules an operator relies on mid-incident: failure dominates (one replacement serving the wrong certificate fails the gate however many others passed) and absence beats success (one replacement nobody probed keeps it unevaluated). A gate an operator asserted is never overwritten by a computed one — they may have inspected something this control plane cannot see. Gates evaluate at read time, so a fleet that has since diverged stops showing a pass.

The other two gates are computed on every read as well, and appended to a run that never named them. Graph enumeration re-reads the live store as its completeness oracle: every identity that still holds an active certificate signed by the compromised CA must be in the run's affected set, so a leaf the plan missed, or one issued under that CA after the plan, fails the gate by name (graph_enumeration_missing_identity_ids); an oracle that cannot read keeps it unevaluated. Revocation publication is R1's freshness signal narrowed to the compromised issuer: a stale or unreachable CRL/OCSP endpoint fails the gate, a fresh artifact whose thisUpdate predates the run's revocations keeps it unevaluated (it cannot carry them yet), and the gate passes only when a fresh or expiring observation postdates the run's last change (revocation_publication reports the counts). No observation for the issuer means no verdict. The watermark is the run's last state change, not each predecessor's exact revocation instant, which is the one approximation here.

Canary-first halts propagation. Starting a run persists its cursor and publishes exactly one internal outbox command for batch one. The bounded worker publishes replacement work only for that cursor, waits without spending its retry budget, and accepts a verdict only when the connector verification row is paired with an agent-job receipt whose signature was accepted. A failure stores the explicit halt reason and leaves later batches absent from the outbox. halted is distinct from failed: a halted batch was never attempted. Resume reuses the same cursor and deterministic replacement ids, so crash recovery and redelivery converge instead of reissuing or publishing a later batch twice.

The SLO counts terminal runs only. A renewal still executing is neither a success nor a failure, and forcing it into either would move the number for reasons unrelated to reliability. A window in which nothing was due reports 100%, not 0% — an estate with no renewals pending is not in breach, and paging on the absence of work is how a team learns to ignore an SLO. Error-budget burn is clamped at zero, because a figure like -340% is not more actionable than none-remaining and the raw counts sit beside it.

What is not served. The fleet state machine does not treat an unsigned control-plane assertion as endpoint proof. A deployment executed only by a control-plane connector therefore cannot advance this signed-receipt gate; it remains waiting_verification until a bound agent reports the endpoint result.

Dated lifetime-reduction readiness

GET /api/v1/certificates/validity-readiness (and trstctl certificates validity-readiness, the Lifetime-reduction readiness panel on the Certificates renewal tab) evaluates the CA/Browser Forum TLS Baseline Requirements 2.3.1 (published 2026-10-04), section 6.3.2: 200 days for certificates issued from 2026-03-15, 100 from 2027-03-15, 47 from 2029-03-15. Every active certificate gets a verdict under each step (renews, at_risk or manual, with reasons), and every public TLS profile whose max_validity exceeds a step's maximum, or that has none, is flagged. The export names the BR version in its file name and on every row. What it decides from, and what it does not see:

  • Publicly trusted means a reserved policy OID. A certificate counts as public TLS when it permits TLS server authentication and asserts a CA/Browser Forum reserved certificate policy identifier (2.23.140.1.2.1–3 or 2.23.140.1.1). trstctl's own CA is never publicly trusted, so a profile is public TLS only when it says so (public_tls: true). Other certificates are still evaluated, for planning, with scope saying the maximum does not bind them.
  • Scheduled means the scheduler's own inventory. A certificate is scheduled when it is the served certificate of a renewable identity, or its issuance evidence binds it to one — the same set the automation plan and the scheduler use. Its verdict follows that identity's delivery evidence (a delivery whose served certificate was read back is verified), its holds (failed renewal, an operator's pause, a destination conflict, a profile renewal cannot resolve) and whether each renewal waits for approval.
  • Client renewals are not observed. Certificates issued over ACME, EST, SCEP, CMP, Kubernetes in-cluster issuance, attested workloads or the agent broker are renewed by their client; trstctl does not see that client's renewal lead, so they read at_risk with client_renewal_unobserved. Measuring a client's observed renewal lead from its issuance history is what comes next.
  • Flagged, not refused. Issuance does not yet hold a public TLS profile to the maximum for the issuance date; the report flags it and a publicly trusted CA refuses or shortens the request. Holding issuance to the dated maximum is what comes next.
  • Bounded. The report covers the 5,000 active certificates that expire first and says when there are more; ephemeral certificates, issued per use and never renewed, are left out as the expiry views leave them out.

Cryptographic risk report

GET /api/v1/reports/crypto-risk (and the crypto_risk scheduled report, trstctl-cli reports crypto-risk) lists each weak, distrusted or quantum-vulnerable key with its dependents, its NIST IR 8547 deadline and its plan, and what changed since the previous scheduled report. What it decides from, and what it does not see:

  • IR 8547 is a draft. The deadlines come from the initial public draft of November 2024 (Tables 2 and 4), and the report cites it as such; a final version may move them.
  • CNSA 2.0 is not encoded. The report does not state the CNSA 2.0 milestones for national security systems, and says so.
  • Only what has been seen. It covers the certificates trstctl holds or has discovered and the crypto assets discovery has inventoried. A key with no recorded dependents may be one nothing has scanned around.
  • Name matches are matches by name. A certificate trstctl knows only by its recorded issuer name is matched to a distrusted CA by that name.
  • Bounded. It rates up to 50,000 certificates and reads up to 100,000 rated discovery findings, and says when it read less; the on-demand report lists up to 1,000 keys and the scheduled one up to 20,000, while its summary and changes count every key.
  • A certificate without bytes is sized from its recorded algorithm name (RSA-2048, ECDSA P-256); when that names no size, the 112-bit rule applies and the report says it assumed it.

Credential assurance records

GET /api/v1/certificates/{id}/assurance-record signs what trstctl already holds about one certificate; it collects nothing new. Read it with these limits:

  • Independent observation is what scans and endpoint checks saw. Discovery findings and endpoint observations (local or relay) are independent of trstctl's record of its own actions; a connector's delivery receipt is not, so a delivery counts as observed only when an endpoint was then seen serving the certificate. External observers' reports (see External observers) are listed apart and count the same way when they match.
  • Endpoint observations are the latest per listener and vantage. Earlier observations of the same listener are not retained, so the record cites the newest one; the verification receipts carry the history of each delivery's check.
  • A profile appears only when the issuing command bound one. Scheduler renewals that bound no profile, imported and discovered certificates, and history older than reviewed profile bindings have no profile entry, and the rendering says so.
  • Bounded. Each evidence list holds up to 200 entries and the audit references up to 100 (truncated names any list that hit its bound); an evidence pack carries up to 50 records, those whose lifecycle changed in its window.
  • Verify against keys you pinned. The response's public_jwks is a convenience; trstctl certificates verify-assurance-record requires a JWK set you obtained separately and never trusts the file's own copy.

Outcome pages

The three outcome pages say how strongly each step is proved and name the test:

  • Kept by hand. The proof column is maintained with the features it describes; no guard checks it against the test suite, so a page can trail a change until it is updated.
  • Starting screens, not guided steps. The console home's Outcomes section opens each workflow's first screen; it does not walk the steps in order.

Evaluation kit

trstctl evaluate runs one certificate journey against an endpoint you choose and writes a signed evidence pack. Read the pack with these limits:

  • Discovery is a handshake, not a sweep. The control plane connects to the endpoint named for the run (a CBOM TLS scan); finding endpoints nobody listed is what a network relay's discovery scan does, and the journey does not run one.
  • One vantage. Verification and the renewal check are the command's own TLS handshakes from the machine it ran on.
  • OCSP, not CRL. Revocation is checked over OCSP; CRL publication is asynchronous and not checked. The issuing CA certificate for the OCSP check comes from the issuance result or the chain the endpoint served; an endpoint that serves only its leaf, with a host-generated key, cannot be checked.
  • Keys from the same server. The run checks the audit export with the keys the server publishes; trstctl evaluate verify checks the pack and its export again with a key set obtained separately.
  • One run, and it changes the endpoint. The journey creates an identity, deploys two certificates and revokes the first. It is not a load, failover or long-running test, and belongs on an endpoint under evaluation.

Customer assurance reports

GET /api/v1/reports/assurance and the customer_assurance scheduled report put a tenant's credential assurance into plain sentences in its white-label brand. What the report decides from, and what it does not establish:

  • Counts are exact; lists are bounded. Each list names at most 10 items (unowned and soon-expiring certificates, failures, incidents); the count beside it is exact.
  • Endpoint results are the latest observation. The report counts each endpoint's latest check from each vantage, not every check made in the period.
  • Only failures this deployment recorded. Failed renewals are renewal runs that ended failed; failed deliveries are connector deliveries that failed or failed their verification. Work done outside trstctl is not seen.
  • Changed credentials are classified up to 50. Whether a changed credential's deployment was independently observed comes from its assurance record, read for the 50 most recently changed credentials; the report says when there were more.
  • Large incident evidence is named, not carried. An evidence bundle over 64 KiB appears as its SHA-256 digest; read it from the incident.
  • Only a brand that embeds safely. A logo is embedded only as a PNG, JPEG, GIF or WebP data URI of at most 256 KiB, never SVG, and the accent only as #rgb or #rrggbb; anything else is left out and the report says so. The HTML rendering never names trstctl; machine-readable format identifiers keep their schema names.
  • The comparison is with the previous signed report. A scheduled run compares with its schedule's previous completed run, read back from the archive and verified first; an on-demand report compares with the tenant's newest completed one, whatever period that covered. With no archive or no earlier run there is nothing to compare, and the report says so.
  • Verify against the key you pinned. trstctl-cli compliance report-runs verify needs the signing key obtained separately and never trusts the key inside the file. A run signed before a key rotation verifies under the key that signed it.
  • The Provider reads reports on its API. The Provider console does not show a customer's reports yet; /provider/v1/tenants/{id}/assurance-reports lists and downloads them. A console view is what comes next.

Endpoint verification

Every other record in this product reports what trstctl DID. An outbox row delivered, a connector returned success, a certificate was issued — all of them can be true at once while the listener serves something else entirely, because a connector's reload is one exec call inside its own Deploy method and nothing downstream observes whether it took effect.

The failure this exists to catch. A renewal succeeds at the CA. The connector writes the file. The reload fails, or the service ignores it. Every delivery receipt stays green, the inventory correctly describes the new certificate, and clients keep getting the old one until it expires. Inventory-based expiry alerting cannot see this, because the inventory is right but does not describe what is being served.

Two vantages, and the difference is not redundancy. The host agent's check runs inside connector.deploy, in the only window where it can: after the reload, before the redeemed material is destroyed. It verifies against the exact bytes it deployed rather than a description of them. The relay's check is a separate endpoint.verify job — network vantage, no credential redeemed — and it is the only witness for an appliance, because nothing runs on an F5. A local pass means the box thinks it is fine; only a relay pass means a client could get it. The two are stored as separate rows and never merged.

A verdict never claims more than it checked. Each record carries whether the name set and chain were compared in addition to the fingerprint. An expectation that supplied no SAN set yields a verification that does not claim to have checked names.

Unreachable is not a pass. It is stored as its own state, with no mismatch class, no comparison flags and no observed fingerprint — enforced in three places (the agent's transcript validation, the projector's decode, and a database CHECK constraint) because an endpoint nobody could connect to, recorded as verified, would be a worse false assurance than the blindness this replaces.

last_good_at is never erased by a failure. The gap between it and last_checked_at is how long an endpoint has been failing, and losing it on the first failure would destroy the only measure of the outage's age. An endpoint that has never once been observed serving what it should reads as never verified, which is a stronger statement than "not recently" and renders as one.

Re-verification is a loop, not an event. A post-deploy check proves the reload took effect at that moment; it says nothing about the weeks afterward, and a listener can start serving the wrong certificate long after a deploy — a failover to a node that never got the file, a config reload elsewhere, a restored backup. Sweeps run hourly by default (EndpointVerificationInterval), batched at 50 endpoints per job so one tenant cannot hold a relay indefinitely. Each sweep's expectation comes from the control plane's own record of what should be there, never from the last observation — re-probing against what was last seen would re-verify a divergence as correct on the next sweep and silence its own alarm. When a queued relay sweep returns after a host deployment has changed that endpoint's expected fingerprint or address, its signed transcript and job receipt remain historical evidence. The obsolete comparison does not replace the current endpoint row or send a divergence alert. A fresh network check is still needed to prove the replacement from the relay's vantage; the next scheduled sweep obtains the latest local deployment expectation. Until it does, the relay row reads not_checked for the current identity, the summary counts it separately, and the dashboard says it is awaiting a relay recheck.

Divergence alerts route by severity, not by finding score. A served-identity divergence is critical; an unreachable endpoint is warning, because a probe that could not connect may be a firewall or a maintenance window and paging at critical for that teaches people to ignore the channel. The alert kind is distinct from credential.drift deliberately: drift drives a file-repair workflow keyed on a filesystem path, and a served-identity divergence usually has a perfectly correct file that a process never reloaded.

Automatic rollback is opt-in, per target, default off. Setting auto_rollback_on_verify_failure on a deployment target closes the loop: a verify_failed deploy queues D4's executed re-bind to the predecessor. It fires only on verify_failed and never on plain failure — a deploy that failed did not change the target, so rolling it back would undo something that was never done. Absent flag means off; a target whose config predates this feature never starts changing itself because a new version shipped. The four appliance re-bind families and all 14 host-local restore families can execute it. Host work is pinned to the agent that retained the predecessor; a first deployment still has nothing to roll back to.

Manual and automatic rollback share a tenant-scoped inventory check: the predecessor must be known and unrevoked, and a bound identity must not be revoked or retired. The control plane repeats that check before handing out queued work; the updated agent requests fresh authorization before opening its predecessor store or redeeming an appliance credential. Upgrade both components to obtain all three checks. A control-plane refusal is recorded as rollback_refused, without claiming agent execution. An operator can cancel a target's in-flight connector work (POST /api/v1/connectors/targets/{id}/cancel): queued commands are withdrawn, agent-leased commands are recalled so the agent is refused at its next check-in and its late result is retained rather than applied, and a command the control-plane worker is executing inside a connector call is reported and left alone rather than interrupted mid-write. Cancellation cannot undo an effect that already landed. Compromised material already installed on a host target is removed by POST /api/v1/connectors/targets/{id}/purge (endpoint.purge): the exact enrolled agent proves the installed leaf, overwrites and removes the certificate and key files the target names, wipes the rollback ledger of that fingerprint and runs the bound stop action; only a purged receipt claims that. A purged receipt drops that target from the rogue listing's installed_on at once, and an identity whose leaf is then served by no target moves from deployed back to issued (the certificate exists; nothing serves it); an identity still served elsewhere keeps its state. Retiring or revoking the identity is the operator's separate next step. Appliance and cloud targets, java-keystore, IIS and Envoy are not covered: their material is not an addressable file on an enrolled host.

Configuring it. Local post-deploy verification runs when a deployment target carries verify_address (and optionally verify_server_name) in its config — for example {"cert_path": "/etc/nginx/server.crt", "verify_address": "api.example.test:443"}. The address is never derived or guessed: on an appliance target endpoint is the MANAGEMENT API, and the connector's target string is a routing label, so an F5's management plane and the virtual server it fronts are different sockets. trstctl suggests one from evidence instead: GET /api/v1/connectors/targets/{id}/verification-suggestions (CLI connector target verification-suggestions, the destination's Verification address panel in the console) lists the host:port addresses where a discovery scan saw a certificate trstctl delivered to that target (a delivered or verified deploy, not purged since), each with the findings that saw it. Nothing changes until an operator confirms one with POST .../verification-address (CLI connector target confirm-verification-address), which refuses an address no finding suggests and a target that changed since the suggestions were read; the confirmed address is written to the target as a new revision and later deployments verify the listener there. Any other address is set by updating the target.

What is not served. Verification only covers endpoints with a listener address an operator gave or confirmed; an endpoint with no address is never verified and never claims to be. Suggestions come only from TLS discovery findings, so a listener no scan has seen is not suggested; addresses read from connector bindings (a load balancer's virtual server, a cloud listener) are what comes next.

External observers

POST /api/v1/observations takes what an external observer saw at an endpoint and compares it with what trstctl expects there; POST /api/v1/observers registers observers, which also receive rotation and rollback events. What it relies on:

  • The expectation is trstctl's own endpoint verification. An observation of an address no trstctl verification checks is recorded as unexpected, with nothing to compare. Report endpoints by the host:port trstctl verifies.
  • The observer's report is taken as reported. trstctl does not probe the endpoint to confirm it; the observer's token (scoped to discovery:write) is what authenticates the report, and the record names the observer.
  • Events are at least once. An observer drops a redelivered event by its id. A projection rebuild does not resend events older than a day.
  • What comes next. Observer management is served through the API and CLI; console pages for it, and probectl's own client for this contract, come next.

GET /api/v1/issuers/capabilities and trstctl issuers capabilities serve one row per authority kind: discover, issue, renew, revoke, whether trstctl can satisfy its domain validation unattended, where the private key is generated, and what the authority validates before issuing. The console shows revocation and domain validation beside each configured issuer.

Unattended DV is checked against the source, in both directions. It is true only where the issuer's package wires a challenge solver into the ACME driver — one authority kind today — and an authority that reads false must say why, because "there is no challenge to solve" (an internal CA) and "a human completes DCV in the vendor's console" (public OV/EV) are opposite operational situations. It is a property of the build; whether a given authority is actually configured for it is the upstream_dns01 flag on that authority.

Revoke is the field that matters, and it is deliberately conservative. It is true only where this build ships an implementation that contacts the authority and is proven against that authority's protocol by a test:

Issuer Revoke How
letsencrypt yes RFC 8555 §7.6 revokeCert, JWS-signed with the account key
vaultpki yes POST {mount}/revoke by serial number
ejbca yes REST PUT /certificate/{issuer_dn}/{serial}/revoke

Everything else reports false with a note saying where to revoke instead. DigiCert, Sectigo, Venafi, AWS Private CA, Google CAS and step-ca all document a revocation API; trstctl does not drive any of them, and a documented endpoint nobody has implemented is a plan rather than a capability. AD CS revocation runs through the CA's own management interface, and Azure Key Vault disables certificates rather than revoking them.

No silent no-ops. Exact-certificate revocation against an authority that cannot revoke through this build returns a failed item with the unsupported reason and queues no upstream effect. The X.509 identity-transition preview marks complete revocation not ready and names the affected certificate. For a non-compromise or non-serving identity, a privileged operator can still record the local lifecycle transition to stop automation; that state does not revoke a leaf at the external CA. For a serving X.509 keyCompromise, ordinary lifecycle execution refuses before an event or outbox intent and directs the operator to the reviewed CA-and-host command. Without an exact managed host, the operator must revoke the exact certificate at its issuing CA and independently stop the workload. The product does not infer host containment from the revoked state. The revocation.publish command attributes every bounded leaf to the authority that signed it from retained issuance evidence: platform-CA leaves publish on the tenant CRL, leaves from a private CA hierarchy authority (host-executed identities whose issuing_authority_source is private) are revoked in that authority's issued-certificate ledger and CRL, external-CA leaves at that CA, and a superseded predecessor whose managed CA key was destroyed by signed retirement is skipped rather than blocking the live successor's publication. This is the specific failure the epic exists to remove: an operator revoking a compromised key and being told it worked, while the authority still considers the certificate valid, is worse off than one told plainly that trstctl cannot do it — the second sends them to the vendor console, the first sends them home.

ACME is worth one further note. The protocol identifies the certificate to revoke by its DER, not by serial, so trstctl cannot revoke an ACME certificate it does not hold a copy of. A request carrying only a serial is refused with the reason rather than sent as something the protocol cannot express.

The local Pebble 2.10 partner-lab issuer is also a limited revocation witness. Its issued leaf in the U5 QA run had no CRL Distribution Point and no OCSP URI. The issuer accepted trstctl's authenticated ACME revokeCert requests for the exact retained certificates, but a stock relying party cannot discover a signed CRL or OCSP responder from that leaf and independently reject it on revocation grounds. The lab can prove upstream acceptance and can use the host agent plus a fresh TLS probe to prove the compromised leaf is no longer served. It cannot claim stock client revocation enforcement for that Pebble certificate. The same upstream ACME path is proven with an independent issuer that does publish revocation status (A114): the lab's step-ca 0.28.3, an ACME CA validating dns-01 through the lab's own DNS-01 webhook, regenerating its CRL on revoke and naming it in every leaf. trstctl ordered a certificate from it, revoked it through its ACME revokeCert, and stock OpenSSL then refused the leaf by step-ca's CRL (openssl verify -crl_check: certificate revoked; a TLS handshake with CRL checking fails the same way), while a client that does not check revocation still connects. That proves the trstctl side for any upstream ACME CA that publishes status; each issuer's own publication is its own to qualify.

After exact host containment, the endpoint replacement preview reads the target's latest host recovery check: a read-only endpoint.recovery_check job the enrolled host answers in a signed receipt with the SHA-256 of its pinned recovery action's executable and command line and the listener state one local TLS probe saw. The preview reads ready, and the replacement can be created, only when a check requested after the stop shows a recovery action the target's connector runs while deploying and a listener that is not serving the contained leaf. The check reports what the host profile pins, not what the action does: whether it re-enables only that listener and refuses every revoked predecessor is still the host operator's to test. The agent's verified deployment receipt and a fresh stock TLS readback are the completion evidence; the preview and accepted binding alone are insufficient. The NGINX procedure is documented in docs/features/deployment-connectors.md.

A CI guard parses every issuer package and fails the build if the matrix and the code disagree in either direction — a claimed capability with no implementation, or an implementation the matrix does not advertise.

Coverage, provenance and blind spots

A certificate count is not an inventory, and the difference is the whole of this section. "We found 4,312 certificates" answers how many discovery happened to turn up. An auditor asks a different question — how much of the estate did you look at, and when — and a system built only from findings cannot answer it, because what it never looked at leaves no trace in what it found.

Coverage is measured against a declaration. An operator names the segments they own, with the ranges, an exclusion flag and reason where a segment is deliberately out of scope, and a staleness window that is theirs to set — a DMZ and a lab do not deserve the same answer. Coverage is then the share of declared, non-excluded segments swept inside their own window. Excluded segments are removed from both halves rather than counted as covered; a number that rose when somebody excluded something would reward exactly the wrong behavior.

The consequence is worth stating plainly: an estate with nothing declared reports no coverage, not full coverage. That reads as unhelpful on day one and is the only defensible answer — the alternative is a system that declares itself complete because nobody told it what it was missing.

Every certificate carries provenance. Which source last observed it, of what kind, and when. last_seen_at is deliberately distinct from created_at: the first says something confirmed the certificate still exists, the second only says trstctl once recorded it. Conflating them makes a stale inventory look freshly verified. A certificate this control plane issued that nothing has since scanned has no observation at all, and is reported that way — it is evidence of an issuance, not of a deployment.

The blind spots are named, not implied. The register lists segments never swept, segments outside their own staleness window, segments declared out of scope with the reason, asset classes no configured source can ever see, the count of inventory rows with no observation behind them, and the addresses trstctl's own sources saw outside every declared segment. Each carries the action that closes it, or says plainly that nothing does.

Addresses outside the declared segments. GET /api/v1/discovery/coverage/candidates (trstctl-cli discovery coverage candidates, and the coverage panel in the console) lists the network addresses recorded in the last 30 days — certificates' deployment locations, endpoints trstctl verified, endpoints external observers reported, and destinations' verification addresses — that no declared segment covers, by the same rule a sweep's targets are held to (an exact hostname or address, a CIDR, or an explicit range; an excluded segment counts as declared). Nothing is resolved, and file paths and other location labels are not addresses. Each is a candidate for a declaration, which the console can make in one step. The list is never complete: an address none of these sources has recorded cannot appear, so a network nobody has touched still has to be declared by the operator.

What this does not do: it does not verify that a certificate found at an address is the certificate that address serves to a real client — that is verification, and it is a separate claim.

Kubernetes deployment

The control plane ships a production-shaped Helm chart (deploy/helm/trstctl): the API/UI with the signing service isolated (its own locked-down, network-unreachable sidecar), external PostgreSQL and NATS as the default, a default-deny NetworkPolicy, and TLS.

  • Kubernetes Operator scope: a focused CRD-driven operator ships today. The trstctl-operator binary (it rides inside the same multi-binary control-plane image and is run by deploy/operator/operator.yaml via an entrypoint override) reconciles TrstctlControlPlane custom resources into a managed control-plane Deployment (replicas, image, the PostgreSQL DSN Secret reference, NATS URL and replica knobs, sidecar-signer socket/volumes and managed-key provider enablement; its manifest documents the leader-elect and coordination.k8s.io controls that keep the reconcile path bounded) and the network surface around it, the way the Helm chart renders them:

    • a Service on the API/UI port (spec.service: type, port, annotations);
    • a default-deny NetworkPolicy (spec.networkPolicy, on unless disabled) that admits the API port only from the named ingress controller pods, the namespaces you list, or the same namespace when you opt in, and lets the control plane reach DNS, PostgreSQL and NATS (optionally only within the CIDRs you list) and its signer;
    • an optional Ingress (spec.ingress: class, host, TLS Secret, annotations) whose backend is the HTTPS port;
    • with signerMode: isolated, the signer as its own pod with its own Service on 9443 and a NetworkPolicy that admits only this control plane, over the mutually pinned mTLS channel. Unless the resource names the two Secrets you provide, the operator generates a cross-pinned pair into <name>-signer-mtls and <name>-cp-signer-mtls, mints a new pair a month before it expires or when the server name changes, and rolls both pods onto it through a pin fingerprint on their templates.

    Signer authorization follows the chart: spec.signer.authTokenCommand (required with production NATS) or, for a sidecar evaluation, spec.signer.allowCoResidentAuthorizer. Combinations the control plane would refuse at startup set the resource Blocked with the reason instead of rolling out pods that cannot start. Every object the operator creates carries its managed-by and owner labels and an owner reference, so deleting the resource removes them; it deletes only objects carrying both labels, and refuses to take over an object of the same name that it does not own. Status reports each surface (create, update, delete, none or absent) with phase Ready, Reconciling or Blocked and a message. The operator also reconciles TrstctlSecretSync custom resources into Kubernetes Secrets plus reload annotations, and TrstctlSecretInjection custom resources into no-code sidecar/env/file injection patches for opted-in Deployment, StatefulSet, and DaemonSet workloads. For SecretSync resources it resolves values through the served secret-store API, writes Secret.data, records status.contentHash, and patches pod-template annotations instead of deleting pods. For SecretInjection resources it reads source Secret metadata and key names, rejects missing requested keys before patching, then patches the shipped trstctl-agent --secret-inject sidecar, app mounts, and optional env references, and records status.injectedWorkloads. It is a real, level-based reconcile loop (poll, diff, converge), not a stub; it speaks the Kubernetes API directly (no client-go/controller-runtime). The shipped operator manifest runs two replicas and --leader-elect; the replicas coordinate with a real coordination.k8s.io Lease so exactly one reconciles while the other remains a hot follower. The kind job in CI drives the reconciler against a real API server with only the RBAC the manifest grants: the API server accepts every object it renders, the generated pair completes the pinned handshake, a converged pass writes nothing, a spec change patches in place and removes what the spec dropped, and an object it does not own is refused. What comes next: the operator does not yet generate the KEK and signer-authorization Secrets or render the chart's PodDisruptionBudget, ServiceMonitor and external-KMS custody arguments (the chart covers those today), it polls rather than watches, and a control plane serving through an operator-managed isolated signer has been proven at the object and channel level, not yet as a running pair on a cluster.

  • Kubernetes certificate CRDs: the Kubernetes agent ships a real trstctl Issuer/ClusterIssuer/Certificate controller. It marks trstctl issuer resources Ready, signs matching cert-manager CertificateRequests through a served trstctl issuance endpoint using a mounted API token, signs approved native Kubernetes CertificateSigningRequests from certificates.k8s.io/v1, and also fulfills a trstctl-native Certificate directly into a kubernetes.io/tls Secret. GET /api/v1/kubernetes/certificate-signing-requests and trstctl-cli kubernetes csr expose the CAP-K8S-04 posture, supported signer names, RBAC, status fields, and residuals. The cert-manager path is proven in CI on kind with real cert-manager from Certificate to TLS Secret; the native trstctl path is proven by the served controller acceptance test from trstctl Certificate to local CSR, signer, Secret, and Ready status; native Kubernetes CSR support is proven by a controller test that writes status.certificate while preserving Approved only after Kubernetes approval (native CSRs do not define a Ready condition). It is an informer/work-queue controller: each resource is listed and then watched (following bookmarks and relisting when the server expires a version), and a de-duplicating, backed-off work queue converges one object per key as its event arrives, with a full re-check every --k8s-resync (default ten minutes) for what no watch reports. The kind job proves it with the agent's own service account and the re-check ten minutes away: a TrustBundle copied within half a second, a deleted copy restored, an approved CSR signed, a trstctl Certificate issued and renewed. trstctl Certificates are renewed with a new key two thirds of the way through the validity they had when issued (status.renewalTime), and reissued when their spec changes or their Secret is deleted. cert-manager is optional: without its CRDs the native CSR and TrustBundle paths run. Because the agent runs as a DaemonSet, the cluster-scoped controller elects a single reconciler through a coordination.k8s.io Lease (trstctl-agent-issuer-controller, 30s duration, identity from the pod's POD_NAME): one pod reconciles, the others idle, and a follower takes over within one lease duration if the holder dies. If the Lease RBAC is absent the agent logs once and reconciles anyway — duplicated idempotent work beats no controller. The namespaced cert-manager bridge does not contend. CSR approval stays a Kubernetes approver's by default. The opt-in --k8s-csr-approver (with deploy/kubernetes/rbac-csr-approver.yaml) has the agent ask the issuing CA about each pending native CSR for a trstctl signer name — POST /api/v1/ca/authorities/{id}/issue/evaluate, the checks issuance makes, without signing — and write Approved, or Denied with the CA's reason (condition reason TrstctlProfilePolicy); an approved request is signed in the same pass. A post-quantum subject key is evaluated on its names and lifetime. cert-manager CertificateRequests keep their own approver. The kind proof of the approver runs in the Kubernetes e2e job.

  • Multi-replica HA: the Helm chart runs the control plane multi-replica by default (replicaCount: 2, RollingUpdate maxUnavailable: 0, PodDisruptionBudget, pod anti-affinity), and running >1 replica is safe: leader election (a PostgreSQL session-scoped advisory lock) gates the continuous background workers — the outbox dispatcher, audit retention, idempotency/outbox GC, the projection tailer, the CRL scheduler, and the read-model snapshot worker — to exactly one replica so they never double-apply, with automatic failover to a follower on leader loss; all replicas serve reads. A shared signer key store (persistence.signerKeysAccessMode: ReadWriteMany) means every pod's locked-down sidecar signer (the isolated key-holder process) loads the same sealed issuing-CA key, so all replicas are the same CA (first-boot provisioning is serialized by an advisory lock). For a single signer pod that serves all replicas independently, set signer.mode: isolated: the signer runs as its own pod reached over a cross-node mTLS gRPC channel — TLS 1.3, AEAD-only, with the control plane and the signer each pinning the other's certificate (an untrusted or merely CA-signed-but-unpinned peer is rejected). The trstctl-signer binary serves --mtls-listen and the control plane dials it with signer.mtls_address; the chart renders the signer Deployment/Service/NetworkPolicy on :9443 when you supply the signer.mtls.* certificate material. The default co-located sidecar (UDS) topology remains the simplest single-pod option and is not required to change for the HA above. See disaster recovery → High availability. (The agent, separately, runs as a DaemonSet across all nodes.)

  • Cross-cluster federation is passive read-state replication: a passive cluster can import a peer event log, keep a durable per-peer cursor, and project the imported tenant/trust/certificate/audit read state locally for failover. It is intentionally not an active-active write conflict resolver — keep one writable region for a tenant at a time, stop or fence primary writes before promotion, and use TRSTCTL_FEDERATION_RPO / TRSTCTL_FEDERATION_RTO as measured runbook targets.

Non-functional targets: what is measured vs. aspirational

We separate NFRs that have executable evidence from ones that are aspirational and not yet measured in CI, so neither is silently over-claimed.

  • Performance & scale NFRs are measured: the hot-path latency/throughput SLOs and the capacity model are pinned to committed measurement receipts by an executable smoke gate (make perf-smoke) and a served realistic/peak live-load gate (make perf-live), and sustained-load endurance is pinned by a soak gate (make soak) that fails on a leak slope or an SLO breach. These are local eval/self-test scale denominators, not a substitute for a customer-specific multi-hour load test at your own capacity tier.
  • Usability outcome NFRs are evidence-gated. USABILITY-SLO-001 has automated wizard timing evidence: the scripts/usability/first-run-receipt.json receipt is generated by scripts/usability/measure-first-run.mjs, which walks the first-run wizard contract (internal CA confirmation, first certificate issuance, served connector deployment, upstream-CA issuance, dynamic-secret lease issuance, enrollment-token minting, and agent detection) and keeps that assisted path inside a 15 minute time-to-first-certificate budget. The scope is intentionally narrow: it measures the browser journey and served API-client contract in CI, not human reading time, package download time, real network latency, or the physical agent installation step. USABILITY-SLO-002 for operator-satisfaction / NPS has a receipt gate but no numeric NPS claim: the current scripts/usability/operator-study-receipt.json is no_numeric_claim, and release tooling fails closed if release notes publish NPS, CSAT, or operator-satisfaction numbers before a real measured external-operator study receipt replaces it. See Usability outcome SLOs.

How to read the roadmap against this

The source-checkout README.md capability table describes what is built and tested; this page tells you what is served by the binary today. When the two differ, this page is the authority for what you can rely on at runtime.

Non-production entitlement boundary

The signed non-production entitlement is an offline deployment binding, not a traffic classifier. Version 2 licenses enumerate one production ID and at most three non-production IDs; the binary refuses an operator-declared runtime ID or environment that does not match. It cannot independently prove that a host called non_production carries no real-user traffic, because doing so would require surveillance or phone-home state that the product explicitly avoids.

Version 1 license files predate environment binding. They remain loadable for upgrade continuity but are production-only and cannot claim the bundled non-production entitlement. Issue a v2 file before moving a legacy deployment into a staging or test slot.

Rendered live from github.com/ctlplne/trstctl — found a mistake? edit this page.