Current limitations & what's not yet served
trstctl is pre-1.0 and under active hardening. This page states plainly what the
running binary serves today versus what is built and tested as library code but
not yet wired into the served product, and which surfaces are explicitly Phase 2.
Maturity is separate from edition gates: Community self-host includes the core
control plane; Enterprise and Provider capabilities activate by an offline signed
license behind the ee/ boundary. trstctl's core is source-available under LICENSE
(BUSL-1.1, converting to MPL-2.0 four years after each release); Enterprise,
Provider and other license-gated features are proprietary material under
ee/LICENSE, while PQC and the patent-pending families ship in the core.
If a capability matters to your evaluation, check this page before relying on it.
The 60-second boundary summary
Read this first; use the exhaustive ledger below only for the capability you plan to operate.
| Question | Short answer | What to verify next |
|---|---|---|
| Can I run the core control plane myself? | Yes. Inventory, lifecycle state, ownership, policy, audit, the REST API, CLI, web console, PostgreSQL tenant isolation, and the separate signer are served. | Complete the blank evaluation, then rehearse production operations. |
| Does every integration work immediately? | No. Protocols, connectors, HSMs, cloud discovery, SSO, and commercial workflows are conditional on explicit configuration, reachable infrastructure, and sometimes a signed license. | Open the relevant feature page and the matching matrix row below before promising it. |
| Does “delivered” mean a workload is serving the credential? | No. Delivery records what the connector did. Only a successful endpoint handshake earns verified. | Review the served status vocabulary and configure a verification address. |
| Is a green System page a production certification? | No. It proves the running process's bounded checks, not an external penetration test, customer topology, disaster-recovery rehearsal, or third-party certification. | Use the production journey, threat model, and your own acceptance controls. |
| Is every feature production-mature? | No. trstctl is pre-1.0. The tables below distinguish Served, Conditional, Partial, and Roadmap without turning source code into a product claim. | Search this page for the feature ID or plain-language capability name. |
The safe decision rule is simple: served says the running binary has a real path; verified says the required external effect was observed; neither word replaces your deployment-specific acceptance test.
Local qualification host limit
The shipped definition-of-done runtime runner targets linux/amd64. Its proof
opens the kernel's /proc/<pid>/map_files object to check that the executable
being tested is the exact binary built by the gate. On an Apple Silicon Docker
daemon using QEMU user emulation, the kernel exposes the emulator's mappings
instead of the guest executable's mapping. The runner then stops at its
map_files preflight. All 81 census rows can show stub before any capability
is exercised; those rows cannot establish whether the capabilities work.
Run this qualification gate on native amd64 Linux, or on a separately
validated Docker Desktop Rosetta setup whose map_files proof succeeds. The
2026-10-02 local partner-lab run on the current Apple Silicon host passed its
18 live journeys, but its 81-row runtime census remains unqualified for this
specific host reason. Changing Docker Desktop's emulation mode requires a
daemon restart; defer that change while unrelated containers use the shared
daemon. Keep the runtime proof intact and record its failed native result.
Agent release distribution
As of 2026-10-02, the published v0.6.0 GitHub release lists zero assets. It
cannot supply a downloadable host agent, checksum manifest, or SLSA provenance.
The current source tree has a four-platform archive builder and a protected
release job that uploads and checks those assets. A task-local exact-commit
archive has been downloaded with curl, checksum-checked, and run on macOS;
that proves the candidate packaging path, not public distribution.
An operator needs a subsequently completed release with the archive for their
host, the source-commit manifest, SHA256SUMS, and provenance. Verify the public
asset roster, download without repository credentials, check the hashes and
provenance, and compare the binary's full reported commit with the manifest
before treating the release install journey as complete. Publishing that release
requires the protected release pipeline and is outside a local-only QA run.
Historical scheduled-rotation detail
The current live event generation does not serve arbitrary provider error text from schema-v1 scheduled-rotation terminal events. Upgrade sanitation replaces that one field with a fixed status-specific class before reads, projections, retention, or export can proceed. This guarantee applies only to repository- controlled live history. Backup/export files and signed or WORM audit archives created before sanitation are external copies: trstctl cannot rewrite them and does not claim they were erased. Operators must govern those copies under their existing custody and retention policy.
Feature served-state matrix
This matrix is the canonical served-state table for the feature catalog. The docs
test suite checks every F* row below against
internal/featureparity/feature-map-backlog.json and runs the repo-native wiring
census against the current source tree. A feature cannot claim Served while a
mapped capability is still library-only, a stub, or unknown; a stale generated JSON
receipt cannot certify it.
- Served means the running binary serves the capability end to end.
- Conditional means the served path exists but depends on configuration, license activation, or an operator-supplied backend.
- Partial means a real served path exists, with explicit residual work called out below.
- Library-only means built and tested, but not yet served by the binary: library code exists with tests, but is not yet wired into the served API.
- Roadmap means Phase 2 or later work with no current served product claim.
Served
| ID | Feature | Primary docs |
|---|---|---|
| F1 | Certificate inventory | docs/features/discovery-and-inventory.md |
| F2 | Network discovery | docs/features/discovery-and-inventory.md |
| F42 | SSH credential discovery and inventory | docs/features/discovery-and-inventory.md, docs/features/ssh.md |
| F49 | Agentless cloud certificate discovery | docs/features/discovery-and-inventory.md |
| F35 | Secret store discovery | docs/features/discovery-and-inventory.md, docs/features/secrets.md |
| F36 | API key / token inventory | docs/features/discovery-and-inventory.md, docs/features/secrets.md |
| F17 | Certificate Transparency monitoring | docs/features/observability-and-risk.md |
| Discovery coverage & provenance | Served: coverage is measured against operator-declared segments rather than against what discovery happened to find, with a per-segment staleness SLO, declared exclusions carrying their reason, and a named blind-spot register. Every certificate carries provenance — which source last observed it, of what kind, and when — distinct from when trstctl first recorded it. Headlined on the Discovery console. Nothing is rollable into coverage until an operator declares a segment: an inventory built from findings can describe what it found and nothing else, so an estate with no declarations reports no coverage rather than 100% | Coverage, provenance and blind spots |
| Revocation through external issuers | Served for letsencrypt (ACME), vaultpki and ejbca — each proven end to end against that authority's own protocol, asserting the AUTHORITY was contacted rather than that the call returned nil. Every other issuer kind reports revoke: false on the served capability matrix with a note naming where to revoke instead. A documented vendor endpoint trstctl does not drive is not counted as a capability. Revocation that cannot reach the authority fails visibly; there is no silent no-op |
Per-issuer capabilities |
| Upstream domain validation (DNS-01 as an ACME client) | Served: obtaining a certificate FROM a public CA now negotiates the challenge type the authority offers and solves DNS-01 unattended, reusing the provider configs already persisted for the server direction. Wildcard issuance from a public CA works for the first time — DNS-01 is the only challenge that can authorize one. Two independent opt-ins, both default off: upstream_dns01 on the ACME authority (which also requires that authority's own caa_issuer_domain) and allow_upstream_dv on each DNS-01 provider config. The console shows a domain-validation column beside every configured issuer — unattended or manual, with the reason — and an upstream authorization freshness panel reporting when each identifier last actually proved control, as against when it last rode a reuse. DNS-01 only: http-01 upstream would need an inbound listener this architecture does not have |
Upstream domain validation |
| F18 | Drift detection | docs/features/observability-and-risk.md |
| F19 | Credential risk scoring | docs/features/observability-and-risk.md |
| F52 | CBOM and cryptographic observability | docs/features/observability-and-risk.md |
| F4 | CA-agnostic outbound issuance | docs/features/issuance-and-cas.md |
| F48 | Private/enterprise CA hierarchy management | docs/features/issuance-and-cas.md, docs/runbooks/key-ceremony.md |
| F53 | Certificate profiles and registration-authority model | docs/features/issuance-and-cas.md, docs/guides/profile-authoring.md |
| F46 | ACME Renewal Information (ARI) | docs/features/issuance-and-cas.md, docs/features/acme-and-dns.md |
| F47 | X.509 revocation infrastructure | docs/features/issuance-and-cas.md |
| F25 | Ephemeral credential issuance | docs/features/workload-identity.md |
| F30 | Workload attestation chain | docs/features/workload-identity.md |
| F59 | Non-human identity lifecycle management | docs/features/workload-identity.md, docs/features/discovery-and-inventory.md |
| F61 | AI-agent / NHI identity broker | docs/features/workload-identity.md |
| F44 | SSH deployment and trust configuration agent | docs/features/ssh.md, docs/design/ssh-trust-rewrite.md |
| F45 | Attestation-gated short-lived SSH user certs | docs/features/ssh.md |
| F6 | Lifecycle automation | docs/features/lifecycle-and-pqc.md |
| F33 | Just-in-time issuance with approval flows | docs/features/incident-and-jit.md |
| F38 | Ephemeral API key issuance | docs/features/secrets.md |
| F28 | Policy engine | docs/features/policy-and-governance.md, docs/cli.md, docs/web-console.md |
| F29 | Notification integrations | docs/features/policy-and-governance.md |
| F8 | RBAC | docs/features/policy-and-governance.md |
| F9 | Audit log surfaces | docs/features/policy-and-governance.md, docs/observability.md, docs/configuration.md |
| F10 | REST API | docs/features/platform-and-api.md |
| F11 | CLI | docs/features/platform-and-api.md, docs/cli.md |
| F12 | Web UI | docs/features/platform-and-api.md |
| F14 | Single-binary distribution | docs/features/platform-and-api.md |
| F15 | Encrypted control-plane transport | docs/features/platform-and-api.md |
| F40 | Multi-tenant deployment topology | docs/features/platform-and-api.md |
| F20 | Plugin SDK with capability sandboxing | docs/features/extensibility-plugins.md |
| F21 | Credential graph | docs/features/graph-query-ai.md |
| F79 | Privacy and data-subject controls | docs/features/policy-and-governance.md, docs/privacy-data-catalog.md, docs/web-console.md, docs/configuration.md |
Conditional
| ID | Feature | Primary docs |
|---|---|---|
| F3 | Agent-based discovery | docs/features/discovery-and-inventory.md |
| F5 | Built-in ACME server | docs/features/acme-and-dns.md |
| F69 | DNS-01 challenge automation | docs/features/acme-and-dns.md |
| F70 | DNS-provider plugin framework | docs/features/acme-and-dns.md |
| F71 | CNAME delegation for validation isolation | docs/features/acme-and-dns.md |
| F72 | CAA policy enforcement and management | docs/features/acme-and-dns.md |
| F73 | Multi-method domain-validation policy | docs/features/acme-and-dns.md |
| F74 | Automated wildcard issuance and renewal | docs/features/acme-and-dns.md |
| F22 | EST server | docs/features/enrollment-protocols.md, docs/guides/est-enrollment.md |
| F23 | SCEP server | docs/features/enrollment-protocols.md |
| F55 | CMP server | docs/features/enrollment-protocols.md |
| F54 | Embedded / IoT enrollment agent | docs/features/enrollment-protocols.md |
| F56 | Intune / MDM enrollment integration | docs/features/enrollment-protocols.md |
| F24 | SPIFFE Workload API | docs/features/workload-identity.md |
| F43 | SSH certificate authority | docs/features/ssh.md |
| F50 | Code-signing service | docs/features/code-signing-and-timestamping.md |
| F51 | Timestamping authority | docs/features/code-signing-and-timestamping.md |
| F26 | HSM integration | docs/features/issuance-and-cas.md, docs/configuration.md, docs/compliance.md, docs/limitations.md |
| F7 | Deployment connectors initial set | docs/features/deployment-connectors.md |
| F27 | Additional deployment connectors | docs/features/deployment-connectors.md |
| F31 | Credential compromise workflow | docs/features/incident-and-jit.md, docs/features/discovery-and-inventory.md |
| F32 | Fleet re-issuance for CA compromise | docs/features/incident-and-jit.md |
| F34 | Break-glass procedures | docs/features/incident-and-jit.md |
| F37 | Secret rotation engine | docs/features/secrets.md |
| F39 | Code/CI secret scanning bridge | docs/features/secrets.md |
| F63 | Native secret store | docs/features/secrets.md |
| F65 | Dynamic secrets | docs/features/secrets.md |
| F80 | AWS IAM decoy keys | docs/features/secrets.md, docs/web-console.md, docs/cli.md, docs/limitations.md |
| F66 | Encryption-as-a-service and KMIP | docs/features/secrets.md |
| F68 | Secret sync / platform integrations | docs/features/secrets.md |
| F67 | PKI as a secrets engine | docs/features/secrets.md |
| F58 | Platform auth-method framework | docs/features/secrets.md |
| F60 | Secret sharing and secret-change approvals | docs/features/secrets.md |
| F62 | Cryptographic compliance reporting & posture dashboards | docs/features/policy-and-governance.md, docs/compliance.md |
| F13 | SSO/OIDC | docs/features/platform-and-api.md |
| F41 | Cross-cluster / multi-region federation | docs/features/platform-and-api.md |
| F75 | Unified semantic query layer | docs/features/graph-query-ai.md |
| F76 | Pluggable AI model adapter | docs/features/graph-query-ai.md |
| F77 | Grounded RCA and natural-language query | docs/features/graph-query-ai.md |
| F78 | trstctl MCP server | docs/features/graph-query-ai.md |
Partial
| ID | Feature | Primary docs |
|---|---|---|
| F16 | Crypto-agility and PQC readiness | docs/features/lifecycle-and-pqc.md |
| F57 | PQC migration orchestration | docs/features/lifecycle-and-pqc.md |
| F64 | Developer secrets experience | docs/features/secrets.md, docs/cli.md, docs/journeys/manage-secrets.md |
Library-only
| ID | Feature | Primary docs |
|---|
Roadmap
| ID | Feature | Primary docs |
|---|
Feature pages without an F* catalog row
A feature page can document a surface that carries no F* catalog ID, so the
generated matrix above cannot hold it. Those pages record their served state
here instead, in the same vocabulary, so no page in the feature catalog is
statusless.
| Page | Served state | Why it has no F* row |
|---|---|---|
| docs/features/agent-delegation.md | Conditional | Part of the core since 2026-09-20. attachAgentDelegation in cmd/trstctl/attach_families.go attaches the /api/v1/agent-delegation/* routes, root-event projection and agentid.issue-chain-bound outbox worker in every build. Root registration writes an immutable first-writer event, rebuilds the tenant registry from it, then writes and independently reads the signer's public floor over the authenticated signer transport. The issuance receipt has a bounded status read that binds the immutable result to the exact public credential ID; historical receipts without a binding are not guessed. Legacy SQL-only roots require exact signer readback before event backfill; a one-shot rebuild refuses them. The control plane provisions and independently reads back its public reachability-verdict floor through the signer RPC. Tenant attestors register the same way: an immutable first-writer event binds each attestor key ID to its public key and one attestation method, and the control plane provisions the signer's tenant attestor floor and reads it back. The signer verifies a tenant's attestation evidence only against that tenant's attestors, for the registered method and no higher class, or against a deployment-wide attestor an operator placed in its floor. The signer refuses a chain-bound mint until the required floors are present. Each tenant has its own agent issuing CA in the signer (its key reloads after a restart; the certificate is re-issued over it before expiry), published at GET /api/v1/agent-delegation/issuing-ca for relying services to pin; GET /api/v1/agent-delegation/credential-status reports active, revoked, expired or unknown, so a relying service that checks it stops honoring a revoked delegation within its status window rather than at expiry. Reference integrations on the MCP Go SDK and the Agent Development Kit for Go (test/interop/agentframeworks) run against the real signer in the integration suite, as does an agent-to-agent pair on the official A2A Go SDK: signed Agent Cards bound to the serving agent's certificate, mutual TLS, and the delegation gate on every request. What comes next: a pushed revocation feed for relying services (the downstream revocation plane only acknowledges today), and an issuing CA an operator provisions per tenant. |
| docs/features/client-sdks.md | Served | The generated clients under clients/sdk/ track the served OpenAPI 3.1 contract; they package the REST API rather than adding a capability of their own. |
Census proof modes
The served denominators quoted in the README and on this page are checked against the
repo-native census gate (tools/dodcensus/manifest.json in the repository) (make dod-gate emits the
local wiring-census.json receipt). Proof modes are not uniform:
12 of 81 census rows launch the shipped binary;
69 of 81 are proved through the production-assembled handler — the production
buildRunDeps output driving the assembled Server.Handler in-process, with a
hand-built Deps rejected. Only the process launch differs, so each served row names
the mode that proved it. This vocabulary is for maintainers and auditors; a customer
reads the README's one-line summary and the journeys.
Status at a glance
One line per domain below, for a reader who wants the answer without the prose. "Served" here always means the running binary, not a library package.
| Domain | Status | Detail |
|---|---|---|
| Core inventory, lifecycle, connectors, discovery | Served end to end | Served by the running binary today |
| Tenant offboarding & audit retention | Served; PostgreSQL rows erased, event log/archive follow separate retention | Tenant offboarding boundary |
| Library-only backlog | Empty — nothing is stuck library-only right now | Built and tested, but not yet served |
| Conditional/partial residuals | Real served spine; specific operator-facing edges remain | Conditional, partial, and residual boundaries |
| Served status strings | Every status is registered with what the code actually did; CI blocks a status spelled stronger than its own flags | Served status vocabulary |
| CA hierarchy expiry horizon | Served; year-scale bands, re-alerting on each tightening, leaf-validity-compression check, horizon on the CA API and console, opt-in drafted re-key ceremonies at a chosen band | The CA calendar |
| ACME external account bindings | Served; kid persisted on the account, per-credential identifier scope / quota / window / profile enforced fail-closed, runtime disable, served policy replacement and rotation by secret-store reference (both replayed from the ACME state log) | Protocols |
| Certificate Transparency monitoring | Served as a headline Discovery capability: exact source replacement, active versus retired watchlist history, immutable per-log success/failure with independent checkpoint progress, unexpected-issuance findings, and remediation hand-off. Covers only the domains and logs configured | Served by the running binary today |
| Key custody per credential kind | CI-checked table; every enrollment protocol, and the identity API given a CSR, generate keys in your environment. Four paths still generate one in the control plane, each named with its successor and each governed by the tenant's control-plane key generation switch: the three deprecated paths (identity issuance without a CSR and its renewal, PKI-as-a-secret by common name and the Vault-compatible pki/issue, the compatibility Workload API socket) and the reissue step of a PQC migration run that predates host-csr migration. Kubernetes TLS auto-issuance never generates a key in the control plane: a k8s_ingress_gateway source bound to the host-role agent running in the cluster hands each TLS block to that agent, which generates the key inside the cluster, sends only a CSR and writes the TLS Secret with a signed receipt; a source without an agent records each resource and reports it not delivered, with the reason |
Key custody |
| Key custody per credential | Served: custody is recorded on the certificate row at issuance from what the issuing path actually did, returned by the certificate API, and shown on the certificate in the console. Certificates issued before this shipped get the origin their own retained issuance event proves, marked as derived and naming the rule and the event (Key custody lists the rules); the rest of them, and every certificate found by discovery, read as not recorded — which is a different statement from any custody claim, and is never rendered as reassurance. A certificate ordered through an external CA records requester custody (A121), and where a host agent's key inventory found a matching key file the detail lists where it was seen, as an observation rather than a custody claim (A121) | Key custody |
| Agent job ledger | Served: agents claim, lease, extend, report and lose work over the mTLS channel; aggregate waiting/claimed health is on Operations. The shipped agent census executes connector.deploy, connector.test, connector.rollback, endpoint.renew, connector.right_size (for bindings that name a relay), endpoint.contain, endpoint.recovery_check (enabled with endpoint.contain), endpoint.verify, discovery.run, ssh.trust.collect (explicit read-only agentless SSH sources), revocation.probe, adcs.inventory, adcs.inventory.enrollment_groups (opt-in bounded nested enrollment links), adcs.inventory.endpoint_probe (opt-in scoped LDAP hostname probes with retained pages), trust.distribute, cmdb.sync, mdm.sync, ticket.sync, agent.upgrade, and agent.self_check (enabled with agent.upgrade); each row's role and agent constraints select the eligible host agent or network relay. Nothing is claimable until an operator names that kind in agent_channel.claimable_job_kinds — including connector.rollback, which must be enabled separately from deploying |
The agent job ledger |
| Agent job receipts | Served: every terminal report is signed by the agent with the key behind its channel certificate, verified against the certificate that authenticated, stored with the event, and refused fail-closed with an audit event when it does not verify. A successful host-generated renewal uses the v2 statement, which also binds the installed certificate fingerprint plus origin, storage, exportability, and generator; the projected custody event is replayable. Verified and refused counts, and the reason for the most recent refusal, are on Operations. The signature is over the report's facts and a digest of its text — it attests what the agent SAID, not that the appliance changed | The agent job ledger |
| Renewal windows, canaries and SLOs (D6) | Served: maintenance windows restrict when the scheduler may renew (lifecycle.maintenance_windows, e.g. Mon,Tue,Wed,Thu,Fri 22:00-06:00 Europe/London); a closed window defers with a recorded reason naming when it reopens, never drops. Fleet re-issuance is a durable outbox-backed batch state machine: start publishes only the canary, an accepted signed agent receipt is required to advance, pause/halt stores the cursor and reason, and resume/restart reuses deterministic ids. Any failed verification fails the gate and any unverified replacement keeps it not_evaluated; later batches remain unpublished after a canary failure. Renewal success SLO with error-budget burn on GET /api/v1/operations/renewal-slo, window and target both operator inputs |
Renewal windows, canaries and SLOs |
| Endpoint verification (D2) | Served: after a deploy the host agent handshakes the changed listener — the only observation of whether the reload took effect — and a network relay probes the same endpoints as a client would, which is the only witness for an appliance. Divergence is classed (fingerprint, sans, chain, expired, not_yet_valid) because the remedies differ; unreachable is neither a pass nor a divergence. Results are signed: the probe transcript's digest travels inside the agent's receipt, so a verdict is checkable rather than asserted. Verification is opt-in per target: an endpoint with no configured listener address is never verified and never claims to be. verified % on the dashboard is a percentage of OBSERVED endpoints and the tile is hidden entirely until something has been observed. Sweeps re-probe hourly; divergence raises a critical alert (unreachable: warning) through the notification outbox; automatic rollback to the predecessor is available per target, opt-in and off by default |
Endpoint verification |
| Connector rollback | Served through two honest execution models. f5, kemp, netscaler, a10 re-bind a fingerprint-named object already on the appliance, and fortigate (bind_to), paloalto (ssl_tls_profile + commit_admin), cisco (node_hostname + certificate_roles) and gcp-certificate-manager (certificate_map_entry, executed on the control plane where that family's deploys run) do the same when the target names the binding to move. All 14 host connectors restore the one encrypted predecessor bundle retained only by the exact enrolled host agent, run the connector reload, and reverify the listener when configured. Deploy and rollback share a serialized per-target lane. aws-acm, azure-keyvault and plugin-backed routes have no executable rollback by their APIs' design; the route returns 409 naming that reason, the destination's rollback_readiness says the same, and no rollback-shaped receipt is written. A re-bind target without its binding is refused at request time. Automatic rollback after verify_failed is opt-in per target; manual rollback uses the same job and signed transcript |
The agent job ledger |
| CA migration waves (H2) | Served for internally issued X.509 identities on host-agent connectors: read-only assessment, reviewed exact-authority manifest, durable trust-before-leaf waves, signed trust/live gates, pause/resume, halt, and newest-first rollback | Migration waves |
| Agent roles (host / network relay) | Served: an operator grants host and/or network at enrollment, the CA stamps it into the certificate, and the claim path refuses out-of-role work. Role badges on Agents. Agents redeem credential material just-in-time, once per job attempt. Connector deploys carry a per-row role demand stamped at enqueue from the shipped vantage census — an F5 deploy is claimable only by a relay, an nginx deploy only by a host agent, a cloud-store deploy by no agent. The control-plane dispatcher structurally refuses host-stamped and legacy host-family rows before native lookup or I/O, so 14 host families execute only on the enrolled host agent; an unavailable agent leaves pending work, never a control-plane fallback | Agent roles |
| React web console | Served: real embedded Vite build at /, generated API types |
The React web console |
| OIDC/SAML/LDAP browser login & tenancy | Served behind config flags; each user maps to a real tenant | Browser login & sessions |
| SCIM 2.0 + NHI inventory/posture | Tenant SCIM requires Enterprise SSO; SCIM Bulk served on both planes; directory writeback served to a SCIM-serving directory, never to a push-only IdP | SCIM 2.0 provisioning |
| AI / RCA / MCP surface | Served, off by default, air-gapped unless an operator opts in | AI, RCA, and MCP surface |
| Secrets, identity frameworks, transit/KMIP | Served (six of six frameworks); Vault shim is a partial subset | Secrets and identity frameworks |
| RBAC / ABAC / OPA policy gates | Served, fail-closed, off by default | Authorization policy gates |
| Plugin isolation | First-party runs trusted in-process; third-party is WASM-sandboxed and signature-verified | Plugin isolation |
| Protocols: ACME/EST/SCEP/CMP/SPIFFE/SSH/TSA | Served end to end, each behind its own enable flag | Protocols |
| Revocation (OCSP/CRL) | Served, signed by the isolated signer | Revocation |
| Single sign-on detail | Served, including SAML single logout, Kerberos/GSSAPI and NTLM (through a domain-joined helper) login on the tenant plane | Single sign-on |
| CA key custody / HSM / BYOK | Sealed local key store by default; six of six HSM/KMS backends served | CA key custody |
| Post-quantum cryptography | ML-DSA/ML-KEM/SLH-DSA available; compatibility-bounded proofs | Post-quantum cryptography |
| PCAS KEM data re-wrap | Not built; the served route prepares successor keys but cannot migrate ciphertext | PCAS KEM data re-wrap |
| Kubernetes deployment | Helm chart plus an Operator that converges the control plane's Deployment, Service, default-deny NetworkPolicy, optional Ingress and isolated signer with a generated mTLS pair; it polls rather than watches | Kubernetes deployment |
| Performance & usability NFRs | Perf/scale measured in CI; usability evidence-gated, no NPS claim yet | Non-functional targets |
Served by the running binary today
The trstctl binary assembles and serves a control plane: the tamper-evident event
log, the read models it projects, the lifecycle orchestrator, and the REST API, with
the signing service supervised as a separate out-of-process child so private keys
never live in the API process. What you can do end to end against the running binary:
Inventory and lifecycle for owners, issuers, identities, and certificates: create, read, list (keyset-paginated), and drive the lifecycle state machine.
Connector delivery and rotation evidence: deployment attempts emit
connector.delivery.recordedreceipts and scheduled renewals emitlifecycle.rotation.recordedruns, both readable through the API, CLI, and console. The receipt is routing/status metadata only — no private key or secret bytes are returned. Rotation replay comparescompleted_atat PostgreSQL's microsecond precision: sub-microsecond bits that cannot survive the first database write are one observation, while a difference of one stored microsecond still fails closed as changed terminal evidence. This lets exact JetStream replay converge without weakening the run ID, tenant, outbox, binding, outcome, or sequence checks.Automated endpoint binding: first call the effect-free
POST /api/v1/lifecycle/endpoint-bindings/previewwith the exact platform, private, or external issuer. The preview reports CA identity, key custody, target revision, writes, queued effects, recovery, verification, and a request fingerprint while performing zero writes and zero external calls. Sending that unchanged fingerprint toPOST /api/v1/lifecycle/endpoint-bindingscreates the X.509 identity for an existing owner, provisions or references the connector target, pins the chosen issuer, and queues issue/deploy work through the outbox. Missing, unavailable, or changed issuers fail closed; the worker does not substitute the built-in CA. The leader scheduler later renews through that same pinned authority and sends the successor throughconnector.deploy.Certificate key custody, corrected (B5 re-audit): the four custody columns (
key_origin,key_storage,key_exportable,key_generated_by) are now actually persisted. They were added to the schema, set by the issuing code and served by the API, butprojections.CertificateRecordedhad no fields for them and the projector's INSERT never listed them — so every projected certificate recorded blanks while three layers of the system agreed custody had been captured. B5 was signed off in that state. Thekey_originhalf surfaced while implementing B2, because B2's own claim depended on it; the other three surfaced only on re-audit. A regression test now reads the DATABASE rather than asserting a struct field was assigned — the distinction that made this invisible — and a structural test fails if a custody field exists on the row without a matching field on the event that carries it.Host-generated endpoint keys (B2): a deployment target whose config sets
executor: "agent"opts out of credential-bearing delivery entirely. When the lifecycle scheduler renews an identity bound to such a target, the issuance dispatcher queues anendpoint.renewjob INSTEAD OF MINTING — the branch is at mint time, not deploy time, because once a certificate has been minted for a server-keygen identity the control plane already holds a private key and no later refusal can unmake that. A host agent then claims the job, generates the subject key on the machine that will serve it, sends a PKCS#10 up throughSignJobCSR, installs the returned certificate with its locally held key, and verifies the listener. The rotation run is recorded as succeeded on the HANDOFF, not on a certificate: the certificate does not exist until the agent's CSR arrives, and D3's three-state truth reports the rest. The control plane never holds that private key, andenforceExecutorParityREFUSES — rather than silently falling back — any deploy that would carry key bytes to such a target, so a target cannot read as migrated while still receiving keys. Scope, stated exactly: this is per target and opt-in; targets without the marker keep the control-plane path unchanged, which is the supported default and not a defect. Renewal is host-vantage only — a network relay cannot claimendpoint.renew, because generating a key for an appliance it merely reaches would reintroduce the custody hop this removes. The CSR is authorized against the names the job payload already carries, so an agent cannot widen its request.GET /api/v1/endpoints/key-custodyand the Connectors console report, per target, which path it is on and how much of the estate has moved.Expiry-alert delivery: the leader lifecycle scheduler honors the configured alert window, writes
notification.expiryoutbox work, stampsalerted_atin the same transaction so one certificate does not spam, and the outbox worker dispatches through operator-wired Slack, Teams, email, SMS, SIEM, webhook, PagerDuty Events v2, or OpsGenie Alert v2 channels. The dispatcher holds credentials in locked memory and wipes them on shutdown; the payload carries the owner, approver escalation recipients, severity, and threshold-day metadata. Tenants manage their own channels at/api/v1/notification-channels(Slack, Teams, webhook, PagerDuty, OpsGenie, SIEM) and routing policies that name them; since A98 each tenant-authored channel delivers in its family's own wire shape (a Slack or Teams incoming-webhook message, a PagerDuty Events v2 trigger or OpsGenie create carrying the key the row'scredential_refnames in the tenant's secret store, a webhook HMAC-signed under that secret, a bearer-authenticated SIEM event) instead of one raw JSON POST, the credential is opened into locked memory for one delivery and released after it, and a row whose credential cannot be resolved fails only its own delivery with a credential-free reason. Email and SMS stay operator-configured: the API refuses a tenant row of those families because the record cannot carry the SMTP or gateway settings they need.Air-gapped alert delivery: operator-configured PagerDuty, OpsGenie and, since A99, Slack, Teams, SIEM and SMS channels can each opt into the operator's bounded private-egress policy (
*_ALLOW_PRIVATE_CIDRS, exact network addresses validated at config load, plus loopback-only*_ALLOW_INSECURE_HTTPfor an emulator), so an internal relay, proxy, gateway or collector can receive alerts while every other destination stays behind the SSRF guard; a channel with a policy also carries the air-gap guard when air-gap mode is on, and a channel without one keeps the public-HTTPS-only default. The partner lab grants only127.0.0.1/32. Tenant-authored channels may use private HTTPS receivers only after the deployment operator grants their exact tenant UUID and networks innotifications.tenant_private_egress. Without that grant they remain public-HTTPS-only. The API and delivery worker use the same grant; DNS is checked again at every dial and redirects cannot change origin. Tenants cannot grant themselves access. Tenant grants refuse loopback, metadata, link-local, wildcard and partly public ranges; HTTPS certificate validation and the air-gap guard still apply. The served private-TLS/outbox test proves one receiver and tenant isolation; fresh-image console and restart proof is recorded separately in the build replay.Deployment connector orchestration serves target metadata, identity binding, outbox intent, receipts, provenance-verified WASM dispatch, and all 24 advertised native connectors. Device proof (E1): every appliance family — a10, cisco, f5, fortigate, kemp, netscaler, paloalto — now carries a faithful in-process double of its management API and tests that drive the REAL connector against it, and the connector catalog reports which families have that proof. This is a different claim from the conformance suite every connector passes: conformance runs against an in-memory double that accepts any request, so it proves a connector respects its capability grant and is replay-deterministic while proving nothing about whether the device would have accepted the call. For an appliance, whose entire implementation is an API conversation, that was the only untested part. Each suite is verified load-bearing by stubbing the connector's Deploy to return nil and confirming the tests fail. A guard test refuses a family that claims device proof without both an emulator package and a test that drives it. Single-box demo (A5):
trstctl --demoserves the control plane AND starts a colocated host agent. Evaluating this product otherwise needs two installs, and until both exist nothing the product is FOR can be shown — no deploy executes, no endpoint verifies, no renewal lands on a host. It is also the cheapest defense against the defect this program keeps finding: six capabilities have been complete, tested and unreachable from the running binary, and every one would have been obvious the first time somebody drove it end to end on one machine. The agent is EXEC'd, never linked.cmd/trstctl-agentmust not link the control plane and a guard pins that; importing it here to save a process would put both on the same side of the boundary the architecture rests on, and the demo would stop exercising the real channel. Two binaries over the real gRPC channel is what a deployment does, so the demo tests the actual path.--demoFORCES two settings and reports both rather than applying them quietly: the agent channel (off by default — correct for production, fatal for a demo, because the agent has nothing to dial) and a conservative claimable-job set (discovery.run,endpoint.verify,connector.test). An empty claimable allowlist is the trap worth naming: the job ledger is served, hands nothing out, and the agent enrolls successfully and idles while every surface looks healthy — the most misleading possible demo.connector.deployis deliberately NOT enabled; an evaluation box must not mutate an appliance somebody pointed it at by accident. Settings an operator did set are never overwritten. A missing agent binary is a FATAL error naming where it looked and how to build one, not a warning.--demois an explicit request for a colocated agent, and a run that serves the control plane while silently omitting the agent is exactly the "looks like it worked" outcome the flag exists to prevent. The bootstrap token is minted through the SERVED enrollment API — not by reaching into the store, because reaching in would let the demo work while the served enrollment path was broken — and written to a 0600 file, never passed as an argument, since process arguments expose bearer credentials and the agent refuses them for that reason. Also fixed here:agent_channel.claimable_job_kindswas the ONE AgentChannel field with no environment key, so a container deployment could enable the channel and had no way to let an agent claim anything.deploy/demo/docker-compose.ymlset no agent-channel variables at all while its seed job advertised enrollment tokens nothing could redeem; it now enables the channel and the same conservative claimable set. SECURITY FIX (provider plane, 2026-08-05): the provider plane authenticated NOBODY.operatorFromRequestparsedAuthorization: Bearer provider:<id>:<email>for SHAPE and returned an operator withRole: OperatorAdminandMFA: true— no verification of any kind, and multi-factor asserted on the caller's behalf./provider/is mounted on the root mux behind only a bulkhead whenever the provider plane is licensed, so on any provider-tier binary tenant create, suspend, offboard and break-glass were reachable by anyone who knew the token format. The format was in the source. Break-glass was worse:consentBreakGlassdid not authenticate at all and took the consenting subject from the REQUEST BODY, so the operator who requested emergency access named whatever approver they liked and consented to their own grant with a second call. Two-person control defeated by a JSON string. Fixed by requiring a configuredOperatorAuthenticator. Every other dependency in the provider Config falls back to a working stand-in; this one deliberately does not, because the safe stand-in for "who is this caller" does not exist and a placeholder is precisely how the original behavior came to ship. A nil authenticator now refuses every request — an unconfigured provider plane is closed, not open — and the consenting subject is the authenticated caller, with asubjectin the body REJECTED rather than ignored so an integration cannot keep sending one and believe it has effect. Unauthenticated requests answer 401 rather than 403: "I do not know who you are" is a different statement from "I know, and you may not". Both bypasses are mutation-verified against the real handler. The consent test creates a real tenant and a real grant first — an earlier version used a made-up grant id and the mutation check showed it caught the regression only because a nonexistent grant 404s, so its primary assertion never fired. Note that this CLOSES a surface that was open: a provider-tier deployment must now wire an authenticator before/provider/serves anything, which is a deliberate breaking change and the right direction. THE AUTHENTICATOR NOW EXISTS (L1):provider.oidcin the config federates operator identity to the provider's own IdP — bearer tokens verified OFFLINE against a pinned JWKS (jwks_file/jwks_json; the plane never fetches keys from a URL the token's minter might control), issuer, audience, expiry and nbf all required, and ROLE AND MFA read from claims the IdP signed (role_claimvalues mapped byadmin_values/operator_values;mfa_claim, defaultamr). A token the IdP vouches for that matches NO configured role is refused outright — federation is not enrollment, and the IdP vouching for the whole workforce must not make every employee someone who can suspend customers. A single-factor operator authenticates and is then refused mutations (403 at the act, not 401 at the door — the gap is MFA and the error should say so). Console sign-in (A222): withprovider.oidc.loginthe console signs in through the IdP's authorization code flow (state, nonce, PKCE S256) and holds an HttpOnly, SameSite=Strict Provider session instead of a pasted bearer; the ID token meets the same offline checks with the client id as audience, no token reaches browser storage, and the directory row is re-read on every request. The session store is in memory, so a restart signs operators out. Federation answers WHO; the per-customer delegation still answers WHICH customers, and the combined test proves a federated admin delegated one customer cannot suspend another. SAML AND SCIM ARE NOW SERVED (AUD-58): Provider SAML mounts a separate SP at/provider/v1/auth/saml/login,/acs, and/metadata, verifies signed assertions plus issuer/audience/time/request correlation and mapped role/MFA, then issues a Provider-only HttpOnly session protected by double-submit CSRF. Provider SCIM mounts/provider/scim/v2; its file-backed bearer is hashed and wiped at startup. Join/update/leaver changes are immutable operator events. When SCIM is enabled, OIDC and SAML query that directory row every request, so a deprovisioned operator's still-valid credential is refused immediately and the same projection revokes all standing customer authority. Provider admin + MFA list/grant/revoke/role routes expose identity source, customer, operation, expiry, last use, and retained revocation evidence. Provider SCIM Bulk (POST /provider/scim/v2/Bulk, A89) applies User operations in order withbulkIdreferences, each under its own idempotency key so a replayed request appends nothing twice; group operations are refused per operation. Custom Provider roles (A90):GET /provider/v1/rolesserves the built-inadminandoperatorroles, the defined custom roles, and the closed permission vocabulary (tenants:provision,tenants:lifecycle,quota:write,brand:write,operators:admin,roles:admin,isolation:drill,breakglass:request,breakglass:consent,breakglass:results);PUT/DELETE /provider/v1/roles/{name}(operators holdingroles:admin) define a role as a subset of that vocabulary or delete one nobody holds, each an immutableprovider.role.upserted/deletedauthority event. Every Provider mutation now asks for its permission rather than for the admin role; a custom role is assigned through the role route, the access console, or a SCIMrolesvalue, and its effective permissions are intersected with the ceiling the signed IdP role allows (adminoroperator), so a custom role never grants what the IdP would not. SCIM Groups expose the two built-in roles asprovider-adminandprovider-operator(A221): an IdP group push adds, removes and replaces members (PATCH in operation order, in the Okta and Entra shapes, and PUT), and a member it removes or leaves out loses the role as aprovider.operator.role_revokedauthority event naming the group and the SCIM token. The operator stays active with no role, which OIDC and SAML refuse on the next request; a user created without roles holds none until a group push assigns one. Standing customer delegations stay until a leaver event (active:falseor DELETE) revokes them, and SCIM Bulk still refuses group operations per operation. What comes next: a live Okta and Entra push (today's proof is handler and PostgreSQL tests built on the request shapes their documentation shows). Crypto migration sequencing (M2):GET /api/v1/graph/crypto-readinessand a Risk console panel order every observed crypto asset by WHO DEPENDS ON IT, not by severity alone. The CBOM already said which algorithms are weak; it could not say which change is hard. Sorted by severity only, a forgotten lab box exhibiting RSA-1024 outranks a load balancer twelve services authenticate through, and the migration gets planned in the wrong order by a table that looked authoritative. The traversal walks BACK from a crypto asset along EXHIBITS to the resources using it, then BACK again along CONNECTS_TO / GRANTS_ACCESS / DEPLOYED_TO to the parties that depend on those resources. Both hops are reverse because every edge involved points from the dependent toward the thing depended on — a first cut walked out-edges from the resource, found nothing, and reported zero dependents for everything while looking perfectly well-formed, which is the worst failure available to a surface whose whole job is counting dependents. Each dependent is served with the resource it was reached through, so a sequencing recommendation can be checked rather than trusted, and owners are attributed so a blocker has names against it. Two honesty constraints are built into the wording rather than left to the reader. Dependents are what DISCOVERY HAS OBSERVED: the graph is built from scans, so an asset with zero dependents renders identically to one sitting on a resource nothing has scanned, and no row is ever labeled safe to rotate — the recommendation for a weak asset with no observed dependents says in words that zero observed is not zero. And a CBOM usage recorded with NO LOCATION has no place on the graph and no computable blast radius; it is counted separately as unplaceable rather than sorted to the bottom, because an asset that cannot be placed is unmeasured, not low-risk. AUD-64 fixes the production source and direction of those claims. A host/network reporter sends a metadata-onlyservice_dependencyfinding through the tenant-boundagent.mtls.ReportInventorychannel. Source kind and finding kind must BOTH beservice_dependency;workload,target, and the targetrefmust be present and agree; and the verified agent certificate stamps the observing host and agent ID. A completed run binds that ID before the graph treats the observation as authority; manual discovery findings cannot forge this edge. The immutable discovery event survives restart/rebuild.graph.Buildjoins the exact workload name to that tenant's owner row and emits workload -> resourceCONNECTS_TO; an unknown owner or mismatched target fails the graph read rather than silently reporting zero. Credential owner attribution separately follows the INCOMING productionOWNSedge (workload -> credential), so tests no longer reverse that relationship. Scope: this sequences only dependencies an agent actually reports. It does not infer network traffic, and its ordering is only as complete as the reporters feeding it. AUD-65 makes this one exportable workflow instead of a Risk-only readout.GET /api/v1/graph/crypto-readinessis the canonical tenant dataset consumed by Risk and CBOM/Posture; it includes a digest and event-projected owner actions.POST /api/v1/graph/crypto-readiness/actionsbinds an existing core PQC campaign finding to the exact row digest and current attributed owner. Missing, foreign, unlocated/unowned, or later-changed topology is refused rather than guessed; stale actions stay visible but return409on mutation.GET /api/v1/graph/crypto-readiness/exportreturns the same ordered rows as bounded CSV and NDJSON plus an audit-key JWS/JWKS, and compliance evidence-pack v5 embeds the same dataset in its signed manifest. This still cannot discover dependencies no reporter observed; the coverage guidance travels in every format. CORRECTION (D2/D3, B2, R1, F1, H2 — 2026-08-05): FIVE agent-claimable job kinds dead-lettered before any agent could claim them. The control-plane dispatcher is the sole handler for every outbox sweep, and its default branch returns a hard error for an unrecognized destination — right for a genuinely unknown one, fatal for work an AGENT is meant to execute. A hard error burns the row's attempt budget and lands it instatus='failed', andClaimAgentJobsonly ever hands out rows inpending. Soendpoint.verify(D2/D3),endpoint.renew(B2's host-generated renewal — the epic's central path),revocation.probe(R1),adcs.inventory(F1) andtrust.distribute(H2) each enqueued rows that died on arrival, while the API had already told the operator the work was queued. This was already understood for two of the nine claimable kinds:connector.rollbackandconnector.testcarry explicit deferral cases whose comment spells out this exact failure. Nobody applied it to the other five. The dispatcher now derives the answer fromagentJobKindAllowlist— the one place that decides what an agent may execute — so a kind added there cannot be forgotten here, because there is nothing here to forget. A genuinely unknown destination still fails closed: the dispatcher must not become a global silent ACK, which loses a row quietly instead of dead-lettering it loudly. Related and separate:RunEndpointVerificationSchedulerwas also never registered as a runtime worker, so nothing producedendpoint.verifyrows at all. Registering it is what made the dead-lettering visible — the scheduler began producing work that immediately died. Both halves are fixed; either alone would have left the capability dark. That makes SIX instances of one defect in this program: D2's VerifyAddress with no producer, B2'sendpoint.renewwith no enqueue, B5's custody projection never written, J2's restore drill with no production caller, the unregistered verification scheduler, and this dead-letter path. The shape never varies — the capability is complete, its unit tests drive it directly, and the composition root does not reach it — and unit tests cannot catch it by construction, because they stand in for the caller that does not exist. Two guards now do:TestEveryDeclaredRuntimeWorkerIsActuallyStartedfails on any declaredRun*(ctx)scheduler that is not started, andTestNoAgentClaimableKindIsEverDeadLetteredwalks the claimable allowlist and fails on any kind the dispatcher would refuse. Both mutation-verified; the second names all five kinds when the fix is removed. Offline renewal resilience (A5, PARTIAL): a host agent now schedules its next rotation from the credential's REMAINING LIFE on its own clock, not from--rotate-everyalone. The old timer was the configured interval and nothing else, which let three ordinary situations end with an agent asleep past its own expiry: an interval set longer than the certificate lifetime, a control-plane outage spanning a whole interval so the single attempt inside it failed and the next was a full interval away, and an agent restarting while holding a credential already most of the way through its life. All three end identically — the agent wakes with an expired certificate and can no longer authenticate to renew it, which needs a person on the box, for a fleet whose value is not needing one. The next attempt is thereforemin(two-thirds of remaining life, --rotate-every), jittered ±10%. The min() is the guarantee and only one half of it carries any weight: two-thirds-of-remaining is strictly below remaining, so no configured cadence can push an attempt past expiry. A first draft also clamped toremaining - remaining/10with a comment calling that the load-bearing line; mutation-testing showed deleting it changed no outcome, and it was removed rather than left as dead code under a false claim. The jitter is not cosmetic: agents installed by the same automation hold near-identical expiries, and without spread a fleet renews in one second — arriving during recovery from the very outage that synchronized it. Local clock, deliberately: the agent that cannot reach the control plane is exactly the agent that most needs to renew early, and it has no other clock to consult. A host with a badly wrong clock renews at the wrong time, so the wrong clock is now made visible rather than corrected: every heartbeat carries the agent's clock and its answer carries the control plane's, and a drift pastagent_channel.clock_skew_warning(default 2 minutes) is named on the Agents page and inGET /api/v1/agentswith the measured drift, counted bytrstctl_agents_clock_skewed_totalfor theTrstctlAgentClockSkewalert, and logged by the agent itself. Renewal timing still uses the host's clock, and an agent built before this release sends no reading. A5's other items have since landed and their honest scope lives with the staged-upgrade entry below: rings run as dispatched, per-agent-targeted jobs when a campaign publishes artifacts; the agent's-self-upgradeexecutor verifies the pinned sha256, keeps the previous binary as.old, reports a signed receipt and restarts; and--demoexists. Agent/control-plane version skew IS enforced —agentProtocolInterceptorrefuses a handshake outsideMinSupportedVersion..MaxSupportedVersion— and was met before this program, so it is not claimed as new work. Constrained edge sub-CA (B6, THE one deliberate exception to AN-3/AN-4): a host with no path to the brain issues leaves locally under a delegated CA, and every bound that makes that defensible is enforced, not documented. Minted BY the central isolated signer over a CSR (the edge key never travels), with name constraints taken from the SEGMENT'S POLICY — never the request — in the certificate itself (permitted and excluded subtrees, exclusion beating permission), path length pinned to zero, and a 30-day ceiling on life that is refused rather than clamped. Default OFF: minting requires a declared segment's explicit opt-in, and the opt-in is one declaration carrying the pinned TPM attestation roots, the segment's identifiers, and a closed allowed-key-provider list. That list defaults totpm2only. An un-attested host is refused. In the default TPM2 lane, an attestation over a different key than the CSR's is refused (a TPM vouching for one key must not license delegating another). The delegation is revocable from the brain and its serial lives in the parent CA's issued ledger, so OCSP and the CRL answer for it with no new machinery; expiry is the certificate's own clock. Local issuance (agent-edge-issue) enforces the constraints read FROM THE DELEGATION CERTIFICATE and fails closed on an out-of-constraint name; every issuance lands in a journal in exactly the reconcile request's shape, andtrstctl edge delegations reconcile -f <journal>posts it when a path or a courier exists. Reconciliation re-verifies each leaf (signature chains to the delegated CA; names re-checked) and records an out-of-constraint leaf AS A VIOLATION — visible, never silently dropped and never silently accepted — while reconciled leaves enter the certificate inventory so there is no shadow estate. Key custody, stated exactly:trstctl-agent --edge-csrdefaults to a persistent TPM2 signing object and writes only an opaque public handle;--edge-attesthas a separately certified TPM AIK run TPM2_Certify over that exact object and CSR, verifies the resulting WebAuthn TPM envelope locally, and can write a complete reviewed CLI mint body;--edge-issuereopens the delegated key handle and asks the TPM to sign. PKCS#11 is a configured alternative whose shipping module creates a token object withCKA_SENSITIVE=trueandCKA_EXTRACTABLE=false. It must be explicitly allowed in the segment policy because the host's WebAuthn TPM attestation authenticates the host and CSR request, not the separate token object; the evidence therefore sayshost_attested_operator_claim, not hardware-key attested. Exportable software PEM custody exists only behind BOTH--edge-key-provider software --edge-allow-software-keyand an explicit segment-policy allowlist entry. Its durable evidence saysfile,key_exportable=true, andhost_attested_software_exception; the console renders that exception rather than borrowing the host TPM's stronger label. Neither hardware path silently falls back when its device/session is absent. Issuances made while the host is unreachable are invisible until its journal reconciles, and the console panel says so rather than rendering silence as inactivity. A host agent run with--edge-journal-uploadsends the journal over its mutual-TLS channel when it connects, after a heartbeat that follows failed ones, and when the journal grows; the control plane reconciles it through the same path as the CLI, counting each certificate against the delegation of that host that issued it, as the agent. Only a host-role agent whose channel certificate names the delegation's host reaches it, and<journal>.uploadedrecords the acknowledged prefix, so only new entries travel. Without the flag the CLI or console reconcile stays the path. Renewal is opt-in twice: the segment policy'srenewal_enabled(withrenewal_window_days, default 7) and the host agent's--edge-renew. Inside the window the TPM key signs a fresh CSR, TPM2_Certify vouches for it over the new challenge, and the agent asks over its channel; the control plane holds the renewal to the mint's gate (the same attestation verifier, the segment's current roots, constraints and provider allowlist), requires the same delegated CA key, and mints a delegation with the old one's lifetime that names it inrenewed_from. The 30-day bound stays: an expired, revoked or already renewed delegation does not renew, and a refused attestation leaves the delegation to expire. Renewal attests with TPM2_Certify, so it serves the TPM2 lane; a PKCS#11 or software delegation asks again through the attested mint. Proven against the TPM fixture end to end and, for the attestation itself, against swtpm. The exact TPM2, PKCS#11, and software-exception command/configuration lanes are documented in Disconnected edge CA key custody. Authority agreement (C4): the pipeline now has a PRODUCER. A scheduled round observes the store-backed authorities, and when two signed digests commit to different state the scheduler hands every disagreeing pair to a sink that builds the witness, signs it in the isolated signer, records it in the event ledger (with both signed digests on the event, so offline verification needs nothing else) and passes it to quarantine admission. An earlier audit found the opposite: rounds were never scheduled,witness.Recorder.RecordWitnesshad no production caller, and every ingredient of "we checked and your authorities agree" shipped except anything doing the checking — thecollecting=falsestate and its wording exist because of that finding, and they still guard the zero-schedule deployment today. Authority agreement (C4, core reconciliation):GET /api/v1/reconcile/agreementand a Posture console panel report whether the configured authorities agree about what was issued, and where they do not. XREC already built the hard part — canonical records per authority, signed Merkle digests over them, and witnesses naming the exact differing subset — and had nowhere to say so: the drift projection accumulated every witness class per authority and was never exposed on the runtime, so no route could read it. The system could detect that two authorities disagreed and could not tell anybody. The design constraint is that SILENCE RENDERS AS AGREEMENT. "0 open witnesses" looks identical whether reconciliation found nothing, is unconfigured, or has consumed no events yet, so the surface refuses to show a reassuring zero it cannot stand behind. A deployment with no reconciliation runtime reports that fact and states plainly that it is NOT a report that the authorities agree; a configured but cold projection says its counts are not yet evidence; and every response carries the replay watermark beside the count, because a projection lagging the event log reports an old world confidently. Resolution time is a MEDIAN over a stated sample size — one witness left open over a weekend drags a mean into fiction, and a median over a single sample is not a trend. Scope, stated exactly: this surface is READ-ONLY. Remediation is deliberately not reachable through it, because a corrective operation runs only through a plan verified inside the signer, and a REST route that could start one would be a way around that verification wearing the same URL prefix as a status page. Authorities are ranked worst-first; an authority that has never been collected from does not appear in the table at all, which is whyconfiguredis a field rather than something a reader is left to infer from an empty list. Relay migration parity (E1, OPEN — four migrated, three architecture exceptions, six unimplemented): the connector catalog and Connectors console publish the thirteen-family source-plan denominator, not the smaller list the current relay binary happens to execute. The exact families are a10, AWS ACM, Azure Key Vault, Cisco, Envoy, F5, FortiGate, GCP Certificate Manager, Kemp, MySQL, NetScaler, Palo Alto, and PostgreSQL; PostgreSQL and MySQL make the plan's formerly vague "+ DB/API variants" phrase executable. Every row carries a closeddisposition: four aremigrated, three are openarchitecture_exceptionrows, and six areunimplemented. The control plane REFUSES a migrated family's deploy when the tenant has a network relay enrolled. Before this, the A3 role stamp reserved an appliance deploy for a relay by writingrequired_agent_roleon the outbox row and nothing enforced it: the outbox claim query has no predicate on that column — it is read byClaimAgentJobswhen an agent asks for work, never by the dispatcher — while the dispatcher swept everyconnector.*row on a one-second ticker. The stamp was correct in the column and decided nothing. The refusal is CONDITIONAL on a relay actually being enrolled, and that condition is the design rather than a hedge. Refusing unconditionally would turn "this estate has not deployed a relay yet" into "this estate's appliance deploys stopped working", and would retire the only path with end-to-end proof through the served API — the DoD connector suite drives a10, cisco, kemp and netscaler through the control plane to their device doubles and enrolls no relay. So what is promised is narrower and keepable: if you run a relay, the control plane will not do its work behind its back. An estate with no relay deploys exactly as it did before. A failed relay-presence lookup DEFERS rather than falling through, because falling through is the direction that silently removes the guarantee. Migrated today: a10, kemp, netscaler, f5 — device proof, rollback, readback, a published support row, a relay deploy proven byte-for-byte against the device double, and the refusal above. F5 closes the last gate it needed: an F5 HA pair keeps its certificate objects in separate stores, so a deploy that reached only the active node reported success while the standby served the old certificate until a failover surfaced it as expired. The relay now drives anHAPairover both peers when a peer endpoint is configured — a deploy must reach BOTH or it fails, a rollback re-binds both, and a readback reports the pair serving only when both peers are bound to the deployed certificate; the two-peer deploy is proven end-to-end throughrelay.Executeagainst two device doubles. cisco migrated once a target could name its ISE node and certificate roles (node_hostname,certificate_roles; A122): its deploys then import fingerprint-named ISE system certificates bound to those roles, a rollback moves the roles back, and readback reports the certificate serving them; a target without a node keeps the legacy import and reports both as not configured. fortigate and paloalto migrated once a target could name the binding to move (bind_to;ssl_tls_profilewithcommit_admin): the same binding is what rollback re-points and readback asks about, and a target without it reports both as not configured. The parity surface and console now say this explicitly (cp_retained+ a scope note) so "not migrated" cannot be read as completion; each family's support-matrix known limits state the same constraint. These rows KEEP E1 OPEN. Re-check on new PAN-OS / FortiOS / IOS-XE majors: a vendor API that grows an addressable installed object re-opens that family's migration through the same gates, not around them. Of the six families the source plan omitted, the three cloud stores now have relay execution (A59): AWS ACM, Azure Key Vault and GCP Certificate Manager carry network-relay constructors, a relay execution proof against their API doubles, and the control-plane refusal while a tenant has a network relay enrolled; where no relay is enrolled the control plane's egress-guarded client remains their executor, for deploys, previews and GCP's re-bind rollback alike. GCP Certificate Manager passes every gate and ismigrated; ACM and Key Vault are the twoarchitecture_exceptionrows because their APIs cannot express a re-bind or a readback (A58). Envoy, PostgreSQL and MySQL execute on co-resident host agents and stayunimplemented: the database connectors write the server's own TLS files and run a local reload, which is not relay work by nature, and Envoy is host work by default; an Envoy target that declaresvantage: networkis driven by a relay instead (A61) without that making the family migrated, since SDS cannot express a re-bind or readback. Their rows name those facts instead of disappearing from the denominator. The executor census remains a separate fail-closed list: publishing an E1 row does not let an agent claim work it cannot construct. Device-generated CSRs are a custody choice, not a migration gate. An F5 BIG-IP (a single device), NetScaler, A10, PAN-OS (with an SSL/TLS service profile) or Cisco ISE destination (withnode_hostnameandcertificate_roles) can declarekey_generation=applianceand pin a network relay: the appliance generates the key and a CSR, the control plane signs the CSR, and the key never leaves the appliance. Otherwise the control plane generates the key (under the tenant's control-plane key-generation switch), seals it to the deploy job, and the relay redeems and installs it. The appliance operations are proven against the management-API doubles, not hardware, and the PAN-OS certificate generate and CSR export shapes need a firewall to confirm; each renewal's key object stays on the appliance and nothing prunes them yet; a PAN-OS renewal picked up again after its certificate import stops instead of finishing, and the same holds for a Cisco renewal picked up after its bind. ISE names each request itself, so the relay keeps ISE's request ID for the job on the control plane before it asks for a signature and a retried attempt reuses that request; the ISE node must already trust the issuing CA. Readback answers for every migrated family: kemp, netscaler and a10 were listed as readback-capable before their connectors could read back, so their relay deploys reported an appliance that "did not answer"; the census is now checked against the code (internal/connector/readback_census_test.go), and a NetScaler certkey that no SSL virtual server uses is reported as installed but not bound. Served DR posture (J2):GET /api/v1/platform/dr-postureandtrstctl platform dr-posturereport when this deployment's backup was last VERIFIED — meaning its artifacts were re-hashed and matched — rather than when one was last taken. The distinction is the point: a nightly job that writes a corrupt file runs perfectly, and reading a manifest's recorded checksums back only proves the manifest agrees with itself, since one process wrote both in one pass. The failures this catches are silent — bit rot, a truncated copy, an artifact restored from the wrong directory — none of which change the manifest, and all of which are otherwise discovered during the restore that was supposed to save you. The posture readsTRSTCTL_BACKUP_DIRECTORYonly: a set copied to the object store (--full-backup-upload) is verified when it is fetched back (--full-backup-downloadre-hashes every artifact against its manifest), not by this served check or the scheduled drill. Three states are kept apart because only two are actionable: no backup directory configured (not a fault — many deployments back up through infrastructure this product does not see), a configured directory that cannot be read (a real finding, and the state that goes unnoticed until a restore), and one read that did not verify. A green verdict reports how many artifacts it covers and how many carried no recorded checksum at all, because "verified" over two of eleven artifacts is not the same claim as over all eleven. An artifact with no checksum is reported unverifiable rather than verified, and an EMPTY backup is never verified — green on an empty set is the most misleading answer available. Restore drills (J2): a drill restores the COMPLETE manifest into an isolated target through the same full-restore function production recovery uses. It creates and later drops a real ephemeral PostgreSQL database, starts a private file-backed JetStream beneath a temporary directory, copies the separately-custodied KEK and backup-decryption key into that directory, and redirects every restored signer, certificate, audit, socket, and archive path there. The production deployment is read only. A drill with its own simplified restore would prove the simplified one works, which is the one nobody runs at 3am; a drill aimed at live targets could cause the outage it is meant to rehearse. After event replay and both projection rebuilds, the drill imports and re-exports everyRecoveredFromPostgresBackuptable and compares exact table counts. It then starts the shipped signer over the recovered sealed key store, assembles a recovered control plane, and requires its real/readyzPostgreSQL, JetStream, and signer probes to pass. The attestation carries event count, independent-row/table counts, restored artifact names, and each health result.restoredis impossible unless the full set and every runtime predicate pass; a missing required artifact or an event-only replay isfailed. The attestation records FAILURES as readily as successes, because without one "no attestation" is ambiguous between "nobody ran a drill" and "the drill failed". A restore that completes having replayed ZERO events is recorded as a failed drill, not a fast one — it proved nothing and is the result most likely to be mistaken for reassurance. A backup that does not verify is not restored at all, so the cause reads as a sentence rather than being buried in a restore error. RPO is measured from the backup's own manifest, so it is what was achievable with the artifacts on disk rather than what was configured — the two diverge precisely when a backup job has been quietly failing. RTO is a FLOOR and the attestation says so in its own signed bytes: real recovery also includes provisioning, networking and people, and that caveat is signed alongside the number so it cannot be edited off a document whose signature still verifies. A deployment that has never drilled serves no drill rather than a zero-valued one, since zeros render as an instant, complete recovery. Each scheduled result now becomes a versionedbackup.restore_drill.recordedevent per live tenant. The deployment-wide attestation is signed through the isolated signer's purpose-constrainedaudit-exporthandle with the exactrestore-drill/v1evidence domain; the signed body includes the complete attestation, recovery-objective thresholds, derived alert reason, signer key id, algorithm, and public JWKS. The projector verifies against the deployment-trusted key before writing an immutablebackup.restore_drillevidence row, so changing an outcome, detail, limitation, metric, threshold, required alert, signer, or verification key is rejected.last_drillis derived from the newest durable row anddrill_historyexposes the newest 50 through the authenticated tenant API, generated SDKs,trstctl platform dr-posture, and the Platform console. A cold projection rebuild reproduces the history; it is no longer a process-memory pointer. The console shows signature status and signer identity and downloads the whole portable signed JSON evidence object. The drill RUNS:RunRestoreDrillScheduleris registered as a runtime worker and fires onbackup.drill_interval(daily by default,"0"to disable, and an unparseable value fails startup rather than silently defaulting; the default is resolved at config-parse time so a zero reaching the scheduler can only mean the operator asked for none — an earlier cut applied the default in the scheduler too, which made the documented"0"run the drill daily instead of disabling it). It is worth recording that the first cut of this work did not have that. The drill, the attestation, the ephemeral target and the endpoint were all built and all tested, and nothing in the running binary ever called any of it — the fourth instance in this backlog of a capability that is complete, tested, documented and unreachable, after D2's VerifyAddress with no producer, B2'sendpoint.renewwith no enqueue, and B5's custody projection that was never written. Unit tests cannot catch it by construction: they are the caller that production lacks. Two tests now assert the wiring itself, and the DR panel on the Platform page reads the attestation the scheduler writes rather than a second source that could agree with nothing. A deployment that CANNOT drill — no external PostgreSQL to build a throwaway database in — records a skipped attestation saying so, rather than reporting that it has never drilled. Those are different facts, and the second is the one an operator would read as an oversight worth chasing. Failed and skipped outcomes create critical/warning alerts respectively. A successful restore whose measured backup age exceedsbackup.drill_rpo(default 24 hours), whose isolated restore time exceedsbackup.drill_rto(default one hour), or both, creates an objective-breach warning. The signed evidence row and its idempotentnotification.restore_drilloutbox intent are projected in one tenant transaction, so a crash cannot commit a red drill while silently dropping its alert. The scheduler logs structural/signing failures instead of discarding its returned error. Enrollment diagnostics (I4): refused ACME, EST, SCEP, and AD CS enrollments now produce a diagnosis naming the protocol, the step that failed, a cause from a CLOSED set, and a remediation. Each protocol emits at its shared refusal choke point, so a new refusal cannot silently bypass the recorder. AD CS keeps only an allow-listed stable HRESULT (0x80094800or0x80094801); its free-form upstream status is never retained because IIS or a proxy can echo submitted credentials. The design constraint is that it must DECLINE rather than guess. A tool that says "your DNS record is missing" when the responder was unreachable sends an operator to the zone file for an hour, and they will doubt it afterward on the occasions it was right. So the classifier matches only unambiguous evidence — RFC 8555 problem types, EST status codes, the two stable AD CS error phrases — and returnsunknownfor everything else, which is a first-class answer carrying NO remediation rather than a fallback. SCEP'sbadRequestis deliberately unclassified: it is the value a SCEP server returns for most refusals, so mapping it to any single cause would invent a diagnosis from a value that carries none. "Unreachable" is detected from typed network errors, never from error wording, so an error that merely mentions a connection is not reported as one failing. All three properties are mutation-verified. Diagnoses are served atGET /api/v1/enrollment/diagnostics, available astrstctl enrollment diagnostics, and shown on the Protocols console — where an unclassified failure renders "cause could not be established" rather than an empty remediation cell, because a blank reads as a rendering bug and the honest answer is a real one. Repeats collapse only when the stable diagnostic id names the same protocol, operation, identity, and endpoint. Two devices refused at the same step remain two rows; a retry storm for one exact operation remains one counted row. Each observation is a versioned immutableenrollment.diagnostic.observedevent whose envelope carries the request tenant. PostgreSQL projects the exact refs into a FORCE-RLS table and retains the newest 200 operations per tenant. Version-1 history replays under a deterministiclegacy:id; version 2 carries the exact id and refs. Restart, snapshot restore, and full replay therefore reproduce both the refusal and its verification link instead of clearing a process-global map.POST /api/v1/enrollment/diagnostics/{id}/prove-fixedrequirescerts:issueand anIdempotency-Key. It appends an immutable queued receipt, then projects the link and writes one network-roleendpoint.verifyintent in the same PostgreSQL tenant transaction. If the event append wins but that transaction rolls back, startup reconciliation recreates the exact intent from the event. The intent contains one explicit deployment-target address and server name; it never guessesSAN:443and never treats the enrollment server as the workload endpoint. The operator must first retry enrollment successfully. A newer active certificate issued after the refusal supplies the expected fingerprint; without that evidence the action returns409and queues nothing. A real relay performs the TLS handshake;verifiedmeans the signed transcript reached that endpoint and matched the expected certificate.divergedandunreachableremain red, and a queued row never reads as success. The result is available throughGET /api/v1/endpoints/verifications/{id}, its matching CLI command, and the Protocols console's signed-evidence link. While a row is queued, the console performs at most 30 bounded read-only refreshes and stops immediately when a terminal signed result appears.The ordinary offline support bundle remains tenant-data-free by default. An operator can explicitly add authorized, redacted aggregates with
trstctl support-bundle --include-enrollment-diagnosticsplusTRSTCTL_URLandTRSTCTL_TOKEN. The fetched shape can contain only protocol/cause/actionable/count aggregates and an unknown count; tenant ids, times, diagnostic ids, and exact operation/identity/endpoint refs cannot fit in its type. The fetch rejects redirects, bounds the body to 64 KiB, and never copies an upstream error body or bearer token into its error. Relay revocation cache (R3): a network-role relay started with--revocation-cache-configserves issuer-specific CRL and OCSP paths to relying parties in one named segment. The JSON file declares a bounded listener, the segment, and multiple issuers; each issuer points at a public certificate file and may declare a CRL GET path, an OCSP POST path, or both. The old--crl-cache-listenflags remain a single-issuer CRL compatibility path and now require--revocation-cache-segment. Revocation checking is the part of PKI that fails quietly — a client that cannot reach a distribution point usually proceeds rather than refusing — so a segment with no route silently stops checking, and nobody finds out until a compromised certificate is used. The relay SIGNS NO REVOCATION OBJECT: it holds the CA's signed bytes and hands them over, so a compromised relay can withhold a CRL (visible: the fetch fails) but cannot forge one (which would not be). It verifies a fetched CRL against the configured issuer before caching, which is not the relay adding trust — a relying party checks the signature regardless — but the relay declining to store what no client would accept, such as a captive portal's login page. The OCSP path accepts only boundedapplication/ocsp-requestPOST bodies. It proves that the request's issuer hashes match the configured issuer, forwards the exact request, and accepts a response only after the issuer/responder signature, requested serial, nonce equality,thisUpdate, andnextUpdateall validate. Nonce-free responses are cached in a 4,096-entry per-issuer bound; nonce-bearing responses are never reused because a nonce binds one response to one request. Expired entries are evicted first. The relay returns 503 with no OCSP response bytes when the upstream is unreachable or its answer is stale, malformed, for a different issuer/serial, or incorrectly signed.IT FAILS CLOSED ON STALENESS, and that is the property the feature exists for. A stale CRL is dangerous precisely BECAUSE it still verifies: nextUpdate has passed and the signature is good, so a relying party accepts it and trusts a certificate revoked yesterday. Past nextUpdate the relay serves 503 and no bytes — not the stale list with a warning header, because a client that receives bytes will use them.
--crl-cache-gracecan extend the window, defaults to zero, and is deliberately an operator decision: serving a list the CA declared expired is a risk only they can weigh. A CRL whose number went BACKWARD is refused, since replaying an older list is how a revoked certificate comes back to life. A failed refresh keeps a still-valid cached list rather than discarding it. Both fail-closed properties are mutation-verified. Before opening its steady-state channel, the relay signs a normalized metadata-only row per segment, issuer, and protocol with the same certificate key used by mTLS. The server verifies the certificate signature, tenant/agent binding, timestamp, network role, and monotonic order, then event-projects it.GET /api/v1/revocation/cachesand Protocols → Revocation cache by segment show fresh, stale, empty, and error separately. Upstream URLs, issuer bytes, OCSP requests/responses, CRL bytes, and credentials never enter the heartbeat, event, API, or console. A missing report remains unobserved rather than green. The repository proves multiple issuers, signature-valid local CRL/OCSP clients, nonce policy, stale refusal, signed assembled-agent heartbeat, tenant isolation, replay, snapshot, and cold rebuild against controlled responders. A domain-joined Windows AD CS deployment using its real CDP/OCSP endpoints remains external lab evidence rather than a repository claim. Enrollment proxy for dark segments (A4): a network-role relay started with--enroll-proxy-listen,--enroll-proxy-segment,--enroll-proxy-public-url, and--enroll-proxy-upstreamserves ACME, EST and SCEP on the LAN so hosts and devices with no route to the control plane can enroll through the one outbound pipe the relay already has. Stock clients (certbot, sscep, estclient) point at the stable public URL unmodified. The agent listener is HTTP; the public URL is HTTPS, so the segment's TLS-terminating load balancer terminates that public TLS connection before selecting a relay. Every redundant relay for one segment uses the SAME public URL. That detail is load-bearing for ACME: directory, account, order, authorization, challenge, finalize, and certificate resources are absolute URLs, and a different authority would send the stock client outside the relay path or strand it on the process that died. IT MAKES NO TRUST DECISION, and that is the design rather than a caveat. The relay sits inside the customer's network, which is where an attacker with a foothold already is, so a proxy that interpreted a challenge, cached an authorization, or spoke with its own agent identity would hand that attacker the ability to mint certificates. Requests are forwarded byte-identical — an ACME JWS is signed, so a proxy that reformatted it would invalidate every request — and the proxy attaches NO credential of its own: the client authenticates as itself through EST's TLS client certificate, SCEP's challenge password or ACME's account key, exactly as it would reaching the control plane directly. Both properties are mutation-verified. The control plane independently requires every ordinary ACME outer JWS protectedurlto equal that stable public request URL byte-for-byte before consuming its nonce. Missing URL, path, authority, and scheme differences fail unauthorized; the assembled two-process relay journey proves the relay preserves the same URL a stock client signed through failover. The forwarded paths are an allowlist, not a catch-all: a segment able to reach /api/v1 through a relay would hold the control plane's entire administrative surface, which is a far larger grant than "devices here can enroll". Two kinds of failover stay separate. Several--enroll-proxy-upstreamvalues let ONE relay choose another control-plane endpoint after a transport failure. Several relay processes behind the SAME public URL let the segment frontend select a secondary relay after the primary process dies. Both work because the proxy is stateless — an ACME order lives in the control plane, not in a relay — so the client retries through another path and its order is still there. A 5xx FROM the control plane is NOT a failover trigger: it is an answer, and retrying elsewhere would ask a second endpoint the same question while hiding the real error. Every heartbeat preserves the relay's segment, public URL, verified, unavailable, and not-yet-verified upstream counts, forwarded/refused/failure counters, and last-forward/last-upstream-failover times in the immutable event stream. The authenticated agents API and Protocols console group those certificate-bound rows by segment AND public URL. That lets an operator distinguish no report, proxy off, no verified upstream, degraded upstreams, one relay, and actual relay redundancy. Counters reset when the relay process restarts; the projected timestamps remain durable evidence of the last observed activity. Third-party connectors in the relay (E4): a relay started with--connector-plugin-direxecutes signature-verified WASM connectors inside the customer's network, under a capability grant its own operator sets. The control plane never loads partner code. Every guarantee is re-proven in the new location rather than assumed to have traveled with the code: an unsigned, tampered, or untrusted-key module is refused AT LOAD and fails the whole runtime rather than being skipped — a relay serving a subset of its configured connectors would be missing precisely the one somebody tampered with. An out-of-grant operation is denied at runtime and FAILS the deploy, so a module reaching outside its grant cannot report success. Both properties are mutation-verified. A module reads its deploy's public inputs (input.read: the certificate, the fingerprint, the target and its configuration) and asks the relay to make its HTTP calls (net.http): the relay adds the target's credential, splices the private key into the body by reference, and removes every redeemed value from the reply, so the module deploys as the appliance's administrator without holding the password or the key. The control-plane plugin path does not offer these two capabilities; third-party connectors that call appliances run on relays. Everything is operator-owned: the modules, the publisher keys, the pinned build digests and the capability grant. None is read from the module or pushed from the control plane, because a publisher who could widen their own grant by editing a file they ship would make the sandbox a formality. A directory with no trust keys is refused, and an unset grant is refused rather than defaulted — a grant nobody set and a grant that permits nothing are indistinguishable at that point, so the reading that cannot surprise anyone is the one that refuses. AUD-34 adds a signed, metadata-only census to that boundary. The census comes from the exact verifier-and-loader that instantiated the module; it names the stable plugin name, verified module and publisher fingerprints, the fixednetwork_relay_wasmexecution context, and the effective grants plus normalized constraints. The relay signs the tenant ID, its certificate common name, that normalized list, and the issued-at time with the same key used by its mTLS certificate. The server rebuilds both identities from the authenticated certificate, verifies the signature, and refuses a tampered, stale, or cross-tenant report. A host-role certificate cannot report relay plugins. Accepted reports enter the immutable event stream and rebuild the newest tenant-isolated projection after restart. The connector catalog API and the Connectors console show each reporting relay separately. A signed empty census means “this relay loaded no third-party modules”; no report means an older agent cannot make the claim. The census is also what routes work (A63): a deploy or preview for a connector no shipped family owns is stamped for the network role when a relay in the tenant has reported carrying that plugin, and the control plane defers the row to it; without such a report the control plane's own plugin surface remains the executor. The report contains no module bytes, publisher keys, credentials, or secrets. The control plane still does not distribute modules or operator trust keys. Third-party connectors also never receive redeemed credential material; a module that could read an appliance password would make the sandbox decorative. Live appliance readback (E2): after every appliance mutation the relay now asks the device what it actually has, alongside D2's TLS handshake against the served listener. The two are not redundant and only together locate a fault: a handshake showing the old certificate with a readback showing ours-and-bound means the VIP is fronted by something else; with a readback showing theirs it means the binding never moved (the deploy patched the wrong profile); with a readback showing nothing it means the deploy did not take. A handshake failure alone reads as "the deploy failed", and the most common cause is the middle case — a binding the deploy never touched, on a device that reported success at every step. A readback that contradicts a passing handshake DOWNGRADES the outcome to verify_failed, because a handshake can pass against a second listener while the object this deploy installed is not the one bound. Verdicts are a closed set: serving, installed_not_bound, diverged, absent, unknown — andunknown(the device named an object but not which certificate it is) is NOT a pass. Readback is available only where the API can address an installed object separately from uploading one: f5, kemp, netscaler, a10, and fortigate / paloalto / cisco targets that name their binding (bind_to;ssl_tls_profile;node_hostnamewithcertificate_roles). A fortigate, paloalto or cisco target without its binding cannot be asked, which is reported as no readback rather than as a clean one. The readback travels inside the deploy's signed verification report, as its verdict and sentence (appliance_readback), and the control plane admitsverifiedonly when the handshake passes and the readback confirms; the verification receipt names a contradicting readback (appliance_readback_<verdict>) and keeps the device's sentence. Earlier builds appended the sentence to that report's JSON, so a deploy with both a verify address and a readback answer was refused as undecodable. Capability disclosures (K3): the connector support matrix and the per-issuer capability matrix are both generated by one tool from the censuses the API serves, with a Go test failing on drift in either. The issuer matrix gained anissue_provencolumn that is deliberately SEPARATE fromissue, because "we implement this authority's API" and "we have run an issuance against it" are different claims and a table showing only the first states the stronger one everywhere. All thirteen advertised authorities now have high-fidelity protocol issuance proof inTestDODExternalCAUniversalProductionAssembly: each driver is constructed by the production server assembly, talks to its own nonce-bound authority-protocol substrate in the expected wire order, and returns a chain that is independently verified. That proves the adapter and custody path. It is not live vendor-account certification and does not prove a customer's policy, entitlement, network, appliance, or service version; operators must still run one scoped issuance against their configured account before depending on it. The revoke and unattended-DV columns remain separately census-checked against the implementations. Every row names the tests backing it, and a manifest-derived guard fails both when a row outruns its proof and when new executable proof remains incorrectly labeled "not tested." Support matrix (E3): docs/features/connector-support-matrix.md is GENERATED from the same census the API serves, and a Go test fails if the two diverge — so a capability cannot be removed while its published row survives. It deliberately publishes no firmware compatibility range: nothing in this repository runs against a physical or vendor-hosted device, so a version claim would be unbacked, and it is precisely the line an operator would plan a migration around. What it publishes instead is the API or local execution contract each connector speaks, the operations exercised by repository tests, and the family's known limits. Eleven rows are proven against faithful management-API doubles: the seven appliances, the three cloud stores and Envoy's SDS management surface (A59, A60). PostgreSQL has no device API to double; its proof instead drives the connector against the pinned bundled PostgreSQL runtime started with TLS on, and a real SSLRequest handshake sees the renewed certificate after the allowlisted pg_ctl reload. MySQL keeps sandboxed filesystem and process operations as its only proof: no MySQL server is bundled or available to the repository tests, and its row says so. A guard fails if a version claim appears, and all thirteen rows explicitly say no external target was tested. Rollback, restated after that work: cisco re-binds only through ISE's system-certificate calls, which a target selects by naming its node and certificate roles (A122); the legacy import is a single call that both uploads and installs, so a cisco target without a node still cannot re-bind. fortigate and paloalto re-bind only when the target names the binding to move (A44), and the same is now true of gcp-certificate-manager through a certificate map entry (A58). aws-acm and azure-keyvault stay absent from the census for the same kind of reason: ACM re-imports in place under one ARN and keeps no previous version, and Key Vault keeps versions but serves the newest by name with no call that makes an older one current. The census tests pin each absence in both directions and the refusal names the family's reason.buildRunDepsconstructs the operator-selected production registry; strict target schemas bind endpoint/filesystem/process and same-tenant secret references before a served issue/deploy route can enqueue work. Credential-bearingconnector.deploypayloads exist only while needed, travel through the durable outbox, and are wiped after delivery. The acceptance proof requires provider-specific mutation plus independent external readback before a connector counts as served. Inventory: nginx, Apache, Caddy, Envoy, IIS, HAProxy, F5, NetScaler, A10, Kemp, Cisco, FortiGate, Palo Alto, Postfix, Traefik, AWS ACM, Azure Key Vault, GCP Certificate Manager, Java keystore, PostgreSQL, MySQL, RabbitMQ, Elasticsearch, and Tomcat.Discovery control plane + relay-owned network/SSH, cloud-certificate, CT-log, and drift execution: the running binary serves discovery sources, schedules, and runs under
/api/v1/discovery/*— create/list a source, create/list a schedule, queue a run (idempotent, deduplicated byIdempotency-Key), read runs and findings (keyset-paginated), and readGET /api/v1/discovery/monitoringfor the centralized continuous-monitoring view across sources, schedules, last runs, findings, and inventory counts.GET /api/v1/certificates/healthandtrstctl certificates healthserve the estate-wide expiry/source dashboard over the same inventory projection, including certificates issued elsewhere and later imported or discovered. Queuing a run is an immutablediscovery.run.queuedevent. For network and SSH sources the same transaction resolves the bounded target list and stamps the declared segment, requirednetworkrole, and optional exact relay ID into both the event and outbox command. The control-plane dispatcher refuses to dial those targets and leaves the row claimable. An enrolled relay executes the sweep on its bounded worker lane, signs the result receipt, and the control plane validates the command/report binding before projecting metadata-only findings, counts, segment freshness, and the executing agent ID. For a cloud_certificate source the worker executes AWS ACM, Azure Key Vault, and GCP Certificate Manager enumeration through credential references — served cloud-certificate discovery execution. For a ct_log source the worker polls configured RFC 6962 log fixtures or public logs, checkpoints each log, and records unexpected issuance asct_unexpected_issuancefindings — this is served through the served discovery worker, queuing notification alerts the same way expiry alerts do. CT monitoring is a headline capability on the Discovery workspace, not a footnote: one surface carries the watched-domain and log watchlist, per-log checkpoint/health state (so you can see whether a log succeeded, failed, or has never been reached), retired audit history that is no longer polled, unexpected-issuance findings with certificate detail, and a one-click hand-off to the rogue-certificate remediation path. It previously appeared there only as a single count shared with drift detection, so the capability was effectively unfindable. It covers only the domains and logs configured — a domain you have not listed, or a log you do not poll, produces no finding, and the surface says so beside the counts so an empty list is not read as an all-clear. Replacing a named watchlist retires URLs absent from the new source configuration in the source event's same tenant transaction; one failed active log does not discard a peer's successful checkpoint or finding. The console polls queued/running CT runs to completion, refreshes per-log checkpoints in place, follows the parent Discovery refresh, and shows a bounded credential-redacted diagnostic for a failed discovery run. It also only sees what a CA chose to log, which in practice means public issuance. For a drift source the worker compares configured credential paths against expected fingerprints/permissions and recordscredential_driftfindings through the same served discovery worker. For an ssh source the worker runs a non-invasive SSH host-key scan through the discovery outbox worker, applying the same reserved-address guard as network scans, recording metadata-onlyssh_keyfindings (fingerprint/key-type/ location) and never authenticating or storing private key material. A manual source records its supplied findings.CBOM scan and migration inventory:
POST /api/v1/cbom/scansruns the cryptographic bill of materials scanner against TLS endpoints and host config files, recordscbom.asset.observedevents, and projects tenant-scopedcrypto_assets.GET /api/v1/cbom/assetsreturns the inventory plus migration targets andmigration_progress. With PQC in the core, the targets are the concrete FIPS 203/204/205 algorithms andmigration_progresscounts which assets are already post-quantum-ready.Credential-compromise incident execution is part of Core: the old
POST /api/v1/incidents/executionssingle-identity mutation refuses with conflict and points to the H2 fleet route; it cannot prove exact trust scope, trust-before-leaf ordering, or signed live health before revocation. Historical evidence remains readable viaGET /api/v1/incidents/executions{,/{id}}. Automated remediation playbooks are also served without a commercial license:GET /api/v1/remediation/playbooks,POST /api/v1/remediation/playbooks/{id}/runs, andGET /api/v1/remediation/playbook-runs{,/{id}}cover revoke, rotate, and NHI right-size. Revoke and rotate run the identity lifecycle gate; they cannot carry an exact dual-control approval yet, so under dual control they refuse and the operator uses the identity transition route, which can. Rotate revokes the original only after the replacement's signed, verified delivery, so it serves an X.509 identity bound to a deployment target whose delivery an enrolled host agent or network relay verifies and signs. A control-plane connector delivery (a cloud store while the tenant runs no relay) carries no such signature: that rotation keeps the original valid and raises arotation-replacement-unverifiedfinding at its deadline, and the operator completes the change through the identity transitions. Owner-driven self-remediation is served throughGET /api/v1/remediation/owner-actionsandPOST /api/v1/remediation/owner-actions/{id}/accept: a bound owner can accept the CAP-POST-01 least-privilege recommendation, and trstctl records the sameremediation.playbook_run.recordedevidence plus aconnector.right_sizeoutbox intent. When the operator configures the matching tenant/connector binding, the shipped dispatcher authenticates to the entitlement service with a tenant-secret reference, sends the stable outbox idempotency key, applies the requested scope removal, reads the effective scopes back independently, and advances the queued connector receipt to delivered or failed. An absent binding and every unknownconnector.*outbox kind fail closed. The rule is simple: a queued receipt is not external-effect evidence. Only the verified delivered receipt is. SIEM/SOAR/chat/ITSM response dispatch is served throughPOST /api/v1/incidents/response-integrations/dispatch, which recordsresponse.integration.dispatchedand queues Splunk HEC, Jira issue, configured Slack notification, and ServiceNow Table API outbox rows in the same event-backed workflow. The served boundary is dispatch plus evidence. The separately packaged Splunk app supplies two disabled audit-feed correlation searches for renewal failures and revocation bursts, with tenant/index scope placeholders and immutable event-ID deduplication. Its owned scaffold evaluates the actual query text ad hoc and tests native audit-feed delivery and retries; it does not qualify Splunk's parser, scheduler, RBAC or product version. Those vendor-runtime checks remain unverified without an authorized Splunk environment. Custom incident-response correlations, Jira automation rules, Slack app/channel installation, arbitrary third-party SOAR playbook execution, and bidirectional ServiceNow ticket-status sync remain operator configuration. Fleet-wide re-issuance is served separately atPOST /api/v1/incidents/fleet-reissuance-runs: it freezes exact certificate/SPKI H1 trust consumers and ordered H2 member bindings before work, installs replacement trust before host-CSR issuance, waits for lease-bound signed trust/live receipts, revokes each exact predecessor only after verification, and seals terminal success/rollback as a compact JWS. The separately authorizedgame_daymode structurally refuses production owner/target environments. It has pause/resume/current-cohort rollback and evidence export routes under/api/v1/incidents/fleet-reissuance-runs/{id}, matchingtrstctl incidents fleet-reissuance *CLI commands, and the/incidentsconsole. Online break-glass atPOST /api/v1/breakglass/issueis conditionally served whenbreakglass.online_enabled=true: productionbuildRunDepsbinds the configured CA and public key to one persisted, purpose-constrained dual-control signer handle. The caller first opens an exact-request-bound ceremony, and distinct approvers use their own authenticated tokens at/api/v1/ca/ceremonies/{id}/approvals; the issue request carries no approver names. Only authenticated immutableca.ceremony.approvedevent actors in the configured roster count toward quorum, and the ceremony is consumed once withbreakglass.issued. Break-glass recovery readback atGET /api/v1/breakglass/issuesandGET /api/v1/breakglass/issues/{id}returns bounded tenant receipts and the exact signed online bundle to callers withcerts:issue. A lost response does not require reissuance; trstctl does not retain the CSR subject private key. Offline break-glass reconciliation remains served atPOST /api/v1/breakglass/reconcile. A disconnected signer uses its sealed, offline-only CA and a pinned public kit; two distinct custodians sign the exact CSR-bound request. Recovery checks their signatures, tenant, deployment, policy epoch, recovery CA key and leaf before recording the public proof as abreakglass.issuedaudit event. The disconnected runbook gives the setup, outage and recovery procedure.Real X.509 issuance: transitioning an identity to issued mints a leaf certificate from the assembled CA (its key held in the out-of-process signer) and records it in inventory. Exercised end to end in CI.
Attested X.509-SVID issuance: workload owners with
certs:issuecan self-serve attester trust sources at/api/v1/workloads/attester-trust-sources(create, list, get, replace, rotate, revoke, delete) before callingPOST /api/v1/workloads/attested-issuance. The route verifies a workload proof (aws_iid,azure_imds,gcp_iit,github_oidc,k8s_sat, ortpm) against tenant trust material or configured process defaults, signs a short-lived X.509-SVID in the isolated signer, recordscertificate.recorded, bindsattestation.bound, and returns the certificate plus verified subject metadata. Mutations requireIdempotency-Key, reads and issue are tenant-scoped, forged proofs fail closed, and offboarded trust sources mint nothing. A tenant/source command is recorded before a trust change reaches the event log; pending changes deny old proof while exact-event recovery completes. Completed commands retain a request-bound response snapshot for a lost-response retry, including a retry after a newer trust change. Subject erasure removes any affected response snapshot but preserves its opaque replay fence.Authentication and RBAC via scoped API tokens (
Authorization: Bearer), multi-tenancy with PostgreSQL row-level security, and a tamper-evident audit chain. A fresh boot fails closed (every route401s until a credential exists); mint the first tenant-scoped token withtrstctl token create --tenant <uuid>(writes through the store, prints the token once). OIDC, SAML, and LDAP / Active Directory login are served by the binary when their respectiveauth.*.enabledflags are set — see "Single sign-on" below — under the same RBAC and per-tenant scoping as an API token, with each user mapped to its real tenant. API-token auth remains the default when SSO is disabled. OIDC stays core; tenant SAML and LDAP require Enterprise SSO, inherited by Provider.SCIM 2.0 provisioning requires Enterprise SSO and
auth.scim.enabled— see "SCIM 2.0 provisioning" below for the route and event detail.Transport security (TLS), idempotency and the outbox, observability (
/metrics,/readyz, W3C trace headers), bulkheads plus per-tenant rate limiting, backup/restore plus disaster recovery, and safe schema migrations.Protocol parity hardening: served ACME supports the explicit ACME trust_authenticated profile mode for scoped EAB-bound internal issuance, plus account-keyed order/hour and concurrent-order limits. Served EST includes
/serverkeygen, RFC 9266tls-server-end-pointbinding, profile PathID dispatch, and an mTLS sibling route when configured. Served SCEP includes the SCEP Intune JWS challenge gate with tenant/CSR binding and durable single-use nonce rejection across control-plane restarts, per-profile RA material, and per-device rate limiting. The MDM SCEP control surface serves/api/v1/mdm/scep/status, policy CRUD, challenge-rotation evidence, CLI commands, and Protocols UI telemetry. Live SCEP validator trust anchors resolve enabled policysecret://references per decision;protocols.scep.intune_challengeis a bootstrap fallback. The selectablehmac-dynamicand Jamf modes still lack a served policy resolver. Enabled policies in those modes are now rejected by preview and create/update; disabled policies can be staged but do not authorize enrollment. A tenant-bound validator, durable replay fence, credential configuration path, stock client replay, and cold recovery remain required before enabling these modes.Revocation hardening: RFC 5280 named revocation reasons, bulk revoke routes, delegated OCSP responders, OCSP nonce echo, nonce-free OCSP response caching, and CRL ETag /
If-None-Matchcaching. CT precertificate/final-certificate submission is served throughPOST /api/v1/revocation/ct-submissionsand thect.submitoutbox worker.NHI decommissioning:
POST /api/v1/nhi/decommissionandtrstctl-cli nhi decommissionresolve departure, vendor-term, and inactivity signals to tenant-local managed NHIs, then drive event-sourced lifecycle revoke/retire transitions with per-item evidence.NHI posture — metadata-only, drawn from the unified NHI inventory; every route below reports findings with recommendations and has a matching
trstctl-cli nhi posture <name>command. Automatic action is opt-in per tenant: annhi_posturerule in the active lifecycle policy module may ask the leader's lifecycle sweep to revoke a managed identity (through the lifecycle gate, assystem:nhi-posture, with RFC 5280cessationOfOperation) or right-size its usage-backed unused grants (the right-size playbook's connector intent), at most 25 actions per tenant per sweep under the automation switch and maintenance deferral; each action or gate refusal is a playbook run naming the rule and the module digest, andGET /api/v1/nhi/posture/automationshows the plan. Under dual control an automatic revoke is refused and retried after 24 hours; the operator revokes through the identity transition with an approval:- Shadow (
GET /api/v1/nhi/posture/shadow): unmanaged, unregistered, and ownerless external NHIs from discovery findings; the Discovery findings table lets an operator claim a finding and run rotate/revoke/decommission/ remediation against the prefilled identity. - Over-privilege (
GET /api/v1/nhi/posture/overprivilege): granted scopes/permissions/roles compared with observed usage; rows without usage evidence are not classified as excessive scope. - Stale (
GET /api/v1/nhi/posture/stale): stale/dormant activity, unused credentials, and orphaned records; finding-backed rows hand off to revocation/decommission/remediation, andPOST /api/v1/nhi/posture/stale/handoff(trstctl-cli nhi posture stale handoff) hands up to 100 rows to a new accountable owner in one step: the ownership assignment, each row's previous owner and owner status in the response, and anownership.handoffnotice to the new owner and their escalation chain on the ownership notification channel. - Static credentials (
GET /api/v1/nhi/posture/static-credentials): long-lived credentials, static lifecycle markers, no-expiry credentials, and overdue rotation age; managed rows can start the served rotate/revoke/remediation workflow prefilled. - Exposure (
GET /api/v1/nhi/posture/exposure): internet-exposed NHIs, public endpoints/callbacks, plaintext transport, weak authentication, missing network policy, wildcard reachability, and insecure-deployment markers; URL query strings are sanitized and no credential values are returned.POST /api/v1/nhi/posture/exposure/probe(trstctl-cli nhi posture exposure probe, discovery:write, idempotent) probes those public endpoints and callback URLs live from a network relay: up to 50 endpoints per probe, most severe records first, as one discovery run under the product's exposure-probe source. The relay connects only to public addresses, checked on the address each socket would connect to, so a name that resolves to a private or loopback address is refused, not probed. Each endpoint becomes a finding (nhi-exposure-probe) with its outcome: a served TLS certificate, an unparseable one, a listener that accepted the connection without TLS, a non-public address, or no answer. The exposure posture then shows each record's latest probe (reachability) and counts probed and answering records. The control plane never dials the endpoints itself. A probe no network relay can claim (none with a fresh heartbeat, orendpoint.verifymissing fromagent_channel.claimable_job_kinds) is refused with409naming what to fix, rather than left queued. Provider-native attack-path validation (whether a cloud's routes and security groups allow the path) is what comes next.
- Shadow (
Rogue and non-compliant certificate posture:
GET /api/v1/revocation/rogue-certificatesandtrstctl-cli revocation rogue-certificatescombine unexpected CT findings with active certificate inventory policy checks (weak keys, expired active state, over-long public-TLS lifetimes, missing owners/issuers) and, whenca.policy.enabledis on, the tenant's owncertificate_posturerules from its active lifecycle policy module — metadata-only, returning projection evidence refs rather than PEM or key material. A posture rule is ordinary Rego in the same module (certificate_posture contains {"rule": ..., "severity": ..., "reason": ...} if { input.action == "certificate_posture"; ... }) overinput.certificate(subject, issuer, serial, fingerprint, DNS names, key algorithm, validity, lifetime and days-until-expiry, status, source, owner, CA, deployment location); each violation appears ascustomer_policy:<rule>with its severity and reason, the response names the module digest, andPOST /api/v1/policy/dry-runwith actioncertificate_posturepreviews a rule against sample facts. A rule that yields"remediation": "renew"makes the lifecycle scheduler queue a renewal (reasonposture:<rule>) for a deployed identity whose served certificate fires it, under the automation switch, maintenance deferral and a 100-per-tenant sweep bound; certificates without a managed, deployed identity are reported, never remediated. A module without posture rules reports nothing, and a certificate whose evaluation fails is counted inpolicy_evaluation_errorsrather than read as clean. CT coverage depends on configured monitored domains/logs. Each finding with a fingerprint listsinstalled_on: the connector targets trstctl delivered that leaf to, or restored it on by rollback (an adopted predecessor), and what can remove it there (host_purgethrough the enrolled agent,appliance_takedownthrough a network relay for f5, kemp, netscaler and a10, ornone).POST /api/v1/revocation/rogue-certificates/takedown(trstctl-cli revocation rogue-certificates takedown) queues that removal with a reason; an appliance object the target still serves is reportedtakedown_boundand left in place until it is re-bound, and onlytaken_downorpurgedclaims removal. Provider-native revocation remains separate; targets trstctl never delivered to, CT-only findings and cloud or FortiGate/PAN-OS targets have no executable takedown.Compliance and inventory reporting:
GET /api/v1/compliance/inventory-reportandtrstctl-cli compliance inventory-reportreturn the CAP-OBS-02 reporting view (frameworks, report types, routes, evidence refs, inventory counts, tenant schedules).GET /api/v1/compliance/evidence-packs/{framework}andtrstctl-cli compliance evidence-packserve signed CAP-CMP-04 framework packs for eIDAS, NIST SP 800-53/CSF, CMMC 2.0, FedRAMP, NIS2, and the existing PCI/HIPAA/CNSA/FIPS/Common Criteria/CA-audit frameworks. Thesoc2pack serves CAP-CMP-05 only when each CC6/CC7/CC8 prerequisite has exact tenant-scoped event/object evidence in the signed 90-day window; absent, stale, malformed, or wrong-tenant prerequisites are explicit gaps. The signed v2 manifest includes the tenant, window, event IDs/sequences/audit-chain digests, object refs, and missing prerequisites. CPA examination and trust-services scope remain residuals, and signature validity is not certification.GET /api/v1/compliance/nhi-reportandtrstctl-cli compliance nhi-reportreturn CAP-CMP-06 NHI compliance mappings for NIST SP 800-53, NIST CSF 2.0, PCI DSS 4.0, DORA, ISO/IEC 27001:2022 Annex A, FedRAMP, CMMC 2.0, eIDAS, and NIS2. These are evidence mappings, not legal certification — operator scope, policy, authorization packages, qualified status, national transposition duties, and auditor sampling remain explicit residual attestations.POST /api/v1/compliance/report-schedulesandtrstctl-cli compliance report-schedules createrecord idempotent, event-sourced audit-export schedule definitions;GET /api/v1/compliance/report-schedulesandtrstctl-cli compliance report-schedules listread them back. The licensed governance signer and a private mountedaudit.archive_dirare prerequisites to enable them. The local scheduler creates signed artifacts and event-backed run receipts with bounded retry and requeue. The archive stays the record of each report. A schedule'sdispatchalso announces every completed run: onecompliance.report_readynotice on each named notification channel (email, webhook, Slack, Teams, PagerDuty, Opsgenie, SIEM; up to 8; each must be configured for the tenant) and one ServiceNow ticket on an operator-approved public binding. The notices and ticket are queued in the transaction that records the completion, restored by outbox reconciliation, and never sent twice; the run receipt names them. They carry the run, its SHA-256 and its download route, never the report body, so a receiver downloads the signed bytes with its ownaudit:readcredential. A binding the operator withdraws later skips the ticket on later runs and the receipt says why. A Jira ticket and attaching the report itself to a notice are what comes next. Report files can contain tenant metadata. The full backup (--full-backup-dir) carries every completed run's report, encrypted, a full restore puts them back where their runs say and checks each digest, and the restore drill attests how many it restored; a log-only restore still cannot recover them. Reports are kept until removed: the opt-inaudit.report_retentionremoves each completed run's report once it is older than the window, unless a person placed a legal hold on the run, and records the run expired only after reading the archive and finding the report gone. A removal the object store refuses, or acknowledges without applying, retries in the outbox and the run stays unexpired. The run receipt and its SHA-256 remain after removal; a backup taken before it still holds the report. With the archive object store under S3 Object Lock (object_store.object_lock_modeandobject_lock_retention), every audit archive segment, report and backup set trstctl writes carries that retention, and the report retention reads an object's lock before removing it: a report the lock protects stays, and its run shows until when (a day at a time under a legal hold on the object). A person places an S3 legal hold on one archive object under the tenant's prefix (POST /api/v1/archives/legal-holds,archives legal-holds place) and releases it; the outbox sets or clears it in the bucket and the hold reads applied or released only after the bucket reads it back, or failed with the reason (a bucket without Object Lock). A report legal hold placed while the archive writes under Object Lock also holds the report's object, in the same event, and its release clears that hold.Audit chain anchoring (J1):
GET /api/v1/audit/export?format=serves the signed JWS bundle (default) plus NDJSON, CSV, Splunk HEC and Microsoft Sentinel record streams. The JWS response and browser download are one versioned JSON envelope containing the compact JWS, chain head, and complete anchor. NDJSON, Splunk HEC, and Sentinel end with a JSONchain_trailer; CSV adds the compatibletrstctl_recordandintegritycolumns and ends with an RFC-safe trailer row. Every saved form therefore retains the record count, archived-prefix predecessor hash, chain head, and complete anchor without depending on HTTP headers. When operators need continuous delivery rather than a download, tenant-scopedGET|PUT /api/v1/audit/feeds[/{id}]schedules bounded Splunk HEC or Sentinel batches through the durable outbox. It persists exact record lag, cursor, retries, safe terminal error codes, and collector request IDs; it does not store credential values or remote response bodies. This native feed is separate from the metadata-only OTLP log exporter and from incident-response SIEM dispatch. The Splunk app consumes this nativetrstctl:auditstream; genericsiemnotifications andtrstctl:responsepackets do not enter its two shipped searches. Collector HTTP acceptance alone is not indexing or correlation proof: independently read the stored event and search result. Whenprotocols.tsais enabled the chain head is countersigned with an RFC 3161 timestamp over a domain-separated imprint; the artifact carries the timestamp info, signature, TSA certificate, and CMS DER token. The shippedtrstctl-cli audit verifycommand checks all five saved formats offline. An auditor first downloads the public-only JWK set through authenticatedGET /api/v1/audit/verification-keys. The commandtrstctl-cli audit verification-keysdownloads it for separate--audit-jwkspinning. The TSA issuer root is separately pinned and supplied as PEM or DER with--tsa-root; the verifier never promotes the TSA leaf embedded in evidence into a root. It reconstructs the archived-prefix-aware chain, verifies the domain-separated RFC 3161 imprint, and can enforce a maximum anchor delay with--max-anchor-delay. A present CMS artifact must verify its own signature, exact signer and signed fields against the authenticated manifest. The receipt'stimestamp_artifact_verifiedflag distinguishes that proof from historical manifest-only tokens, which remain verifiable but have no portable CMS artifact. The TSA's own certificates are entries of the issuing CA's ledger, so its CRL and OCSP responder answer for them;POST /api/v1/protocols/tsa/certificates/rotate(trstctl-cli protocols tsa certificates rotate, Protocols console) moves the TSA onto a fresh signer-held key, revoking the outgoing certificate when given a reason, and a certificate rotated out earlier can be revoked later. Every export carries the CA's OCSP answer for each TSA certificate its timestamps rely on, captured when the export is made, and the verifier checks it against the pinned root: a certificate revoked for key or CA compromise refuses every timestamp it signed, one revoked for another reason refuses only timestamps at or after its revocation,--tsa-crlbrings a fresher CRL, and--require-tsa-revocationrefuses a certificate whose status no evidence issued after its newest timestamp establishes. Its limits: the evidence is the status when the export was made, so a revocation recorded later is seen only through a CRL the auditor fetches; the long-term evidence is the timestamps, their TSA certificates and this OCSP evidence, not a renewed archive timestamp (RFC 4998), so verification still depends on the TSA root and SHA-256 remaining trusted; and an export whose evidence could not be captured says so as an unchecked status. This independently pinned trust is what makes a BACK-DATED head detectable: a chain rebuilt to remove a record hashes differently, so no earlier token exists for it, and a freshly-taken token is dated long after the events the bundle describes. Scope, stated exactly: an anchor proves the head is no NEWER than the timestamp. One anchor does NOT prove the chain was complete when anchored — a record withheld before anchoring was never in the chain. Continuous anchoring at a known cadence is served for that: with Enterprise audit compliance and the TSA served, the leader timestamps each tenant's full chain whenever it has grown, onaudit.anchor_interval(hourly by default), and appends the anchor to the tenant's own log as anaudit.anchoredrecord;GET /api/v1/audit/anchors(trstctl-cli audit anchors, Audit console) shows the cadence, state and recent anchors. A selection by time or sequence alone exports the hashes its records have in the tenant's full chain (prev_hashis the chain head before its first record), sotrstctl-cli audit verify --tsa-root ... --anchor-cadence 1hre-checks every anchor inside it, refuses a record that entered the chain after an anchor it predates, and requires every record to be covered within the cadence. Its limits: a record withheld for less than one cadence plus the clock tolerance (five minutes by default) is not caught; a content-filtered selection cannot carry the series; an anchor taken before a subject erasure that changed the records it covers is reported as superseded, not verified; and anchoring stops while the TSA is unreachable (the status reads failing and the next pass catches up). A deployment without a TSA exports successfully and says in the payload that it is unanchored; that is a weaker claim, not an invalid one, and the export states which it is. The CLI verifies a plain signed JWS using separately pinned--audit-jwks, reportingaudit_signature_verified: trueandanchor_verified: false. It checks the signed slice's tenant, count, archived-prefix chain and head, without claiming an independent time or complete tenant history.--require-anchor, a supplied TSA root, or a positive maximum anchor delay refuses unanchored evidence. Timestamp failures never fall back to plain verification, and unsigned record streams still require TSA trust. Each tenant's continuous anchors are also appended, in order, to an RFC 6962 anchor log (internal/translog): every export carries a head of that log signed with the audit export key (its own artifact domain) and an inclusion proof for each anchor record it holds, andtrstctl-cli audit verify --audit-jwks ...recomputes each leaf from the anchor record, refuses an export from which an anchor between two it holds was cut (the log's entries would skip), and, given--previous-anchor-logwith the head kept from an earlier receipt and an export taken withanchor_log_from=<that size>, proves the log only grew in between. Its limits: the head is signed by the deployment's own audit key, so a split view is caught only by an auditor who keeps and compares heads — there is no external witness or gossip; and the newest anchors a lagging projection has not logged yet are reported as an unlogged tail, not refused.Signed invoice evidence (L2):
GET /api/v1/provider/usage-evidence(trstctl usage evidence, Platform console panel) serves a per-customer, per-period document that states its own completeness and carries THREE attestation layers, each of which can refuse: coverage (the durable metering store must vouch for the whole closed period), reconciliation (the certificates_issued meter is recounted from the identity_transitions projection of the event log, and a divergence names BOTH numbers and blocks signing — a counter that disagrees with the log must not be invoiced from), and the detached RS256 signature over the canonical bytes, made with the SAME audit-export key every other auditor-facing export uses. An unsignable document still returns 200 with its reason — "your usage is incomplete and here is how" is actionable where an error is not — and NEVER carries a signature; the absence is the point.?format=csvexports the table with the verdict and digest on EVERY ROW, because a spreadsheet slice detaches headers, and the JWS stays in the JSON document where a verifier can use it. Per-customer quotas are durable (provider_tenant_quotas), administered only through the provider plane behind the per-customer delegation gate (PUT/GET /provider/v1/tenants/{id}/quota), counted against the live read model, and ENFORCED at the served issuance transition with a structured 429 before the orchestrator accepts anything. Reconciliation sources, stated exactly:certificates_issuedis recounted from certificate issuance receipts;agentsfrom agent lifecycle receipts (first heartbeat or renewal before the period end, not offboarded before it; existing agents were captured once from their projected rows when the receipts arrived);tenantsfrom the core tenant registration projection;secrets_storedfrom application-secret mutation receipts, vouched for as independent only while the tenant holds no connector credentials, which have no event history and are then named in the line's note.certificates_storedstill has no independent record (receipts carry mints but not revocation or expiry) and its line says so. A checked gauge that diverges blocks signing like a diverging issuance count. The Provider console labels unchecked meters separately from a checked meter that diverged; the document's signable verdict determines whether finance may invoice. The certificates_stored cap is enforced at a served create path. The agents cap is enforced when a new agent first heartbeats over the mTLS channel: the server serializes the live quota/count check through event projection across replicas, returns ResourceExhausted before recording an over-cap agent, and lets an existing agent keep heartbeating after a cap is lowered. A one-time enrollment token and signed bootstrap certificate do not reserve a fleet slot; a cap changed after token mint still applies at the first heartbeat. The secrets_stored cap gates first writes through both the native and Vault KV APIs, and connector credential insertion; those objects share the same live stock count. Each tenant's first writes serialize across replicas. A durable secret command already admitted but not yet projected reserves a slot so a process crash cannot oversubscribe the cap. Signed usage gauges count only materialized objects, not reservations. Lowering a cap still permits exact idempotent retries, rotations, and deletion of existing objects. New writes over the cap return 429; concurrent first writes can return a retryable 503. The licensed tenant band gates the managed-offering create path:POST /api/v1/managed-offering/tenantscounts the hosted tenants already registered under the projection lock and refuses the next one with a 429 naming the band and the count once it is full (an unreadable count is a retryable 503, never a zero); the Provider plane's own provisioning enforced the same band already, andGET /api/v1/managed-offering/statusnow reportsmanaged_tenants_registeredbesidetenant_band. Quota-read/count outages return a retryable 503 for new capped creation rather than silently treating an unknown count as zero. The tenant-self-service route still serves only the caller's own tenancy and refuses a foreigncustomer_id. Provider staff use authenticatedGET /provider/v1/tenants/{id}/usage-evidenceinstead: the handler requires that exact customer'sreaddelegation before billing opens the customer's forced-RLS transaction, then returns the same signed JSON document or?format=csvfinance CSV.GET /provider/v1/evidence/verification-keyssupplies the separately fetched public-only JWK set; the Provider console reconstructs the exact displayed canonical bytes and shows Signature verified only after checking the RS256 signature, protected billing-invoice domain, key id, and digest. An undelegated customer or wrong-operation grant is refused before metering is touched. Reconciliation compares against the transitions projection, so a rebuild-in-progress can transiently refuse to sign — the refusal names the numbers, which is the correct behavior while the projection catches up.notification routing matrix and inbox: expiry, CT, drift, and workflow alerts resolve through the configured severity-to-channel matrix, dedup by per-subject/threshold/channel, and are inspectable through the served notification inbox with owner/approver escalation fields and dead-letter requeue.
MCP-vs-REST parity guard: the served MCP automation surface includes broad route-backed REST tools in addition to the named investigation tools, and CI fails when a served REST route is missing both an MCP mapping and an explicit allowlist.
Cert-ops console parity: issuer catalog and Test connection, operations queue, Alerts and delivery, richer certificate filters, dashboard charts, CTA empty states, onboarding carousel, and server-side command-palette search are in the served console.
The trstctl-cli drives this same served surface, including OIDC/SAML/LDAP
browser login, the React web console, and the AI/RCA/MCP surface covered in
their own sections below.
Tenant offboarding boundary
Tenant offboarding erases PostgreSQL read state for the target tenant: it
deletes every tenant-scoped table under that tenant's row-level-isolation
context, verifies zero residue, and returns a deletion attestation. The
tenant.offboarded event is replayed by projections, so a read-model rebuild
does not resurrect the tenant's PostgreSQL rows.
That is the limit of what tenant offboarding deletes — it is not a promise that
the append-only event log or a signed audit archive disappears when a tenant is
offboarded. Those records are governed by audit/privacy retention policy:
configure TRSTCTL_AUDIT_RETENTION plus an archive target
(TRSTCTL_AUDIT_ARCHIVE_DIR, or TRSTCTL_AUDIT_ARCHIVE_OBJECT_STORE with an
object store) for archive-backed retirement from the served audit view, and use Privacy Retention
for non-audit personal data pseudonymization. The underlying AN-2 event envelopes
stay retained for rebuild and disaster recovery. WORM/object-store archive cleanup and legal
hold decisions remain operator privacy/compliance work, but the product gives
that work a queryable evidence ledger: record deletion, legal-hold exemption, or
cryptographic shredding with POST /api/v1/privacy/archive-erasure-attestations,
inspect it with GET /api/v1/privacy/archive-erasure-attestations, or use
trstctl privacy archives attest/list. The attestation stores subject_ref and
redacted evidence refs rather than the raw subject.
An empty Change history search reports the tenant's sealed archived-prefix count when one exists. The API returns only its count and chain head, never the archive file, host path or object locator; retention does not silently turn an online audit read into an archive retrieval. An archive custodian must supply the signed JWS files (from the directory, or downloaded from the object store with any S3 client) and pinned verification keys for offline checking.
Retention elects one archiver per audit scope across control-plane replicas.
It pins the source generation until the signed segment, audit.archived event,
and PostgreSQL checkpoint are durable. Ordinary Provider quota and metering
writes can continue during that work; a history rewrite or full backup waits
for the active segment to finish. A large or slow archive can therefore delay a
privacy-erasure cutover or full backup. The retention worker reports its failure
for retry without claiming the unfinished segment is archived.
Built and tested, but not yet served by the binary
No current feature-map row is in this bucket. The empty section is deliberate: future library code that is built and tested but not yet wired into the served API must be listed here as library-only until its required wiring-census proof passes — Phase 2 residual breadth stays under Partial or Roadmap instead of being quietly promoted.
Conditional, partial, and residual boundaries
These notes explain the matrix's conditional and partial rows: some are served only when an operator enables or configures a backend, and some have a served spine with explicit residual work. The matrix above is the authority for whether the running binary serves a capability; this section records the operator-facing edges and follow-up integration work.
PCAS KEM data re-wrap
POST /api/v1/pcas/kem/rewraps creates a tenant-bound request, mints the
signed paired successor signing/KEM key record (key_ready), and then moves
the epoch's registered data keys through the isolated signer. The ciphertext
target is a KEM-wrapped data-key envelope: an operator fetches the epoch's
public encapsulation key (GET /api/v1/pcas/kem/public-key, CLI
pcas kem public-key), builds the envelope offline (pcas kem envelope
encapsulates to that key and seals the data key under the derived key; the
data key never reaches the control plane) and registers it
(POST /api/v1/pcas/kem/data-keys, pcas kem data-keys register). A re-wrap
request snapshots the predecessor epoch's inventory at acceptance and moves
exactly those envelopes; an envelope registered later is accepted, waits for the
next request, and keeps the retirement gate closed until it is moved. A stage
with no envelopes completes only when the request carries
empty_inventory_attested; otherwise it stays key_ready instead of reading as
migrated data. For each envelope the
signer's RewrapKEMDataKey opens it under the predecessor KEM key and seals
the same data key for the successor; the control plane then asks the signer to
read the new envelope back under the successor key alone (VerifyKEMDataKey)
and records an immutable per-target receipt bound to the request id and
digest, both epochs, the predecessor ciphertext digest and the data-key digest.
Each stage closes with a completion event over its target count; the job row's
data receipt turns complete, rewrap_complete becomes true and the
retirement gate opens. GET /api/v1/pcas/kem/data-keys serves the successor
envelopes for stock readback. Boundaries: the operator owns the inventory (the
status shows data_keys against data_keys_rewrapped per stage); the PCAS
console shows completion but has no registration workflow yet; the first KEM
epoch of an identity comes from an attested-empty request against its
predecessor epoch, since no earlier KEM key exists.
Workload-held co-sign mode (off by default)
trstctl-agent --workload-cosign-listen serves the PCAS workload-held predecessor
co-sign RPC with the workload's predecessor key, and the control plane now calls it.
Register the identity's genesis with predecessor_custody: "workload" and the
agent key's public half (public_key_pem); the anchor records that key instead of
minting one in the signer. Configure the agent's endpoint under
pcas.workload_cosigners (unix:/path for a co-located agent, or host:port with
the control plane's client certificate, the agent's CA, its SPKI pin and server
name). The identity's first succession request then runs the FIG. 5 flow in the
succession worker: the control plane forms the commitment over the agent-held
predecessor and a successor the isolated signer generated, obtains the predecessor
co-signature over the co-sign RPC, verifies it against the commitment it supplied,
has the successor sign inside the signer, binds the record to a tenant-root
transparency-log inclusion proof (mandatory for this mode) and records it with the
same transaction every succession uses. The agent refuses commitments for any other
identity, tenant or deployment. After that first succession the successor lives in
the signer and later successions are signer-held. Boundaries: one endpoint per
identity, no console flow yet, and the operator supplies the agent's public key at
genesis. The listener authenticates callers:
host:portrequires pinned mutual TLS (--workload-cosign-tls-cert,--workload-cosign-tls-key,--workload-cosign-peer-ca,--workload-cosign-peer-pin). Without them the agent refuses to start and never reads the key.unix:/pathcreates the socket0600in a0700directory and admits only--workload-cosign-peer-uid(default: the agent's own uid). Platforms that cannot report a peer's uid, such as Windows, refuse a unix socket; use a pinned-mTLS listener there.- Requests are limited to 64 KiB, 16 concurrent streams and 8 in-flight calls, with a 10-second deadline.
Agent certificate renewal keeps revocation and identity
Both renewal paths, POST /enroll/renewal on the agent mTLS listener and the agent
channel's Renew, refuse a revoked agent certificate, an offboarded agent, and a CSR
that names a different agent than the presenting certificate. Renewal rotates a key; it
never restores a revoked identity or changes which agent a certificate names. Such
agents get 403 with the reason and must be enrolled again with a new bootstrap token.
Revocation and offboarding first commit a tenant-scoped removal command. That row
blocks every new agent RPC on an already-open connection before the event projects;
startup and the bounded recovery worker publish and project its exact retained
event after an interrupted attempt. Agent channel refusal is a direct admission
check, independent of CRL or OCSP distribution to other relying parties.
The CA calendar: year-scale hierarchy expiry
Leaf expiry alerting runs on 7/30/90-day windows. That is the right clock for a
leaf and useless for a certificate authority: replacing a trust anchor means
getting the new one into every relying party first, which is a quarters-long
program, so a 90-day warning arrives long after it could have helped. Nothing
evaluated ca_authorities.not_after at all before this, and the CA console
printed the raw date and called anything past 90 days healthy.
CA authorities now run on their own clock. The leader lifecycle sweep walks every active authority against year-scale bands — 36, 24, 12, 6 and 3 months — and alerts once per band an authority crosses into, with severity scaled to runway: a planning signal beyond a year, a warning inside one, critical inside three months or already expired. The band already notified is recorded on the authority and only ever tightens, so a repeated sweep is silent, each tightening re-fires, and a clock skew cannot replay an alert the operator already saw.
The second check is the one that hides. A CA cannot issue a leaf that outlives
it, so once an authority has less life left than the validity its leaves are
issued with, every new leaf is silently truncated to the parent's expiry.
Issuance keeps succeeding; certificate lifetimes become shorter until something
downstream rejects one. That case raises ca.validity_compression rather than a
plain horizon alert, because the fix is different: renew or re-key the authority.
Exact contract. Alerts land on the notification.ca_horizon outbox destination
as ca.horizon or ca.validity_compression, carrying the authority, its band,
how many active certificates chain to it, and the renew-by date; they fan out
through the same operator-configured channels as expiry alerts. The v2 immutable
decision is appended first; its alert intent and band stamp then commit in one
tenant transaction. Startup and each CA-calendar sweep reconcile a retained
decision whose database transaction was interrupted, including when a tail
projector stamped the band before the intent was queued. A broker append failure
leaves neither a stamp nor an intent. The reference leaf validity is
lifecycle.leaf_validity (TRSTCTL_LIFECYCLE_LEAF_VALIDITY, default 2160h /
90 days) — a yardstick for horizon reporting that caps nothing.
GET /api/v1/ca/authorities carries a horizon object per authority, and the CA
Hierarchy console shows the band, the renew/re-key-by date, and the truncation
warning. GET /api/v1/certificates/health resolves beyond 90 days into 180-day,
1-year, 2-year and 3-year bands; later now means beyond three years.
The scheduler appends ca.authority.horizon_alerted v2 events for new horizon
decisions. Builds before the closed v1 privacy schema was registered could enqueue
an alert and stamp its band, then fail the audit append; a broker outage in that
older SQL-first flow could do the same. Those historical missing events cannot be
treated as present: inspect the retained notification outbox and delivery evidence
for those bands. V1 events are audit-only history; v2 events carry the exact alert
facts needed to recover an interrupted outbox command without recomputing it. A
subject erasure rewrites any still-retained v2 notification outbox command from
the rewritten event before reconciliation may execute it again.
Two edges that used to be silent now speak. An authority with no recorded
not_after is evaluated from its own stored certificate: when that parses, the
sweep's decision records the derived expiry on the row and alerts in whatever
band it lands in; when it does not, the sweep raises one ca.horizon alert at
warning severity, stamps the reserved unknown-expiry band, and the served
authority carries horizon.expiry_unknown so the console shows a warning rather
than a blank. An unknown expiry is a real state, not a passing one. An authority
beyond the widest 36-month band announces itself once as a planning entry
(severity low, band 37, "on the expiry calendar") and is then quiet until it
crosses into a band; the stamp orders before every real band, so each later
tightening still re-fires. The compression check also looks back: every active
leaf issued under the authority whose expiry equals the authority's (the clamp's
signature) and whose validity is shorter than the reference is counted, the
compression alert says how many already carry the parent's expiry, and the
served authority lists horizon.truncated_leaves with the first fifty
truncated_leaf_ids by validity start, so the shortened certificates can be
walked to rather than inferred.
Scheduled rollover drafts are opt-in: with ca.rollover_draft_band_months set to a
band, an active signer-backed authority entering it gets a rekey_ca ceremony
opened for it on the same sweep — the reviewed plan (its profile and lifetime, a
"Scheduled rollover" reason) at the approval floor, opened by trstctl:ca-calendar
so every custodian may approve it (ca.rollover.drafted). GET /api/v1/ca/authorities and the CA console show the draft with the ceremony's
approvals. A repeated sweep drafts nothing new; a cancelled ceremony drafts again
only at a tighter band; an authority that cannot be re-keyed online (an offline
root needs its offline package) records why instead. Nothing rotates on its own:
custodians approve the ceremony and a person executes the re-key with the drafted
plan, as the re-key and rotation routes always required.
Agent roles: a vantage in the certificate
An agent runs in one of two places, and where it runs decides what it can be asked to do. On a host, it acts on the machine it lives on. In a network segment, it acts for things that cannot run an agent at all — load balancers, appliances, cloud certificate stores — and probes endpoints the way a client would. The second kind holds the credentials that drive those devices, which is exactly why it is a grant and not a startup flag.
An operator picks the roles when they mint the enrollment token. The grant is
recorded with the token, and at redemption the CA stamps it into the issued
certificate as an additional SPIFFE URI SAN
(spiffe://trstctl.example/tenant/<id>/agent/<cn>/role/<role>). It is never read
from the CSR: an agent that could put a role in its own certificate request would
be choosing its own capability, and the whole control is that it cannot. A CSR
that tries gets a certificate carrying only what the operator granted.
At claim time the served channel reads the roles off the certificate the agent
authenticated with — the same certificate the tenant is derived from — and
intersects them with the vantage each job kind needs. trust.distribute acts on
the agent's own machine and is host work. discovery.run, endpoint.verify, and
revocation.probe are observations from a segment vantage and are relay work. A
host agent reaching for relay work is handed nothing and the reach is recorded as
agent.jobs.role_refused, because it is either a misconfiguration or the thing
the gate exists to catch.
Granting the network role additionally requires the agents:relay.grant
permission, separately from agents:write. Enrolling a host agent is routine
fleet work; placing a relay puts appliance credentials on a machine of the
operator's choosing, and those are different decisions.
The agent binary itself is now held to the boundary the roles imply: a CI guard
(docs/agent_binary_import_boundary_test.go) fails the build if trstctl-agent
links the database driver, the store, the event spine, or any served
control-plane package — before the guard existed, two dead store-backed sinks
had already pulled all of those into the binary. A companion guard pins the
connector core (internal/connector/...) to a host-neutral dependency set, which
is what will let the same connector implementations execute on a relay.
An agent enrolled before roles existed carries no role SAN, and is read as
host-only rather than as capability-less. That is what those agents already
were, and reading them any other way would strand a live fleet mid-upgrade. A
renewal carries the roles across unchanged — it cannot gain a capability, and it
cannot silently lose one either — unless an operator recorded a role change. An
operator holding agents:relay.grant records the complete role set
(POST /api/v1/agents/{id}/roles, Change roles on the Agents page); the
agent's heartbeat asks it to renew, and the renewal over the agent channel signs
exactly those roles, so a host agent becomes a relay without re-enrolling. Roles
never come from the CSR: one asking for a role the renewal is not signing is
refused by name. An agent built before this release ignores the renewal request
and takes the roles at its scheduled renewal; the HTTP renewal endpoint carries
roles over and does not apply a grant; and removing a role leaves the old
certificate valid until it expires unless it is revoked.
Per-row locality is now gated at the kind the roles share. connector.deploy
and connector.rollback remain both roles' work at the kind level, and the row
decides: at enqueue, the control plane reads the connector name out of the raw
payload — the one moment it is not yet sealed — consults the shipped vantage
census (nativeConnectorVantage), and stamps required_agent_role onto the
outbox row. The claim SQL filters on that plain column, so a host agent asking
for connector.deploy receives the nginx deploy and never the F5 deploy. A
cloud-store deploy (ACM, Azure Key Vault, GCP Certificate Manager) is
network-relay work since A59, reached over the store's management API from the
segment a relay sits in; where a tenant runs no relay, the control plane's
egress-guarded client carries it. The stamp is durable in the
lifecycle event's side-effect record, so reconcile-replay reproduces it rather
than re-deriving it against a possibly-changed census; rows enqueued before the
census existed carry the empty demand. The dispatcher reclassifies those legacy
rows by connector family before native lookup, so an old nginx row cannot use an
upgrade as a path back into control-plane execution.
The census is per connector KIND, written by hand in the composition root, and deliberately not derived from transport: Envoy is HTTP-driven yet classified host-agent, because its admin surface binds loopback in the deployments we ship for. The agent has constructors for the same 14 host families, including Envoy, and a generated parity test compares the two closed sets. A host stamp is carried through the control-plane outbox claim rather than discarded; the dispatcher defers it before unsealing, and its direct handler repeats the family check before registry lookup. Classification failure is therefore a refusal, never permission to touch the control-plane host.
Per-target vantage overrides are served for Envoy only (A61). An Envoy
target may declare vantage: network in its configuration: its deploys and
previews are then stamped for the network role, need no host assignment, are
refused by the control plane like any host-family row, and are executed by an
enrolled network relay against the remote SDS endpoint with the same connector
the host agent uses. The declaration is validated when the target is saved —
any other family, or any value but network, is refused — and the row's role is
derived from it at enqueue, so neither an agent nor the console can relocate
work by editing anything else. The roles column on the agents read model is
still a projection for the console only: writing network into it grants
nothing, because the certificate still says host and the claim path still
refuses. Host availability also never re-enables the old path: with no eligible
host agent the deploy stays pending for Operations to surface.
AD CS template posture, read from inside the domain
A Windows PKI's real attack surface is not its CA — it is the template list. A template that lets the enrollee supply their own subject, grants enrollment to a broad group, and carries a client-authentication EKU is a domain escalation path that looks, in every console the organization owns, like an ordinary certificate template. Nobody has an inventory of these, because the information lives in the directory rather than anywhere a PKI product looked.
An operator creates an adcs discovery source and schedule with an LDAP/LDAPS
authority, configuration naming context, read-only bind DN, secret://
credential reference, and optional exact relay UUID. The mutation emits the
immutable discovery source/run events and writes one adcs.inventory outbox
intent in the same transaction. Only a network-role relay can claim it; an
exact UUID pins it further. The control-plane worker recognizes the job as
estate-owned and performs no directory I/O. Concurrent schedule leaders use a
database transaction lock and database-clock due check, so one due interval
produces one run rather than two.
adcs.inventory reads pKICertificateTemplate and pKIEnrollmentService
objects under CN=Public Key Services, capturing schema version,
msPKI-Certificate-Name-Flag, the enrollment and private-key flags, EKUs,
which CAs publish each template, and the template DACL. It runs in-domain
because a domain controller's LDAP is not reachable from a hosted control plane
and should not be — an in-domain relay is the only vantage from which this
inventory exists.
The source may also name at most 24 enrollment endpoints, each bound to the
exact LDAP enrollment-service name and one closed kind: web_enrollment,
ndes, or ndes_admin. The relay performs one bounded GET from inside the
domain, requests at most one response byte from cooperative servers, never
follows a redirect, and closes the body without reading or retaining it. Only
the configured URL, closed reachability/authentication state, HTTP status,
authentication scheme names, verified-HTTPS fact, and Extended Protection
state cross the relay boundary. Cookies, redirect targets, authentication
challenge bytes, and response bodies do not. This configured-only behavior
remains the default. With endpoint_probe.enabled: true and 1..32 explicit
endpoint_probe.dns_suffixes, a selected upgraded relay also derives fixed HTTP
and HTTPS /certsrv/ and /certsrv/mscep/mscep.dll candidates from each LDAP
dNSHostName. It never generates mscep_admin, follows redirects, or treats a
response as successful SCEP enrollment. CES metadata is a separate surface.
The adcs.inventory.endpoint_probe capability requires the server allowlist and
the exact relay's fresh enabled census; it also supports nested-group reads when
both options are selected. The planner rejects address literals, URL/port syntax,
wildcards and hosts outside the configured DNS suffixes. Explicit/generated
duplicates are probed once, with both origins retained. Public addresses are
accepted through the shared resolved-address SSRF guard.
Private endpoints require the source's explicit allow_private_endpoint opt-in
and a bounded private_egress_cidrs list limited to RFC1918 or IPv6 ULA
sub-prefixes. The resolved address must fall inside that list; link-local/cloud
metadata, loopback, multicast, unspecified, and CGNAT addresses remain blocked.
The relay's own destination grant is an additional check; LDAP hostname data
never grants egress or disables HTTPS verification.
Each opted-in run represents at most 24 targets with four HTTP workers,
five seconds per attempt and a 90-second overall budget; the legacy job keeps
its 60-second budget. Larger plans carry a source/run-bound digest and next
offset recovered from retained immutable evidence. The next scheduled or manual
run advances that cursor; a failed read retries its page and a changed plan
starts at zero. Every service identifies its place in the latest page, including
invalid/out-of-scope hostnames and services outside that page. Cancellation before
an attempt yields not_checked, with no fabricated status or TLS fact.
Partial or unverified coverage produces ADCS-ENDPOINT-PROBE-INCOMPLETE.
The API, CLI, console and compliance evidence preserve provenance, page coverage
and timestamps. The latest page never implies simultaneous full-estate evidence.
The bounded posture service view flags its 500-row ceiling; retained observations
carry the full directory plan. See the endpoint probe runbook.
Windows IIS authentication/EPA and live NDES enrollment still need a Windows lab;
Samba and private HTTP/TLS stand-ins establish only the implementation boundary.
The combinations are named, not the flags. Nobody spots an escalation path
by scanning four boolean columns across ninety templates, so the analysis reports
consequences: ADCS-ESC1 fires only when supplies-subject, an authenticating
EKU, the absence of manager approval, and an observed broad/low-privilege
enrollment trustee are all present, because removing any one of them changes
the answer. Everyone, Authenticated Users, Builtin Users, Domain Users, and
Domain Computers are recognized by canonical SID; an unknown custom SID is
never guessed broad. Enrollment-agent templates get their own rule
(ADCS-ESC3-AGENT) rather than being folded into the client-auth checks,
because the primitive is different — an agent certificate requests on behalf of
any principal, so one of them is a master key rather than an impersonation of
one account — and so is the remediation: restricting who may enroll is not
enough, the CA must also bound which templates accept agent-signed requests.
The same sweep checks the surfaces outside template LDAP. Plaintext published
CES URIs, reachable plaintext Web Enrollment/NDES, anonymous mscep_admin, and
Windows-authenticated Web Enrollment whose Extended Protection is disabled or
unobserved produce separate findings with the exact endpoint/status/header
facts. msPKI-Enrollment-Servers is treated only as the CES metadata it is; it
is never mislabeled as proof that legacy /certsrv or NDES exists or is absent.
On Windows relays, certutil.exe queries each publishing CA's
CA\\EnrollmentAgentRights: a present policy is enabled, Microsoft's stable
missing-value result is disabled, and access/transport/tool failures are
unobserved. A non-Windows relay reports requires_windows_relay. Missing
evidence therefore produces an explicit visibility finding instead of becoming
either a vulnerable or hardened guess.
Every finding carries the attributes and values it was derived from. A posture finding an operator cannot check against the template's own property page is one they have to take on faith, and the first false positive they cannot check costs the credibility of every true finding after it. A test asserts a deliberately hardened template set produces no findings at all, which is the harder half of getting this right: firing a rule is only half the problem; keeping it quiet is what makes the output worth reading. The SAN variant is reported separately and is the more urgent of the two, since SAN-based mapping is what Windows authentication actually reads. An empty EKU list counts as authenticating, because unrestricted is not harmless. Every finding names the specific change that removes it, and reports whether a CA actually publishes the template — a dangerous template nobody publishes is a latent risk an operator can fix calmly.
Read-only, structurally. The directory interface has exactly one method and
it is Search; a test asserts that. An inventory tool pointed at a domain
controller must be incapable of modifying one, not merely careful. The queries
name the attributes they use rather than requesting a wildcard, are bounded to
2,048 templates and 512 enrollment services, and are scoped to the Public Key
Services container. The template query requests nTSecurityDescriptor with
Microsoft's critical LDAP_SERVER_SD_FLAGS_OID (1.2.840.113556.1.4.801) and
the DACL-only flag; owner, group, and SACL data are not requested. A plain
ldap:// connection is upgraded with StartTLS or the read does not happen: a
template inventory is the map of a domain's escalation paths, and reading it in
the clear publishes that map to anyone on the segment. Anonymous binds are
refused rather than attempted — where they would succeed, the directory is
misconfigured in a way worth reporting rather than quietly relying on. The
redeemed bind credential is attempt-scoped and never enters the intent, event,
report, or posture row.
The Posture console shows it: configured directory readers with
pending/running/succeeded/failed lifecycle, templates worst-first, what each one
permits and the specific fix, whether a CA publishes it, canonical Windows SIDs
granted the enrollment extended right, and which relay observed it when.
It also shows each enrollment service, its published CES URIs, configured live
endpoint probes, CA restriction state/source, exact finding evidence, and the
same observation time/relay. The Discovery form exposes the bounded endpoint
targets; the control plane never probes them.
The empty state distinguishes "no relay has read a directory yet" from "no AD CS
estate", because those are opposite facts an empty table cannot tell apart. The
posture table is only an event projection. The immutable
adcs.template.inventory.observed v2 event is the authority, snapshot/restore
includes both template and enrollment-service projections, and a cold replay
rebuilds the same rows atomically. Historical v1 template-only events remain
replayable but cannot masquerade as complete service evidence. Observation
time and relay identity stay visible so restored evidence cannot masquerade as
a fresh directory read.
Changes between sweeps are reported semantically. A textual diff of two directory dumps is useless — attribute values are bit fields, and "msPKI-Certificate-Name-Flag changed from 0 to 1" tells nobody anything. Each change says what it means and which way it moved: somebody turning on enrollee-supplies-subject, or removing manager approval, or publishing a template that carries findings so a latent risk became an offered one. Only a change for the WORSE emits an alerting event. An operator who has hardened a template does not need waking, and a tool that alerts on improvement teaches people to mute it — after which it will not reach them on the day it matters. Better and neutral changes are still recorded, because an incident timeline needs them.
Those records are now first-class, tenant-scoped adcs_template_drift
discovery findings rather than event-log-only notes. Each one is bound to the
real source, run, relay UUID/name, domain, and observation time, and preserves
the semantic change plus exact normalized before/after facts. Trustee changes
carry canonical SIDs (who gained or lost enrollment access), never raw security
descriptor bytes. GET /api/v1/posture/adcs/drift, the generated SDKs, and the
Posture console expose the same bounded history. A worsening v2 drift event
projects the immutable finding and one notification.drift outbox intent in the
same PostgreSQL transaction, so it appears in the notification inbox after one
sweep; first, unchanged, neutral, and improved sweeps do not page. Event ID and
outbox-key replay guards keep a retried signed receipt from duplicating either.
Pre-v2 drift events lacked run/source authority and therefore replay as audit
history only rather than being attached to a guessed source.
Licensed compliance evidence packs copy the latest internally consistent service observation for each domain in the bounded window plus every authoritative v2 drift record into the signed manifest. Each carries tenant-local event ID, type, sequence, chain digest, source, run, relay, and observation time. The outer API copy is the same semantic object for the console; offline verification still treats the signed manifest as authority. Template/service findings are recomputed from normalized event facts before export, so an agent cannot omit an uncomfortable verdict while keeping the facts that require it.
A first sweep is deliberately not drift. Reporting an entire estate as "added" the first time anyone looks would bury the real change that comes next under ninety notifications. The template is stored exactly as the directory reported it so the next sweep diffs against what was really there; a row written before that column existed is skipped and re-baselines on one quiet sweep, rather than being reconstructed into a template whose flags all read false and reported as becoming dangerous.
Nested enrollment links are served with an explicit opt-in. Set
nested_groups_enabled: true, membership_domain_dn to the domain naming
context, and relay_agent_id to the exact upgraded network relay. Enable
adcs.inventory.enrollment_groups in agent_channel.claimable_job_kinds and
start that relay with --relay-claim. Preflight and run admission require its
fresh authenticated census to advertise the enabled reader. An older relay
cannot silently ignore the nested settings. Default adcs.inventory retains
direct-trustee collection.
The reader resolves canonical grant/deny SIDs inside that domain and uses the
LDAP in-chain rule 1.2.840.113556.1.4.1941, with a critical single-naming-context
control and no referral following. Known direct member links positively check
the matching rule. Users/computers appear only with a reconstructed path through
observed security groups. Direct and expanded denies win conservatively;
conditional/compound permission ACEs keep the result incomplete. Unreadable,
unresolved, range-limited, unsupported or budget-limited evidence carries no
verified member list. An empty LDAP result alone never proves an empty group.
The console and trstctl-cli posture adcs show completeness, observation time,
domain, path and trustee. A failed later read is a coverage change, never proof
that rights were removed. Signed receipt retry, source-log replay and tenant
isolation preserve these facts. The served member list is checked against its
retained immutable observation; a changed SQL path is refused until replay.
Bounds are 256 membership queries, 4,096 returned nodes, 1,024 descendants per trustee, 16 path nodes, 256 KiB of membership attributes and 384 KiB of serialized membership evidence across templates. The nested collector has 45 seconds inside a 60-second relay attempt. LDAP authentication has a 20-second deadline, and searches have 20-second request timeouts. The TLS-only byte-backed bind wipes the borrowed password immediately after authentication; directory frames are capped at 2 MiB and the connection at 16 MiB before library allocation. Exceeding a bound is an explicit incomplete/failed read, not successful absence.
What is not inferred: a complete effective Windows token. The parser handles self-relative DACLs, ordinary/object allow and deny ACEs, Generic All, the certificate-enrollment extended-right GUID, SID byte order, exact-trustee deny, null DACLs and malformed-value refusal. Link evidence does not evaluate conditional ACEs, primary groups, SID history, cross-forest tokens or full Windows AccessCheck; it is never permission to issue. Focused served tests use real Samba AD 4.15.13 and a user → inner → outer enrollment grant. Samba's lab TLS identity override is explicitly unverified. Live Windows forest and trusted production directory identity qualification remain release-lab work. See the nested enrollment runbook.
External Windows-lab boundary. The repository fixture runs the real LDAP client over a BER wire, accepts exactly the two bounded SearchRequest operations, checks the critical DACL-only control and named attributes, returns a binary self-relative descriptor, and proves the normalized enrollment SID reaches the inventory. The served journey separately proves real PostgreSQL, JetStream, out-of-process signer, mTLS relay claim, single-use credential redemption, signed result, event projection, cold replay, and console/API readback. This development environment does not contain a licensed Windows Server forest with AD DS and AD CS, so forest policy, domain-controller authorization, and Microsoft implementation interoperability still require the release lab. A release must run the same read-only account against that lab and retain the domain-controller audit showing Bind/Search only, certutil read evidence, and IIS probe outcomes. Repository tests use live-shaped HTTP/certutil outcomes and cross-compile the Windows collector; they do not claim a real CA's registry encoding or IIS Extended Protection configuration was observed here.
Segment sweeps run from inside the segment
Network scanning ran from the control plane's worker, which meant it could only ever see what the control plane could route to. For a hosted deployment that is the public internet — so the scan inventoried the estate's least interesting surface and reported it as the estate. The segments that actually hold unmanaged certificates are the ones behind a firewall: a management VLAN, a DMZ, a lab nobody admits owning.
discovery.run is now a relay job. A network-role agent already sits in those
segments, and the scanners needed no changes to allow it — they were already
dependency-light, and the one thing keeping them out of the agent binary was a
store-backed sink whose only caller was a control-plane test. Both modes travel:
TLS sweeps for served certificates, SSH sweeps for host keys.
The source must bind a declared segment; it may also bind one
relay_agent_id. Declare the denominator first with idempotent
POST /api/v1/discovery/segments (name, ranges, and the segment's
staleness_hours SLO). That mutation emits discovery.segment.upserted, so a
cold event replay reconstructs the declaration instead of relying on a seed-only
database row. Queueing then resolves CIDRs into at most 10,000 stable host:port
targets and persists that exact command with required_agent_role=network in the
same event/outbox transaction. The generic control-plane worker recognizes both
new commands and legacy network/SSH rows and defers them before receiver I/O, so
an upgrade cannot make an old pending scan dial from the control plane. A wrong
role or wrong exact agent cannot claim the row.
The reserved-range guard travels with the scanner, not with the control
plane. A relay does not escape it by being somewhere else: loopback,
link-local and multicast targets are refused inside the scanner (a network
source's allow_loopback admits loopback for an authorized local diagnostic;
link-local, which covers cloud metadata, and multicast cannot be enabled), and the sweep
reports how many were blocked rather than presenting a refused range as an
empty segment. Attempted, discovered, failed, rejected and blocked are all
reported, because a sweep that reached nothing and a segment with nothing in it
must not read the same.
Findings come back over the channel the agent opened and the control plane writes them, as with every other agent finding — a relay holds no database. The result is accepted only while that exact lease generation is held and only after its receipt signature, mode, assigned targets, counts, metadata bounds, and optional exact agent selector validate. A projection error leaves the claim retryable; a replay after completion is refused. The run API and Discovery console show the segment, role/agent binding, blocked count, and the relay ID from the verified terminal receipt.
What changed in the vantage table: discovery.run was host work when it
meant "enumerate this machine's filesystem". It is now a segment sweep, which is
a vantage question, so it demands the network role. A host agent's own
filesystem inventory still travels on the inventory path rather than as a
claimed job, so nothing was taken away from it.
Revocation distribution points are monitored and recorded
Every inventoried certificate has carried its CDP and OCSP URLs since discovery
shipped — internal/crypto/certinfo parses them — and until now nothing fetched
one. That gap is quietly serious. A CRL whose nextUpdate has passed does not
announce itself: relying parties either fail closed and break the service, or
soft-fail and stop checking revocation at all. Neither appears on any dashboard
until an incident, while the CA's operator believes revocation works because
publishing succeeded once.
revocation.probe is now an hourly, database-clocked relay job, and an operator
can ask for one now: POST /api/v1/revocation/health/probe (certs:write),
trstctl-cli revocation health probe, or Probe endpoints now under
Certificates → Revocation & CT records who asked and queues the same targets under
the request's own bucket, so an operator who fixed a relay's trust bundle or a
distribution point need not wait for the next hour; requests inside one minute
share one probe. The leader
scheduler derives distinct CRL distribution points and OCSP responders from
inventory, matches every leaf to its real issuer key, and carries the exact
public leaf/issuer certificate context needed to verify the answer. It reads
them from the certificates whose signed bytes trstctl holds: those issued
through trstctl or its external CAs, adopted from a host, or imported with
trstctl-cli certificates ingest, which keeps the certificate it is given (an
import never replaces the public chain of a certificate trstctl issued). A
certificate known only from a network scan carries its metadata, not its bytes,
unless agent_channel.probe_scanned_certificates is on: then a relay's TLS sweep
keeps the leaf and the first issuers served after it (each checked against the
hash the relay reported), and the scheduler also checks that certificate's own
serial at its CRL distribution points and OCSP responders, verified against the
served issuer that signed it; its certificate detail shows those signed answers,
including whether the CRL lists its serial. A leaf no served or known issuer
verifies is not probed, scans by relays older than this setting keep no bytes,
and a certificate known only from a cloud scan is not probed; import it to have
it watched. It queues
at most 32 endpoints per bounded network-role command through the event/outbox
transaction. Concurrent leaders converge on the same command ID; if the
representative leaf changes, its OCSP serial changes the ID and creates new work
instead of colliding with an old command.
The relay performs a real fetch. For CRLs it parses the signed list, verifies the
issuer signature, and records thisUpdate/nextUpdate, latency, and revoked
count. For OCSP it builds a request for the exact leaf, POSTs the RFC 6960 media
type, verifies the signed response, checks the serial and response status, and
applies the same freshness window. Endpoints are deduplicated before probing:
one CA's endpoint is named by every certificate it issued, and walking the raw
list would be monitoring that causes the outage it watches for.
It is a RELAY job deliberately. The distribution points that matter most are
internal — an AD CS CRL on http://pki.corp.internal/certenroll/ is unreachable
from a SaaS control plane by design — so monitoring only what is reachable from
outside would inventory exactly the endpoints least likely to break.
Five outcomes, kept distinct because they need different people. fresh and
expiring differ by a warning window (the last quarter of the answer's own
signed lifetime, at most 24 hours, so a responder that keeps signing
short-lived answers reads fresh), and expiring is the one worth alerting
on: after nextUpdate passes, relying parties are already failing. stale
means that has happened. unreachable is deliberately not the same as stale.
And unparseable catches the case that fools status-code checks: a proxy or
captive portal answering 200 with HTML. An AD CS LDAP CDP
(ldap:///CN=<CA>,...,CN=Configuration,DC=...) is read through the tenant's AD
CS discovery binding whose configuration DN covers it (A107): the batch is
pinned to that source's relay, which redeems the bind credential for one
attempt, connects only to the binding's directory under TLS, reads the entry's
certificateRevocationList, and judges it like an HTTP CRL. When it cannot read
the directory the code says why: ldap_connect_failed, ldap_tls_failed (with
the TLS stack's own reason), ldap_bind_failed (the directory refused the
credential) or ldap_credential_unavailable (no credential was redeemed for the
attempt). A directory certificate with a negative serial number is refused by
the relay's TLS stack; Samba's autogenerated certificate draws one about half the
time, so give such a DC a certificate with a positive serial, as an AD
CS-issued DC certificate has. An LDAP CDP no AD
CS source covers reports unparseable as needing a binding, rather than
unreachable, because sending someone to check a network path that was never
the problem wastes the hour that mattered. Whether
the signature was checked is reported explicitly — "we did not check" and "it
verified" must never read the same. The signed terminal receipt is accepted only
for the live lease and exact command, and its evidence digest must be the
SHA-256 of the findings document it reports. A report that can never validate
(an older relay's report carried no digest, so before B13-7 no real relay's
observation was ever recorded) is refused once: the receipt is kept, the job
fails, and the next hourly sweep asks again, so one bad report cannot hold a
relay's other work behind it. It emits immutable queue and observation
events, projects tenant-scoped endpoint health, and queues one warning/critical
notification per unhealthy endpoint. GET /api/v1/revocation/health and
Certificates → Revocation & CT show the endpoint, issuer/certificate context, relay,
evidence digest, status, latency and freshness window. Before a signed
observation exists, the API says observed:false and the console says unknown;
an empty projection never reads as healthy.
The view gives client-context guidance, not a universal recommendation. CRLs and OCSP are different relying-party mechanisms. A fresh responder proves the relay could validate that answer at that time; it does not prove every client is configured to check it, nor whether a client fails closed or soft-fails when the endpoint disappears. Operators must test those client policies separately.
External lab boundary: the repository acceptance test uses a real embedded
PostgreSQL/JetStream/signer stack, an enrolled network relay, and controlled
cryptographically valid CRL and OCSP responders. It does not claim a run
against a domain-joined Windows AD CS CDP/OCSP estate. That externally served AD
CS lab remains infrastructure evidence to collect when the Windows lab described
under F4 is available. LDAP CDP reading (A107) is exercised against a stand-in
directory in the repository tests and against a lab Samba AD DC on the running
product: through the AD CS binding (StartTLS, simple bind) the relay read the CRL
entry under CN=CDP fresh and signature-verified with its one revocation, and a
CDP naming no entry read ldap_entry_missing. A run against a production AD CS
forest is not claimed.
Just-in-time credential leases: the brain ships references, not secrets
A relay executes against things that cannot run an agent — an F5, a NetScaler — which means it needs the credentials that drive them. Handing a relay standing credentials would put a copy of the estate's admin passwords on a machine in the estate, permanently, whether or not any work was pending. So nothing standing is shipped at all.
A claimed connector job carries a reference-only intent: what to deploy,
where, and the NAMES of the credentials it may redeem. Not the credential, and
not the sealed container holding it — the agent has no key for that seal, so
shipping it would be pointless, and would leave the tenant's credential
ciphertext sitting on a host waiting for a future key compromise. At execution
time the relay calls RedeemJobCredential over the same mTLS channel it claimed
on, and the control plane resolves the references — opening the sealed payload
and reading each secret:// name out of the tenant secret store — into locked
buffers that are wiped as soon as the response is encoded.
Redemption is once per attempt, ever. It is a single statement: an
INSERT ... ON CONFLICT DO NOTHING keyed (tenant, job, attempt), performed only
if the caller currently holds the job's claim lease. A replay inserts nothing. So
does a second agent that stole a lapsed lease, and so does a stale attempt
number. All three get the same coarse PermissionDenied with nothing to
distinguish them; the reason is classified afterward for the audit event only,
so probing the endpoint cannot map the claim table. Material is resolved BEFORE
the gate is taken, so a custody outage refuses the call without burning the
attempt's one redemption — the relay retries rather than failing the job. The
redemption's expiry is bound to the claim lease and never chosen independently:
a credential must not outlive the claim, or a second agent could take the job
while the first still holds live material.
The agent's own words no longer become durable history when it held a
credential. A1 already kept agent free-text out of outbox.last_error behind a
closed set; the event log took it raw, which was safe only while agents held no
secrets. An appliance password is short and word-shaped — no redactor recognizes
hunter2-lab, and no entropy floor fires on it — so any attempt that redeemed
material records a closed-set marker instead of the agent's text. The operator
still gets the failure reason, the redemption's audit reference, and the evidence
digest. The relay keeps its own record on its host: a bounded failure log
(--deploy-failure-log, beside --key as deploy-failures.jsonl by default,
mode 0600, the newest 200 entries) that names the step that failed (intent,
executor, redeem, material, connector, sandbox, predecessor or
verify) and why, with every redeemed value (and every line of a multi-line one),
every PEM block, and the value of an authorization header or credential-shaped
field removed. An operator with access to that host reads it with
trstctl-agent --show-deploy-failures. (Earlier builds discarded the error,
despite this promise, then kept it only in the process log.) An attempt that
redeemed nothing never held a secret to echo, so its detail flows through
redaction as before.
Operations shows credential custody directly: how many redeemed credentials are held by relays right now, how many have ever been handed out, and how long the oldest live one has been held — counts and one age, never a tenant, agent, reference name or value. A live count that does not fall, or an age past the maximum claim lease, is a stuck attempt holding material.
The relay executor ships. internal/agent/relay in the agent binary claims
connector.deploy, redeems the credential for that attempt, builds the same
connector implementation the control plane would have built — same constructors,
same sandbox, same capability grant — drives the appliance over its API from
inside its own segment, wipes, and reports. It is armed by --relay-claim and
only when the agent's certificate actually carries the network role; an agent
without the role says so at startup instead of polling forever and presenting as
a stalled queue.
The order inside an attempt is deliberate. Work this build cannot execute is refused BEFORE redemption, because a credential redeemed for an attempt that was never going to run is material outside the seal for nothing — and it burns the attempt's one redemption, so no other agent can take the work either. Redeemed values are moved straight into locked buffers and the wire copies wiped, so the only surviving copy is the one destroyed on the way out, including on panic. What a connector or an appliance says on failure is never forwarded: the relay reports a closed phrase and keeps the target's words local, because an appliance can and does echo the credential it received back in an error body. A sandbox denial is reported as a failure, not as a deploy with a footnote.
Ten connectors are relay-executable — f5, netscaler, a10, kemp,
cisco, fortigate, paloalto and, since A59, aws-acm, azure-keyvault and
gcp-certificate-manager — and the Agents console shows exactly that set for
network-role agents, derived from the agent package's own census so the console
cannot advertise an executor the binary lacks. Host-role agents see their host job
kinds and host connector families instead. The server uses that same job-kind role
census at claim time. That list describes this server build and certificate
roles. Since A62 each agent also reports its own executor census on its
heartbeat — the job kinds and connector families the enrolled binary ships, its
version, and the enabling flags that process was started without — and the
console shows that reported list, role-filtered the same way, with the server
census beside it (capability_source says which one describes the agent; an
older agent that has never reported one shows the server census alone). The
report authorizes nothing: certificate roles and the server census still gate
every claim, and terminal receipts remain the evidence that work completed.
Host connector execution ships too. The thirteen file/exec connectors — nginx, Apache, Caddy, HAProxy, IIS, Postfix, Traefik, Java keystore, PostgreSQL, MySQL, RabbitMQ, Elasticsearch, Tomcat — now execute on the host agent that serves the machine, not against the control plane's own filesystem. The connector implementations moved unchanged: they were always host-neutral, and what changed is which filesystem they resolve against.
Envoy is the fourteenth host connector. It pushes SDS over HTTP instead of writing files, but the shipped target is a co-resident loopback listener. The agent therefore supplies its own HTTP client and the control plane refuses the row exactly like every file/reload family; transport shape does not make a loopback address reachable from the right machine.
The exec profile moved with them, and had to. An allowlist naming
/usr/sbin/nginx is a statement about a host; leaving it on the control plane
while the exec happened on an agent would mean an operator authorizing a binary
on one machine and a different binary running on another. It is now a file on
the host (--host-exec-profile), read and validated at agent startup so a
mistyped path surfaces when someone is watching rather than an hour later during
a renewal. Without it an agent claims no file/reload deploys at all: there is no
safe default for "which commands may run on this machine", so an absent profile
refuses rather than permits. NewLocalOps re-canonicalizes the roots and
re-Lstats every command on the host that will run them, which is the point —
the check and the execution finally happen on the same machine.
One binary serves both vantages. A relay claims appliance work, a host agent claims file/exec or co-resident Envoy work, an agent granted both roles claims both, and the per-row role demand stamped at enqueue decides which agent may take a given job. The two executor sets are disjoint by test.
Rollback executes now through two custody-safe models (D4/G1). After CSR-first issuance the control plane holds no subject key, so it never rebuilds a predecessor bundle and never sends one back down.
The executable form is a re-BIND. The predecessor is already installed on the appliance; what a deploy changed was which installed object the listener points at, and a rollback points it back. Nothing is uploaded, no key moves, and the operation is possible precisely because the control plane holds nothing.
That required a change to deploys, in addition to a new operation. Every appliance connector installed under a name derived from the target, so each deploy overwrote the object before it — there was never a predecessor to bind back to. Deployments now install under a name carrying the certificate's fingerprint, which makes two deployments two objects (and keeps deploys idempotent for free, since the same certificate computes the same name). The listener-facing object — an F5 Client SSL profile, a NetScaler certkey, an A10 client-SSL template, a Kemp virtual service — keeps its name, so existing bindings are untouched.
Six appliance families ship re-bind: f5, kemp, netscaler, a10 — the ones whose API
addresses an installed object separately from uploading one, which is the
property a re-bind needs — and fortigate and paloalto when the target
names the binding to move: FortiGate bind_to (admin re-points
system.global admin-server-cert, ssl-vpn re-points vpn.ssl.settings
servercert) and PAN-OS ssl_tls_profile with commit_admin (the shared
SSL/TLS service profile is set and a partial commit scoped to that administrator
pushes only its own changes). One cloud store joins them (A58):
gcp-certificate-manager with certificate_map_entry (<map>/<entry>)
creates a fingerprint-named Certificate resource per deploy and points the map
entry at it, so a rollback re-points the entry at the predecessor still in the
project; because that family's deploys run on the control plane's
egress-guarded client, its rollback row is stamped control_plane, no agent
can claim it, and the outbox worker performs the re-bind and records the same
rolled_back receipt an agent would. A fortigate, paloalto or GCP target
without that binding deploys in place under the target name as before and its
rollback is refused at request time as not configured, never reported as done.
A cisco target re-binds when it names its ISE node and roles, and one without them
is refused as not configured rather than offered a rollback that would return
success having changed nothing; aws-acm (one ARN,
re-imported in place, no previous version) and azure-keyvault (versions kept,
newest always served by name, no call to make an older one current) are refused
with those reasons on the route and in rollback_readiness. An operator's
rollback also pauses automatic renewal of that identity on that target until
an operator resumes it (POST /api/v1/identities/{id}/renewal/resume,
trstctl identities renewal-resume): the identity's renewal_paused list and
the target's renewal_paused_identities show the hold, the lifecycle plan
reports it as paused_by_operator_rollback, expiry alerts keep firing, the
scheduler and posture remediation never redeploy there on their own, the
identity's other targets keep renewing, and the resume is an audited event
(identity.renewal_resumed) carrying the operator who made it. An explicit
operator-driven renewal transition is not held by the pause. Plugin-backed
connectors are refused because the plugin ABI exports deploy only. A rollback whose predecessor object is no longer on the appliance
fails, with a distinct error, because reporting success there tells an
operator that a bad certificate stopped serving traffic when it did not.
All 14 host connectors use a different model. After a successful host deploy,
the exact enrolled host agent stores active plus one predecessor bundle in a
machine-local AES-GCM ledger (--host-rollback-dir, default beside the agent
key). The key file and state files are 0600; plaintext certificate/key bytes
exist only in locked memory while recording or restoring. A rollback is pinned
to that exact agent, reopens the ledger after restart, restores the predecessor,
runs the same connector reload, and repeats local listener verification. A
second predecessor is never retained, so this is a rollback buffer rather than
a key archive. Losing that directory means the host predecessor is gone and the
job is refused; the control plane cannot reconstruct it.
Two things to know before relying on it. First, an install that predates
fingerprint naming holds its last certificate under the old target-derived
object name. An appliance rollback tries the fingerprint-named predecessor
first and, when none exists, re-binds to that legacy object (f5, kemp,
netscaler, a10); the receipt then reads rolled_back_to_legacy_object and says
the object's contents were not re-read, so verify the listener independently.
A host agent adopts the certificate and key it finds installed before its first
deploy to a target as its retained predecessor, reports the adopted leaf's
fingerprint inside its signed receipt together with the public leaf, and the
control plane records that leaf as an observed certificate and names it as the
target's predecessor, so the first trstctl deploy over a pre-trstctl install is
rollable through the served route; an agent that did not retain the pair still
refuses the restore. The control plane also names as predecessor the latest
delivery to a target by any host agent, so a re-enrolled agent can restore what
the host served. A target with no earlier trstctl delivery and no reported
adoption becomes rollable after its second deployment. The single-target read
(GET /api/v1/connectors/targets/{id}, the Connectors console's destination
detail) now says so up front: rollback_readiness applies the same refusals
the rollback route applies — no executable route for the family, a fortigate or
paloalto target without its binding, no bound identity, no predecessor
certificate, a revoked predecessor, a predecessor never delivered to this
target through trstctl under its fingerprint-carrying name, or a host target
without a second successful agent deploy — and names what would make the target
rollable. A refused rollback still reports that the predecessor object is not
installed, which remains the truthful answer at execution time.
Second, objects now accumulate. Each deployment leaves its predecessor on the appliance rather than overwriting it, which is the entire point, and nothing prunes them automatically: trstctl removes an object from a customer's appliance only on an explicit takedown request (f5, kemp, netscaler and a10), never as housekeeping. On a target renewed every 90 days that is a handful of objects a year; on a short-lived-certificate target it is not, and operators running those should expect to prune. Automatic pruning is deliberately not served: deleting a crypto object that something else might be bound to is a worse failure than leaving one behind.
Deploy and rollback now use the same connector.bind:target:<id> effect lane.
The agent claim query admits one unexpired holder per lane and returns at most
one row from that lane in a batch, so two agents cannot race a deploy and restore
against the same listener. Unrelated targets remain parallel.
Automatic rollback is served when a target explicitly sets
auto_rollback_on_verify_failure; manual console rollback queues the same
command. Unsupported connector families and targets with no predecessor are
refused at request time with no memo-only success receipt. A plugin-backed
connector executes on a network relay once one of the tenant's relays has
reported, in its signed census, that it loaded that plugin (A63): the deploy and
preview rows are stamped for the network role from that census at enqueue, the
control plane leaves them for the relay while it is enrolled, and falls back to
its own plugin surface only when no relay carries the plugin any more. Plugins
have no rollback. connector.right_size executes on a network relay when its
connectors.right_size binding sets executor: network (A64): the relay redeems
the binding's token for one attempt, runs the same mutate-and-readback
conversation as the control-plane worker, and its signed report becomes the
operation's terminal receipt; the server verifies the report's shape and
digest, not the endpoint, so the readback evidence is the relay's. Envoy TLS
posture changes run on one pinned agent: a co-resident Envoy's on its assigned
host agent, and a remote Envoy that declares vantage: network on the network
relay its required_agent_id names (A104), which keeps the predecessor in its
own encrypted store so the rollback returns to it. No job kind is claimable by default:
agent_channel.claimable_job_kinds must name each kind before any of this
moves, which is deliberate rather than unfinished. connector.rollback is named
separately from connector.deploy on purpose — an operator should be able to
enable undoing a deployment without enabling deploying, and during an incident
that is the order they will want.
A relay runs its own egress policy, not the control plane's. The control
plane validates a target's endpoint at admission and drives it through an
SSRF-blocking, egress-guarded transport. A relay cannot run that guard: it exists
to reach devices on private, non-routable addresses inside its own segment, which
is precisely what the guard refuses. Instead every connection a relay opens to
run a job refuses link-local addresses, cloud instance metadata, unspecified and
multicast addresses, and --relay-egress-allow refuses everything it does not
list (see Configuration). What that does not cover:
- Without an allowlist, the rest of the network is reachable. The always-on refusal covers only the addresses no job needs. An operator granting the network role without an allowlist grants every other address the relay's own network can reach.
- Addresses, not names. An allowlist names addresses and ranges; a host name cannot be listed, because a connection sees only the address it resolved to.
- Two probes are checked before they run, not at the dial. The PQC posture
and dry-run listener probes may fall back to the system
openssl, which dials outside the agent, so their destination is resolved and checked first; a name that resolves elsewhere between that check and the connection is not caught for those probes. - Plugins keep their own grants. A plugin's dials follow its grant and the reserved-address refusal plugins always had, not the relay allowlist.
- The Agents page does not show it yet. The policy in force is printed at the agent's startup; the agent census does not report it.
The agent job ledger: served executors and signed receipts
Work that touches your estate has to execute inside your estate. The control plane has no route into a host and never gets one, so the agent comes and takes the work over the connection it opened — the same mTLS channel it heartbeats on, the same certificate-derived tenant, no inbound port anywhere.
Agentless SSH trust collection uses ssh.trust.collect through the selected
network relay. It calls only a fixed root-owned read-only helper using pinned
host identity and a vaulted PKCS#8 reference. Every target has an explicit outcome;
missing permissions, failed authentication, denied egress and oversized readback
cannot become an observed posture. The host-agent collector stays the default.
Approved additive SSH trust uses ssh.trust.apply in its own one-worker loop,
with local --ssh-trust-fleet --ssh-trust-confirm plus the staged CA and local
command policy. It is not claimed by the shared relay pool. The exact skipped
roster, approval and signed readback/health/rollback facts are served under
/api/v1/ssh/trust-fleet/rollouts; queued jobs remain pending.
The ledger is the outbox, unchanged. An entry is still committed in the same
transaction as the state change that caused it, still carries an idempotency key,
still at-least-once. What changed is the consumer: ClaimJobs and
ReportJobResult on the agent channel let an enrolled agent lease work, extend
while it is still going, and report executed or failed.
A claim is a lease, not an assignment. An agent that is killed, partitioned,
or stops calling home leaves work behind; because the claim expires rather
than sticking, that work returns to the queue without anyone noticing the machine
is gone. Claims use SKIP LOCKED, so a fleet polling in lockstep fans out across
the queue instead of serializing on its head. Extend, complete and release all
require the caller to hold the lease, so a stalled agent whose lease lapsed cannot
report on work another agent has since done. GET /api/v1/operations/jobs, trstctl-cli operations jobs and the
Operations console show per-kind waiting and held counts plus the oldest wait —
counts only, never a tenant identifier, payload or credential. The same posture
read publishes Prometheus series (queue depth and oldest wait per kind, live
credential redemptions and their oldest age, claims and refusals as counters),
so the metrics and the console cannot disagree about what the fabric is doing.
Three alert rules ship in deploy/observability/alerts.yml: a stalled queue
alerts on the oldest WAIT rather than depth, because depth alone cannot
distinguish a busy fabric from a stopped one; a credential held past the maximum
claim lease is critical, because that is live material on a machine whose claim
should already have lapsed; and refused redemptions alert at all, because each
one is an agent asking for material it did not hold a claim for. trstctl doctor
gains FABRIC-1, which sweeps agent-claimable work nobody has taken and names
the ROLE the waiting work demands — usually the answer, since work demanding a
role no enrolled agent holds waits forever and looks exactly like a busy queue.
It sits beside DUR-2 rather than inside it because a control plane that is not
delivering and a fleet that is not claiming need different runbooks.
Reports are signed, and the signature is what an auditor gets. Every terminal report — executed or failed — carries a detached signature the agent made with the same key behind its channel certificate, over a canonical statement naming the tenant, the agent, the job, the claim attempt, the outcome, the evidence digest, a digest of the report's text and when it was signed. The tenant and the agent name in that statement come from the certificate the caller authenticated with, never from a request field, which is why a forged receipt and a cross-tenant receipt are the same refusal: both were signed over different bytes than the ones the server rebuilds.
This is worth being precise about, because mTLS already authenticates the connection. What the signature adds is evidence at rest. Without it, "agent-7 executed this deploy" is a sentence the control plane wrote about itself, and anyone who can write to the event store can write that sentence. With it, the record is one the control plane could not have produced. The statement and the signature are stored on the event and in a receipt ledger, so the check can be repeated later by someone who does not trust that it happened the first time.
The production event log checks an explicit, closed payload schema for agent claims, refusals, signing, credential redemption, workload issuance and terminal reports. Unexpected fields or undeclared schema versions are refused. Privacy erasure cannot rewrite a signed terminal report: it refuses that rewrite rather than changing the bytes covered by the signature.
Agent audit publication runs after the operation's state change, with the
event's identity, time and actor fixed before the first attempt. If the event
bus does not acknowledge the append, the server queues that same event, under
the same identity, on the control-plane-only agent.audit.retry outbox
destination and logs agent job audit event queued for retry with the tenant,
event type and event ID, never the payload. The outbox dispatcher re-appends it
with the outbox's usual backoff and attempt cap, and because the stream
deduplicates message IDs within its 24-hour window, a publish whose
acknowledgement was lost is not recorded twice. An append the log refuses for a
deterministic reason (a payload the privacy schema does not declare, a missing
tenant) is not retried: the server logs agent job audit event could not be stored with the reason. When either line appears, compare the job and retained
receipt with the event history; a successful agent response alone does not
establish that its audit event was retained, and a retry row that exhausts its
attempts is visible as a failed outbox delivery on Operations.
Anything that does not verify is refused fail-closed, with an
agent.job.receipt.rejected audit event and a counter on Operations: unsigned,
signed by a key that is not the connection's certificate, altered after signing,
or signed outside a ten-minute window against the server's clock. That last one
is what stops a captured receipt being replayed later — a signature does not
expire on its own. Lease extensions are deliberately NOT signed: an extend
claims nothing about the world, and signing every keepalive would put the
agent's key on the heartbeat path for no evidentiary gain.
Two honest limits. The signature attests what the agent SAID, not what the
appliance did — an agent that is lying, or that is wrong about its own outcome,
produces a perfectly valid receipt for a false statement; making the claim
checkable against the endpoint itself is the verification work (WS-D), not this.
For a deploy with a verify address the transcript behind evidence_digest now
travels in the report: the control plane checks it against the digest, against
its own record of the target and the issued certificate, and keeps it as the
verification receipt. A deploy without a verify address has no transcript, and
the receipt binds only the agent's signed statement.
What is served, and what is not. connector.deploy has both the network-relay
executor from A3 and the host executor for the 14 host-vantage families; a role-
eligible agent claims it, redeems its credential for one attempt, and performs the
effect from the required machine or segment. Connector test, rollback, endpoint
renewal, exact-fingerprint host containment and verification, discovery,
revocation probing, AD CS inventory, trust distribution, CMDB/MDM/ticket
observation, and self-upgrade also have the executors
named in their own sections. The claimable set remains empty by default and
agent_channel.claimable_job_kinds is the only way to fill it. That is deliberate
rather than unfinished: handing out work nothing can perform fills a queue while the
control plane's own worker stops doing it. The kinds the allowlist recognizes
— connector.deploy, connector.test, connector.rollback, endpoint.renew,
endpoint.contain, endpoint.recovery_check (the read-only check a replacement reads after a
containment, enabled with endpoint.contain), endpoint.purge, connector.takedown, endpoint.verify, discovery.run, ssh.trust.collect (explicit read-only agentless SSH sources), revocation.probe, adcs.inventory, adcs.inventory.enrollment_groups (opt-in bounded nested enrollment links), adcs.inventory.endpoint_probe (opt-in scoped LDAP hostname probes with retained pages),
trust.distribute, pqc.posture, pqc.posture.rollback, connector.right_size, cmdb.sync,
mdm.sync, ticket.sync, agent.upgrade, and agent.self_check (the
post-upgrade functional check every polling agent answers, A71) —
all have shipped executors. Enabling a kind exposes that already-built executor; it
does not turn a placeholder into work.
Anything outside that allowlist is dropped even if an operator names it in
configuration, so ca.issue and notification.expiry cannot be moved onto a host:
those are the control plane's own effects and CA-adjacent work does not belong in
the estate.
Agent-signed result receipts are served and verified against the same certificate
that authenticated the mTLS channel. Aggregate waiting and claimed counts are served
on Operations, together with verified/refused receipt totals. Each agent's live
claims are attributed on the Agents page and in GET /api/v1/agents (live_claims:
job id, kind, attempt and lease expiry, read from the ledger's unexpired leases, never
a payload). agent_channel.max_live_claims_per_agent caps how many leases one agent
may hold at once; at the cap the channel withholds new claims until a lease completes,
fails or lapses, and the Agents page says so (claim_quota, claim_quota_reached).
Unset is the shipped default and means the shared agent bulkhead is the only cap: the
quota is an operator choice, not something the server infers from fleet size.
Served status vocabulary: what each status claims
A status string is a claim you act on, so each one has to mean exactly what the code did and nothing more. Three statuses on the deployment surface used to read stronger than the work behind them. They are corrected below, and the correction is enforced rather than remembered.
The enforcement has two closed layers. internal/servedstatus records, for the
action-bearing deployment and fleet surfaces, whether an attempt contacted the
target, changed it, independently re-read it, or computed a verdict from
evidence. The API builds those receipts from registry constants and CI rejects a
spelling stronger than its flags or a bare string that bypasses the registry.
Separately, docs/status_vocabulary_test.go derives an exhaustive census of
every status, outcome, verdict, health_gate, and canary_state field in
the generated OpenAPI contract. Each field must bind a closed evidence class to
an exact, AST-resolved production Go symbol; a planted future DTO proves the
census fails closed. The same gate declaration-checks the assembled negative
proofs and console renderers for fleet re-issuance, full-set restore drills, and
MDM lifecycle traces. Shared console labels cannot add a green status until its
evidence predicate is reviewed.
| Status | Surface | What it means | What it does not mean |
|---|---|---|---|
queued |
connector delivery | Intent committed to the outbox in the same transaction as the state change. | That any connector has run. |
delivered |
connector delivery | A connector reached the target and applied the credential. | That the endpoint is serving it. Live verification is a separate state, served since D2 — see GET /api/v1/endpoints/verifications and the endpoint verification vocabulary below. A delivery receipt says what this control plane did; only a handshake says what the listener answers with. |
failed |
connector delivery | The attempt ran and did not succeed. | — |
config_validated |
connector delivery | POST /api/v1/connectors/targets/{id}/test resolved target metadata, schema, and credential references locally, because an agent-owned target has no eligible connector.test path enabled. Cloud-store targets do not use this fallback. |
That the target was contacted, reachable, or willing to accept the credential. Nothing was changed. |
verified |
connector delivery | A connector applied the credential AND a TLS handshake against the endpoint afterward observed it serving that exact identity. The only delivery state that says the certificate was live at that check rather than merely sent. | That it is still live now. A delivery receipt is historical. Compare the endpoint verification row's last_checked_at and probe again before relying on a prior pass. |
verify_failed |
connector delivery | A connector applied the credential and a handshake found the endpoint serving something else. | That the delivery failed. It succeeded; the endpoint did not take it. This is a renewal that did not land, and it is what triggers rollback where a target has opted in. |
verified |
endpoint verification | A TLS handshake observed the listener serving the expected identity at last_checked_at. The record carries which comparisons ran — fingerprint always, name set and chain when an expectation supplied them. The summary percentage counts last recorded checks, not current listener state. |
That it still serves the identity now, or that every vantage agrees. A local row means the serving host's own agent confirmed it then; only a relay row means a client across the segment could get it then. Re-probe after a restart or other out-of-band change. |
diverged |
endpoint verification | A handshake succeeded and the listener is not serving what was deployed. The mismatch class says which way: fingerprint, sans, chain, expired, not_yet_valid. |
That the deploy failed. It usually succeeded — this is a renewal that did not land, which is exactly the failure inventory-based expiry alerting cannot see. |
unreachable |
endpoint verification | The handshake did not complete, so nothing was observed. | A divergence, and emphatically not a pass. An endpoint nobody could connect to is not verified. |
not_checked |
endpoint verification | No check has established the current expected identity from this vantage. A prior relay check may exist, but if a newer local deployment changed the expected fingerprint or address, that predecessor check cannot qualify the new generation. | That the endpoint is fine. An endpoint with no configured listener address stays here permanently — absence of a current check is not absence of a problem. |
dry_run_queued |
connector delivery | An effect-free test was queued for the eligible host/network agent bound to this target or for the bounded control-plane outbox worker that owns its cloud connector. | That anything is yet known about the target. No executor has reported. |
dry_run_planned |
connector delivery | The correct execution vantage validated every prerequisite a real deploy needs and returned the mutation plan. A network relay proves its appliance credential with a read-only request. A host agent validates its operator-owned roots and logical commands and performs a read-only TLS handshake when verify_address is configured; without one, the plan explicitly says live verification is not configured. |
That anything was deployed or, when no verify_address exists, that a certificate is serving. The test path never invokes a connector's Deploy, writes a file, or starts a reload command, so zero writes is structural rather than promised. |
dry_run_blocked |
connector delivery | The correct agent vantage ran the effect-free test and a real deploy would not proceed. The reason names the step that stopped it. Deterministic blocks are terminal test answers rather than endlessly retried jobs. | That the target is broken in every respect — one step failed, and the plan says which. |
rollback_recorded |
connector delivery | Legacy evidence from releases that recorded a memo instead of executing. New unsupported/no-predecessor requests are refused and do not create this status. | That a rollback executed. Nothing was restored. |
rollback_queued |
connector delivery | An executable rollback was queued for the required enrolled agent. Host rows also pin the exact agent that retained the predecessor. | That anything happened yet. No agent has reported. |
rolled_back |
connector delivery | An agent restored the predecessor and returned a signed receipt. reason=rolled_back_and_reverified additionally proves the same agent handshook the listener after reload; otherwise the detail explicitly says verification was not configured. |
That an unverified restore is live when the reason does not say reverified. |
rollback_refused |
connector delivery | An agent declined the rollback before contacting the target — it cannot execute that connector, no supported inverse exists, the named predecessor is absent, the exact host ledger/profile is unavailable, or the sandbox blocked the operation. | That the target rejected anything. It was never reached and is unchanged. |
rollback_failed |
connector delivery | An agent reached the target and the family-specific re-bind or local restore/reload/reverification did not succeed. The reason names the failed stage. | That the target is broken in every respect, or that the predecessor is absent unless the reason says so. |
not_evaluated |
fleet re-issuance health gate | No evidence exists from which a verdict could be computed, so trstctl asserts none. | It is not a pass. |
passed / failed |
fleet re-issuance health gate | Either an operator attested the verdict, or the replacement-deployment gate was computed from signature-verified agent receipts. | That a plain delivery row or an unsigned report proved endpoint health. |
planned |
fleet re-issuance batch | A partition of the affected identity set that has not been published. | That anything in the batch was attempted. |
queued |
fleet re-issuance batch | Exactly one durable outbox command exists for the cursor batch. | That the worker or a target has run. |
waiting_verification |
fleet re-issuance batch | Replacement work was published and the cursor is waiting for accepted signed endpoint receipts. | That silence is success or that a later batch was published. |
executed / failed |
fleet re-issuance batch | The batch ran as a mutation unit and its signed verification gate passed or failed. | That a later batch necessarily ran. |
halted |
fleet re-issuance batch | The batch was never published because an earlier signed verification failed. | That this batch or its targets failed. |
Two spellings are retired and are no longer written: test_succeeded (now
config_validated) claimed a successful test on a route that opens no
connection, and a batch completed was stamped at planning time before anything
ran per batch. Both remain in the served OpenAPI enum and render in the console,
because receipts written before the correction still carry them and removing an
enum member would put stored rows outside the contract that describes them.
Making these statuses stronger is real work, not relabeling. The dry-run half
is now served from every execution vantage. With connector.test enabled,
POST /api/v1/connectors/targets/{id}/test queues a job for the bound host agent
or network relay. A network relay redeems the appliance-management credential
for one attempt and probes the endpoint with a read-only GET. A host agent
validates the target paths and logical commands against its local
--host-exec-profile, then performs a read-only TLS handshake against the exact
verify_address and optional verify_server_name when configured. If the
address is absent, preview explicitly marks listener verification skipped while
still planning the deploy; the later delivery can say delivered but cannot
say verified. It does not redeem a
certificate or private key: those do not exist in a target-only test and are
supplied only after a deploy is authorized. It redeems only a host-target secret
the preflight actually needs, such as a Java keystore password reference.
The three cloud-store connectors queue the same event-sourced job on the bounded
control-plane outbox worker. That worker resolves the exact saved target revision,
leases its provider credential for one attempt, and performs an authenticated
read-only list operation before returning the later import plan.
Both vantages return what a real deploy would change as dry_run_planned, or
name the first failed step as dry_run_blocked. Zero writes is structural rather
than promised: this path never calls a connector's Deploy, never writes a file,
and never executes a reload. A deterministic block is reported as a completed
test answer, so an unavailable listener or insufficient host profile does not
become a rapid retry and credential-redemption storm. A signed connector plugin
answers the test through its test entry point under a read-only grant: no file
writes, and HTTP requests limited to GET and HEAD, a write being refused
before it is sent. A plugin without a test entry point is blocked before it is
invoked or any credential is redeemed. Without an eligible agent path enabled,
the route keeps the honest local answer, config_validated, rather than queueing
work nothing will claim.
Restoring a predecessor is served as a relay-executed re-bind, not a
re-upload. After CSR-first issuance (B1) the control plane never holds the
subject key, so the rollback command names the predecessor already installed on
the target by serial and fingerprint. A connector that supports re-binding can
switch to that object and report rolled_back; missing capability, credential,
predecessor, or sandbox permission reports rollback_refused before contact,
while a reached target that cannot re-bind reports rollback_failed. Where no
predecessor exists — a first deployment — the durable evidence says so rather
than sending an operator looking for a credential that was never there.
Remaining private CA hierarchy operator flows beyond root/intermediate/leaf issuance. Root/intermediate CA creation, existing signer-backed CA chain import, offline-root import, offline-intermediate CSR generation/import, m-of-n approvals, signer-backed leaf issuance, and configured upstream CA issuance are served at
/api/v1/ca/ceremonies,/api/v1/ca/authorities,/api/v1/ca/authorities/offline-roots,/api/v1/ca/authorities/imported,/api/v1/ca/authorities/imported/csrand its ceremony-ID readback,/api/v1/ca/authorities/{id}/offline-intermediates/csr,/api/v1/ca/authorities/{id}/offline-intermediates, and/api/v1/external-cas. Public/private direct-CA discovery is served at/api/v1/ca/discovery(returns configured public/private upstream CAs and imported hierarchy authorities, without PEM or key material). Zero-downtime CA rotation activation is served at/api/v1/ca/authorities/{id}/rotate(marks the predecessor superseded, records the successor'sreplaces_id, keeps the predecessor issue URL live while new certificates route to the successor). Signer-backed renewal/re-key is served at/api/v1/ca/authorities/{id}/rekey(consumes aca-rekey-v2ceremony bound to the authority profile, lifetime, and reason; mints fresh CA key/certificate material, recordsca.authority.rekeyed, keeps the stable issue URL live), and cross-signing at/api/v1/ca/authorities/{id}/cross-sign, with durable public certificate readback at/api/v1/ca/cross-signs/{ceremony-id}. Offline-root re-key (/api/v1/ca/authorities/{id}/offline-rekey) and cross-certificate verify/import (/api/v1/ca/authorities/{id}/offline-cross-signs) never accept a private key: operators produce the successor and both cross-certificates on the disconnected root system, then submit only certificates under an exact ceremony purpose (see the key-ceremony runbook). The console's Imports workspace now previews and executes offline-root succession under that purpose. Its event-projected public receipt is read by ceremony ID at/api/v1/ca/offline-rekeys/{ceremony-id}after a restart or a lost response; no private key enters the browser or control plane. The offline-root import profile accepts any key algorithm the platform can verify and label — ECDSA P-256/P-384/P-521, Ed25519, RSA-2048/3072/4096 — and the reviewedsignature_algorithmmust name the key the certificate actually carries (a root reviewed as one algorithm and carrying another is refused before the ceremony starts); the console's offline import and offline re-key forms select it. CAs the served signer mints stay ECDSA P-256. For offline-root import,ttl_secondscaps remaining certificate validity and is checked before the ceremony starts and again at import.All 14 external CA integrations are served when configured.
buildRunDepsconstructs tenant-bound AD CS, AWS PCA, Azure Key Vault, DigiCert, EJBCA, Entrust, GlobalSign, Google CAS, Let's Encrypt/ACME, Sectigo, shell CA, Smallstep, Vault PKI, and Venafi TPP/TLS Protect clients. The authenticated served issue route journals the request before the upstream call, and the acceptance proof independently validates the returned chain for every provider. The Kubernetes CSR and TrustBundle posture routes beside them serve the authenticated reports the cluster agents' controllers send, and the same census proves both against a real kind cluster, so CA-agnostic issuance (F4) is served end to end.Discovery collectors with residual connector-owned execution: SSH host-key scanning is served through the discovery outbox worker, and on-host SSH/private-key inventory is served through the agent mTLS inventory report path. Connector-specific external secret-store/API-key scanners remain source-plugin or provider-owned unless a native served source kind supplies findings. The network, ssh, cloud_certificate, cloud_secret, ct_log, drift, k8s_ingress_gateway, and manual source kinds are wired through the served discovery worker (see "Discovery control plane" above), alongside the NHI kinds (nhi_cross_surface, oauth_grant, service_account, nhi_behavior, credential_compromise) and observation-shaped api_key sources; secret-repo and third-party artifact scans dispatch through the same worker from their
/api/v1/secrets/scans/*routes. Asecret_storesource runs through the same served secret-manager connectors ascloud_secret(aws-secrets-manager, gcp-secret-manager, azure-key-vault, hashicorp-vault; the kinds share the providers config shape), and a provider outside that set fails with the connector's specific "unsupported provider" error. One accepted kind carries no worker executor:agentsources report through the agent mTLS channel instead of the worker. On the agent side, the shipped binary collects filesystem certificates, OS/Java/NSS/browser trust stores, and private-key material (each behind its own default-off--inventory-*flag), Windows certificate/trust stores (--inventory-windows-stores) and in-cluster Kubernetes TLS Secrets (--inventory-k8s-secrets, widened across namespaces with--inventory-k8s-namespaces). PKCS#11 token collection needs the cgo agent image: the default statically linked agent cannotdlopena module and fails--inventory-pkcs11-modulewith "built without cgo" rather than reporting an empty token, while the releasedtrstctl-agent-pkcs11image (deploy/docker/Dockerfile.agent-pkcs11, glibc plus SoftHSM and OpenSC modules, vendor modules mounted in) readsCKO_CERTIFICATEobjects over a read-only session and is exercised against a real SoftHSM token in the repository's container test. Its live proof is a lab SoftHSM token, not a vendor HSM. Discovery schedules tick server-side: a leader-only scheduler sweeps every minute and queues a run for each enabled schedule whose source has no in-flight run and no run newer than the schedule'sinterval_seconds, through the same event + outbox path an operator-initiated run takes (runs it queues carryrequested_by: discovery-scheduler; a failed run counts as an attempt, so a broken source retries next interval instead of hot-looping; one sweep queues at most 100 runs per tenant). The CBOM scanner is also served, through its own/api/v1/cbom/*API rather than the discovery-run worker.Broker-issued agent credentials can be task-scoped:
POST /api/v1/broker/agent-identitiesaccepts an optionaltask_envelope_base64(the AGID-05 task envelope) and returns thetask_envelope_digestthe credential binds, so an AI/MCP agent badge can be scoped to one authorized task instead of standing scope alone. Verification is the core AGID gate's: the requester signature is checked over the envelope's canonical bytes against an operator-provisioned requester key (the caller cannot supply its own), plus the expiry window, and the bound digest is the verified envelope's own — a substituted envelope cannot be bound in place of the signed one. Every build attaches this gate; an unverifiable envelope is refused, not ignored: silently returning an unscoped credential in place of the scoped one the caller asked for would be the dangerous outcome. Requests carrying no envelope are the ordinary single-hop badge, unchanged.A migration can be reviewed before it runs:
POST /api/v1/pqc/migrations/plan(andtrstctl-cli migration plan, available in every build) previews the plan — which assets would be re-issued and to what, which TLS findings would be rolled out, and the residuals it will not touch — without queueing a run, minting a run id, or writing an outbox row. It calls the same plan builder the start path calls over the same CBOM assets, so the preview cannot describe a different migration from the one that would execute. Both paths also preflight each exact TLS deployment target; an Envoy host-agent target requires an enrolled, assigned host, co-resident management endpoint, and loopback served-listener binding. A TLS finding without an explicit operator target binding returns HTTP 400. An ML-DSA host-csr re-issue is refused by the host agent's name when that agent's executor census reports notls_probe_opensslin its host execution profile (the native probe the agent needs to verify a served ML-DSA leaf;reported_host_tls_probeon the agent), so the run is not queued only to fail fifteen attempts; an agent that reports no census is admitted, and its own refusal phrase then names the cause in the finding. The same census carries the binary's own reason when its host execution profile could not be loaded (reported_host_profile_erroron the agent; for example atls_probe_opensslthat is a symbolic link, which the agent refuses while naming the resolved path to pin): that relay claims no host work, so a connector target deploy or an ML-DSA host migration bound to it is refused with HTTP 409 naming the agent and the reason, instead of waiting on a claim that never comes. The same holds while an agent has paused job claims over a pending job report it cannot open under its current registration (reported_pending_report_error: after re-enrollment, a changed CA bundle or server name, a missing sealing key, or damage); it keeps heartbeating and renewing its own certificate, and claims resume after the operator restores the previous settings or runs the agent once with--set-aside-pending-report. A host renewal whose report was set aside keeps its retained key in the host rollback state: a retry of that job resumes with it, but if the control plane fails the job instead, a later renewal of that target is refused as "the encrypted pending host key belongs to another job" until that renewal is rolled back on the target, which clears the slot. When the certificate was issued but the host could not install it, the agent's receipt names the step that failed (the host sandbox refused the write, the connector's configuration test refused the new pair, or the reload action failed) and the finding carries that phrase; the underlying error stays in the agent's own log on that host and is never served.Signing history is verifiable after the fact:
GET /api/v1/code-signing/identities(andtrstctl-cli code-signing identities) lists recent signing operations with their identity kind (managedfor a signer-held key,keylessfor an ephemeral Sigstore/Fulcio identity) and the transparency-log state of each —verified,pending,failedwith its reason, ornot-published. Verification is not a separate flag that could drift: Rekor publication rides the outbox and the handler refuses to acknowledge an entry whose signed receipt does not verify, so a delivered row is a verified entry. The view reads no sealed command bytes, so the plaintext identity assertion and the artifact digest never leave the signer boundary through it.The SSH key estate is readable:
GET /api/v1/ssh/fleet(andtrstctl-cli ssh fleet) rolls the tenant's discovered SSH key locations up with standing-access and orphaned counts, key types, and the observation window, worst location first. A raw public key can also be the key inside a CA-signed host certificate. The same view now answers whether a host is under the SSH CA (A115): each host agent run with--inventory-ssh-sshd-configsreports its sshd configuration's trust posture, andhost_trustjudges it against this tenant's SSH CA and served KRL: which CAs sshd trusts for user certificates (this CA only, this CA and others, another CA, none), the host certificate it presents (signed by this CA and valid, renew soon, expired, another CA, a signature that does not verify, none), whether itsRevokedKeysfile is the current list (by digest), another list, a plain key list, missing (sshd then refuses every key) or absent, and whetherauthorized_keysstill logs in. The control plane names each host from the agent's certificate. Limits: the agent reads the configuration file's global section the way sshd resolves it (first value wins,HostCertificateaccumulates,Includein place) rather than runningsshd -T, soMatchblocks are counted, not applied; the posture is reported when the agent connects, so a changed configuration shows after the agent restarts. It reports metadata only (fingerprints, key types, paths, digests); no private key material is read or stored by discovery.The connector catalog reports sandbox truth, not description: each row in
GET /api/v1/connectors/catalogcarriesnative(this build has a native implementation),capabilities(the declared sandbox grant —fs.read,fs.write,net.dial,process.exec), andreplay_safety(reconciledwhen the receiver converges on retry, otherwiseat-most-once). Those come from the live registry, so the catalog cannot claim a capability the process would not enforce. Each row also says who executes the family in this estate (executor:host_agent,network_relay,control_plane, ornone— listed for its description, nothing runs it until the family is enabled here or a relay reports a plugin by that name) and where its grant was read from (grant_source): a factory-registered family declares its operation kinds at registration from a prototype of its own connector (A75), and a family an enrolled agent executes reports the shipped binary's census for the same connector code (A74), so a host family the control plane never enabled for itself no longer reads as having no implementation. A real attempt's per-target grant narrows the declared kinds to the target's paths and host; it never widens them.The running system is readable:
GET /api/v1/platform/system(andtrstctl-cli platform system) reports the build version/commit/date, the Go toolchain, process start time and uptime, the live signer topology (child/external/none), whether the FIPS module is active, and per-dependency reachability for the database, event log, and signer. It reuses the same probes as/readyz, so the console and a load balancer cannot disagree about whether the spine is up; it reports no addresses, DSNs, or configuration values. The same response and the System health console page, under Configuration evidence, includesidempotency_results: RLS-scoped counts of legacy, sealed, pending, and indeterminate mutation responses, plus the explicit fleet-readiness assertion and whether PostgreSQL has the validated sealed-only floor. Result bytes and raw datastore errors never enter this view; partial, failed, empty, recovery-required, and complete states carry concrete operator recovery guidance.Worker-pool backpressure is readable:
GET /api/v1/operations/bulkheads(andtrstctl-cli operations bulkheads) reports each bounded pool's workers, capacity, queue depth, saturation, and its submitted/completed/rejected/panicked counters, so AN-7 pressure is visible without scraping the metrics endpoint. The counters are process-wide operational telemetry — subsystem names and numbers, never tenant or credential data — and a control plane assembled without the bulkheaded surfaces answersserved: falserather than 404.SSH trust rewrite (the privileged
authorized_keys/CA-trust mutator): the applier that installs a trusted SSH CA and rolls it back on failure is wired into thetrstctl-agentbinary behind a default-off operator opt-in (--ssh-trust-add-ca) that additionally requires explicit confirmation (--ssh-trust-confirm) before it rewrites trust. The op is additive (it never removes existing trust), validates the new config withsshd -t, reloads, runs a separate operator-supplied post-reload health command (--ssh-trust-health-cmd) as a validated argv command line rather than a shell string, and auto-rolls-back to the last-known-good on any failure — so a bad rewrite cannot lock operators out. Reload success alone is not treated as health. Because weakeningsshd/authorized_keystrust is a high-blast-radius mutation, the feature stays off unless the operator turns it on and confirms; with the flag off the agent only discovers SSH trust, it does not mutate it. Trust removal still requires its own explicit confirmation.Approved SSH fleet trust changes (A226) are served through
/api/v1/ssh/trust-fleet/{preview,approvals,rollouts}, the matchingssh trust-fleetCLI, and SSH access → Trust rollout. Each enrolled host still needs local--ssh-trust-fleet --ssh-trust-confirm, a staged public CA and its own file/command policy. Skipped hosts stay in the exact reviewed roster and receive no job. Default two-person approval binds the CA, deadline, ready policies and skipped hosts; the requester cannot approve.ssh.trust.applyjobs have their own one-worker loop, and signed receipts prove file readback, effective global trust selection, validation, reload and the configured health command. Pending work never implies success; verified rollback stays attention_required. Completion must match retained source evidence. New work expires after its reviewed window; delayed jobs refuse before writing. This is additive user-CA trust: no remote command selection, trust removal, authorized_keys pruning, or host-certificate install. Backups are memory-only, so process/power loss can interrupt rollback. Do not edit host files concurrently; keep independent recovery access and backups. A receipt proves the configured checks at checked_at, not future host state. Real stock-OpenSSH served tests cover two applied hosts, an unchanged opt-out, strict host-key pins, retry and a checked failed-change restoration. Final image/console/restart replay and covering release gates remain qualification work; see the runbook.Posture collectors and agents: CT-log monitoring and path-based credential drift detection now run through the served discovery worker when operators create
ct_logordriftsources; findings are tenant-scoped and alert intents are outbox-backed. The Posture page has dedicated CT-log and drift dashboards (A105): each lists its own source kind's findings a page at a time, filters them by triage status, and shows counts the server takes over every finding of that kind, not a first-page sample. Resolution workflows (A106) run throughPOST /api/v1/discovery/findings/{id}/resolutionanddiscovery findings resolve: investigate with an assignee and a due date, record a fix with its evidence, accept the risk until a time within a year (the leader's sweep reopens it at expiry), or reopen. Posture reads leave a fix that held and an unexpired acceptance out of the open posture and count them; a fix seen again after it was recorded counts as open. The console's CT-log and drift dashboards offer the same actions on each row, show assignee, due date, overdue, acceptance expiry, fix evidence and regression, and filter by every status with the server's counts. Automatic remediation exists for certificate posture findings whose tenant rule asks for"remediation": "renew"(the lifecycle scheduler renews the deployed identity), not for SSH-trust or drift findings. Generic agent/endpoint discovery is served through the mTLSReportInventorychannel and visible through/api/v1/agents,/api/v1/discovery/findings,/api/v1/graph, and the Agents console. The default SSH host/trust collector runs on the enrolled host agent. An explicitssh_trustsource can instead use an enrolled network relay, pinned host keys and vaulted PKCS#8 credential references to call the installed read-onlytrstctl-ssh-collect-v1helper (A227). Itsssh.trust.collectjob reads the on-disk configuration throughsshd -Tand only root-allowlisted public CA, KRL and host-certificate files. The target needs no running agent; it does need Python 3, the fixed root-owned helper/policy and a restricted forced-command account. Match contexts, per-user keys, other SSH files and whether the running daemon loaded that disk configuration remain unassessed. Read failures and denied paths stay unknown; the remote target, source and collecting relay remain distinct in the fleet view. The key is encrypted at rest and in transit and is borrowed into locked relay memory for the bounded attempt; no credential value enters configuration, events, outbox or reports. See agentless trust collection. The credential graph and risk-scoring read APIs (/api/v1/graph*,/api/v1/risk/credentials,/api/v1/risk/contextual-priorities) are also served. The contextual-priorities response carries a canonicalurgent_summary: it merges the named credential-score and contextual-priority projections for one tenant, exposes both source counts, and deduplicates the union bycredential_id. Dashboard and Risk use that same answer, and a projection read failure is unavailable rather than zero. Critical/high discovery findings enqueue one same-transactionnotification.risk/risk.urgentoutbox intent from the same score band.status=completemeans every named projection read completed; it does not mean an undiscovered machine is safe, because discovery coverage still bounds what these projections can know. The AI/RCA/MCP surface is also served behindai.enable_api.Ownership depth (I1): owners carry an application/service model —
application_id,service,business_unit,environment— plus a storedescalation_chain, attributedownership_verified_by, verification time, and digest of the exact application/environment pair a human confirmed. The stored chain is deliberately distinct from the computed approver snapshot: the snapshot answers who could approve right now, the chain answers who to wake and in what order, and the approver graph cannot answer the second because it is about responsibility rather than permission. Every field is nullable and EMPTY MEANS UNKNOWN, never "none" — an estate predating this model has owners nobody can retroactively classify, and treating blank as a deliberate answer would hide exactly the rows the queue exists to surface.Store.ListUnownedIdentitiesreports four distinct reasons rather than one boolean (no_owner,owner_missing_application_model,ownership_never_attested,ownership_attestation_stale) because they need different actions: a data-entry gap, a classification gap, a first trust decision, and a repeated decision. Editing application ID or environment invalidates the prior digest immediately; no timestamp is allowed to pretend it confirms a different model. New steady-state deployments require current evidence at the configurable cadence (90 days by default) or an active identity-bound exception with authenticated grantor, reason, and expiry of at most 30 days. The scheduler writes one immutable request and one outbox notification per stale verification edge. Served through the owner and ownership-exception APIs,trstctl owners attest/exceptions, and the Owners console, which can create/edit the full model, attest it, separate all four queue counts, and grant a bounded exception. Expiry is enforced from event time and needs no cleanup timer to become effective.Asset-specific ownership assignment:
POST /api/v1/ownership/assignments,trstctl owners assign, and the Owners action queue accept one durable owner, 1–100 canonical NHI inventory IDs, and an attributed reason. The mutation is tenant-scoped, permission-gated, idempotency-protected, and capped at 2,000 reason characters and 1,024 characters per inventory ID. It emits one immutableownership.assignedevent. The projector updates the current override and, for native identities or certificates, the lifecycleowner_idin one transaction. A cold replay reconstructs both. An asset-specific decision wins over older native or imported owner hints; its event ID stays visible as exact attribution evidence. The NHI inventory and every NHI posture route honor the assignment for every source, so a reassigned discovery finding, API token or agent is no longer reported orphaned.Certificate CI export is opt-in and separate from ownership synchronization. A tenant can queue a certificate at
POST /api/v1/integrations/cmdb/certificate-exportsonly through its own enabled deployment-operator write binding. The binding pins the HTTPS instance, registered discovery source and file-backed or tenant-secret credential reference. The outbox worker sends public certificate metadata through ServiceNow IRE with the class fixed tocmdb_ci_certificate, then independently reads that exact CI and compares every exported field before recording a verification receipt. An HTTP success alone never completes it./api/v1/integrations/cmdb/certificate-exports/{id}retains the verified receipt after queue retention; an unverified command without a queue row reports recovery required. The Connect other tools console andtrstctl-cli cmdb certificate-exports create|showexpose the same result. Exports are requested explicitly; later lifecycle changes need another export. Rows lacking certificate bytes need verified DER recovery first. The TLS stand-in proves one certificate CI, duplicate-safe retry, revoked state, independent readback and tenant isolation; a live ServiceNow instance with its certificate class, identification/reconciliation rules, discovery source and read/write ACLs is still required for vendor qualification.Ownership provenance and CMDB reconcile (I2): owners record WHERE an ownership claim came from —
ownership_source(unset /manual/csv-import/cmdb),ownership_source_ref,ownership_source_observed_at. All three are nullable and unset means UNKNOWN, never "manual": an unrecorded origin is not evidence a human said so, and the Owners grid renders it as "not recorded". Two ingest paths write it.POST /api/v1/owners/import(trstctl owners import) takes a CSV;PUT/GET /api/v1/owners/cmdb-schedule(trstctl owners cmdb-schedule set|show) configures a per-tenant, leader-only ticker that re-reads ServiceNowcmdb_cion the tenant's own interval. Both funnel through one reconcile rule: a source FILLS IN what nobody recorded and NEVER overwrites what a human attested — that becomes a row inowner_ownership_conflicts, carrying both values and the source record that caused it, surfaced in an "Ownership disagreements" panel on the Owners console. A change to a value nobody attested IS applied and is still listed, because a change nobody was told about is how ownership data quietly stops matching reality. A blank cell is silence, not a deletion. Read-only is structural, not a flag: the network relay builds only GETs against a fixed/api/now/table/cmdb_ci, andorchestrator.NormalizeServiceNowTablerejectscmdb_cifor ticket writes. Neither path can acquire certificate-export write authority: that requires its own operator-configureditsm.servicenow.certificate_exportsbinding. A CI naming an owner this estate has never heard of does NOT create one; it is reported as unattributed, because a CMDB assignment group is not evidence that a trstctl owner should exist and auto-creating would build a parallel estate out of the CMDB's typos. The sync is RELAY-ONLY: the leader commits a tenant-scopedcmdb.syncoutbox job, and a NETWORK relay inside the segment claims it over its outbound channel. There is no control-plane HTTP fallback and therefore no private-egress hole into the estate. The relay redeems the ServiceNow token per attempt through the job-credential path (the approved ServiceNow binding setsrelay_only: truewith asecret://reference; anenv:reference belongs to the brain process and is refused), reads the one permitted table through the shared endpoint builder, parses in place, and reports records — never the raw response, bounded at 8MB. The RECONCILE stays in the control plane on the reported records, so the never-overwrite-an-attestation rule has exactly one implementation. The signed result is bounded before ingest and projected before the job closes; stable event identities make a crash/retry converge. One relay job reads at most 500 CIs in strictsys_idkeyset order. A full page commits an immutable page event, the exact next cursor, cumulative read count, current expected count, and the nextcmdb.syncoutbox intent in one tenant transaction. A short or empty page is the only terminal proof. Dispatch and intermediate pages never setlast_run_at;GET /api/v1/owners/cmdb-schedule,trstctl owners cmdb-schedule show, and the Owners console exposeread_count, optionalexpected_count,pages_completed,next_cursor,coverage_complete, and the retained failure. Restart recovery derives the same idempotent next job from the event/checkpoint, so it resumes rather than rereading page one.The relay verifies the CMDB's HTTPS certificate against host roots plus an optional operator-supplied
--relay-ca-bundlePEM. That file is separate from the control-plane--ca-bundle; adding appliance trust must not widen bootstrap trust. An invalid bundle fails agent startup. After startup a changed bundle file is re-read before the next request (checked at most every five seconds) and takes effect with no restart; an unusable replacement keeps the previous bundle and is reported, with the digest and load time of the bundle in force, asreported_relay_ca_bundleonGET /api/v1/agentsand on the Agents page. No skip-verification switch exists.The source inventory stores only CI key, matched local owner ID, and the four contributed ownership values. At terminal completion it withdraws a vanished/reassigned CI's still-identical, unattested values, never a human attestation or later edit, and serves changed/removed counts. One page is in flight per tenant; a second due tick leaves that incomplete checkpoint untouched instead of stacking an identical read. Scope, stated exactly: relay execution is mandatory.
POST /api/v1/owners/cmdb-schedule/run(trstctl owners cmdb-schedule run, the panel's Run CMDB sweep now) starts a sweep without waiting for the interval, as the scheduler's own durable relay job: it refuses an unconfigured (404) or paused (409) schedule by name, answers 409 naming the sweep already in flight rather than stacking a read, replays under its idempotency key, and starts a fresh sweep after a terminal page. A newly enabled schedule is still due immediately and fires within one scheduler tick. A disagreement is CLOSED throughPOST /api/v1/owners/ownership-conflicts/{id}/resolve(trstctl owners resolve-conflict), which REQUIRES both a reason and an attributed operator — "resolved" with no explanation tells the next reader nothing about which side was right, and an unattributed judgment cannot be questioned later. The command checks the tenant-bound open row under the projection lock before appending; unknown IDs return 404, and a second decision returns 409. The projection also pinsresolved_at IS NULLso a second event cannot overwrite the first judgment. The console provides a reason field and Resolve control for each open disagreement. The CI-to-owner mapping joins on the owner reference's EMAIL first (the page asks ServiceNow forowned_by.email,assigned_to.emailandsupport_group.emailbeside the display values) and on the owner NAME second, both compared without regard to case; an ambiguous key is no match. A CI whose mailbox and label both miss this estate's owners reconciles nothing and is counted as unattributed; no owner is created from it.Issuance requests as first-class objects (I3): the effect-free
POST /api/v1/issuance-requests/preview, the mutatingPOST/GET /api/v1/issuance-requests, and/{id}/approve|deny|cancel(trstctl issuance-requests preview|open|list|approve|deny|cancel) give a request a real lifecycle —requestedthenapproved,denied,expired, orcancelled, andapprovedis NOT terminal because issuance can still fail. Collapsing approved and issued would make a request whose mint failed read as fulfilled. Denial and expiry are deliberately distinct: a denial is somebody's decision and REQUIRES a reason (a denial with none teaches the requester only that somebody said no, so they re-ask); an expiry is nobody's, and the leader-only sweep records it with an EMPTYdecided_by, because stamping a person on it would put a decision in the audit trail that no human made. The requester can withdraw their own request and can never decide it — self-approval would leave an approval record that looks legitimate while nobody independent looked — and only the requester may cancel, so anyone else closing it is a denial and is recorded as one. Permissions reuse the existingcerts:request/certs:issuesplit rather than inventing a parallelcerts:approvethat could drift out of agreement with the gate guarding direct issuance. Requests default to a 7-day expiry; the list surface serves closed rows too and counts open separately, because one total cannot say whether a queue needs attention or is merely long with history. Direct API/console requests requireowner_id. Preview and submit share one admission path: they validate an optional public CSR, resolve a profile name to the exact active version, and accept only a syntactically valid owner UUID that resolves through the caller's tenant RLS context. Preview returns blockers, key custody, approval authority, and later submission effects but writes no event, projection, identity, certificate, outbox job, or idempotency row and contacts no CA. The console refuses to submit a stale, missing, blocked, or unavailable preview and links to Profiles, Owners, and CA hierarchy for configuration. Missing/malformed identifiers return 400 before event append; missing and cross-tenant owners share one 422 response so the endpoint does not reveal another tenant's roster. Historical requests and intakes without a unique configured owner match can still project with no owner; that absence stays explicit and blocks preparation instead of being backfilled with a guessed account. TICKET-DRIVEN INTAKE is now real:PUT/GET /api/v1/issuance-requests/intake-schedule(trstctl issuance-requests intake-schedule set|show|list|delete) configures named ServiceNow or Jira reads within a tenant.nameis a 1-40 character lowercase slug; omitted names and retained legacy jobs bind only todefault. The list route isGET /api/v1/issuance-requests/intake-schedules; exact reads and removals take?system=servicenow|jira&name=<name>. Removal preserves requests and audit history and refuses an unfinished sweep or pending relay job. The Approvals console can configure either provider, shows its exact HTTPS destination, secret reference, ticket source, field mapping and enabled state for review, and retains the idempotency key when a save response is uncertain. Retrying the exact same schedule, even with a fresh HTTP idempotency key, returns its retained sweep cursor without replacing the running job. Changing a schedule during an incomplete sweep returns 409 until that sweep reaches a terminal page. The console reports schedule acceptance separately from relay read coverage. Each(tenant, provider, name)has its own schedule, job lane and cursor, so one table or project cannot replace or suppress another source's progress. An optionalowner_fieldreads an exact mailbox: a ServiceNow scalar or dot-walk such asassigned_to.email, or a Jira scalar or dotted user field such asassignee.emailAddress. Jira privacy settings may hide that field. Exactly one matching tenant owner suppliesowner_id; missing, ambiguous or foreign owners remain unassigned. The console shows that absence. Existing requests retain their originally recorded owner; correcting the mapping applies to new tickets, and an unassigned request needs a new ticket/request naming a real owner before issuance. Named jobs require upgraded relays, because an old report cannot identify their schedule. ServiceNow is bounded to one of the four request-shaped tables (incident,sc_req_item,sc_request,change_request); Jira is bounded to one validated project key. Both bounds are database CHECKs, because an unbounded source name would aim the intake token at records that are not this tenant's certificate-request queue. The intake opens one issuance request per ticket, IDEMPOTENTLY by ticket reference: a re-seen ticket opens nothing, and a DENIED request does not reopen — the denial was the answer to that ticket, and a fresh ask needs a fresh ticket. The field mapping is explicit (subject_field,profile_field, optional requester/justification); a ticket missing the mapped subject or profile is SKIPPED AND COUNTED, never guessed at, because an intake that opened requests from prose would fill the approval queue with noise. Requests opened here carryorigin=servicenow|jiraand the exact provider ticket reference, and the existing lifecycle — separation of duties, denial with a reason, expiry (7 days for intake-opened requests) — decides them unchanged. Each sweep first commits a tenant-scopedticket.syncoutbox job. A NETWORK relay redeems the provider'ssecret://token for one attempt and performs the only external call; the control plane has no ServiceNow/Jira HTTP or token fallback. ServiceNow advances by strict ascendingsys_idkeyset pages and Jira advances by the provider's opaque enhanced-searchnextPageToken, with a hard 100-ticket page bound. The signed report must echo the exact provider, sweep ID, input cursor, source references, read count, and provider total. The page event and next outbox command commit together. Dispatch or failure updateslast_attempt_atand retained error only;last_run_atandcoverage_complete=trueappear only after a terminal page whose observed count equals the provider total. The API and console serveread_count, optionalexpected_count,pages_completed,next_cursor, eligible/skipped counts, retained failure, and terminal coverage separately. Replay, relay retries, scheduler restart, snapshot restore, and cold event rebuild therefore resume the same cursor; stable request/page identities collapse duplicates without skipping later tickets. The GITHUB ACTION lives in-repo atclients/github-action(composite,action.yml+ README with the sample workflow): the workflow's ambient OIDC token is fetched with the requested audience, an EC key is generated INSIDE the runner (only the public half travels), andPOST /api/v1/workloads/attested-issuancewithmethod=github_oidcreturns the certificate. One certificate belongs to the GitHub-signedrun_idandrun_attempt, scoped by issuer, repository and tenant. A fresh runner on a new signed attempt receives its own certificate. Within an attempt, the same effective command/public key recovers the original; a second key or changed effective command returns HTTP409 even under a new caller idempotency key. Unsigned environment changes cannot allocate an attempt. Missing/noncanonical signed coordinates are refused. Issuance evidence retains each certificate's run ID, attempt and original verification time; earlier certificates expire normally. The Action preserves existing runner key and certificate files, so a new attempt needs a fresh output directory. An unfinished mint whose public template was lost requires its original bookkeeping to be restored; the signer journal refuses changed signing input. Fork pull requests are refused locally before OIDC, and the server still verifies the signedrepository_owneragainstallowed_owners. CI extracts and executes the shipped composite run block against the full served test deployment, then proves success, foreign-owner and same-attempt mint refusals, a successful new signed attempt, and unsigned-attempt refusal. Local native tests also cover concurrent issuers and source-backed recovery after cache cleanup/read-model rebuild. A real Actions run still needs the owner's push. The tag release pipeline packages the Action,SHA256SUMS, and SLSA provenance. Repository state is release-ready forv0.6.0, but that immutable tag is not published until an authenticated maintainer pushes the reviewed commits and creates the tag; the current local remediation does neither. The Action no longer needs an API token: agithub_oidctrust source created withself_authenticating: trueand anallowed_ownerspin lets the workflow send its verified OIDC token as the bearer credential ofPOST /api/v1/workloads/attested-issuancewith the tenant inX-Tenant-ID(the Action'stenantinput). That principal isgithub:plus the repository name (for examplegithub:acme/payments), holdscerts:issueon the attested-issuance routes only, and must present the same token aspayload_base64; the server refuses to flag a source without an owner pin, and a disabled or revoked source closes the door again. Atokeninput scoped tocerts:issueremains supported. The live proof on a real GitHub Actions run still needs the Action published from a pushed tag. Approving a request does not itself mint. The servedPOST /api/v1/issuance-requests/{id}/preparecommand creates or recovers one deterministic requested identity, keeps the request atapproved, and returns only the stored public CSR plus a stable request-derived issue key. The console sends that identity through the ordinary guardedrequested -> issuedtransition, so RA separation, policy, optional dual control, profile binding, signer isolation, outbox delivery, and idempotency are not bypassed by the convenience journey. Finally,POST /api/v1/issuance-requests/{id}/completemoves the request toissuedonly when its linked identity has an immutable issuance transition AND inventory contains its real certificate under the exact canonical issue key. The signing worker also completes this receipt, so closing the browser cannot lose it. Recovery after revocation or retirement records historical fulfillment without reactivating the credential or signing again. Reviewer (decided_by) and original issuance actor (issued_by) remain separate facts;issued_atis the retained certificate-recording time, not the later recovery time. A signer or outbox failure therefore leaves an honest, retryableapprovedrequest instead of producing a false green status. The console keeps that approved row visible, explains the interruption, and labels the recovery action Retry safely; prepare and issue repeat with the same deterministic identity and stable request-derived issuance key rather than creating a parallel operation.Attested issuance is reachable (AUD-10, I3 prerequisite):
attested_issuancein the config file turns onPOST /api/v1/workloads/attested-issuanceandPOST /api/v1/ssh/attested-user-certs. Before this there was NO config key at all —Deps.AttestedIssuancewas never assigned anywhere in production, so both routes were registered, documented, and permanently 503 on every deployment, and the six attestors behind them (including the GitHub OIDC attestor a CI pipeline needs) were constructed by code no request could reach. Off by default is correct for a mint that trades a cloud attestation for a certificate; unreachable when on was the defect. Attestors stay per-tenant from the workload attester-trust API rather than process-wide, so one tenant's trust decision does not become every tenant's. The other two members of this family were wired after their deadDepsfields were found:agent_brokercan turn onPOST /api/v1/broker/agent-identities, whilepamreaches the broker constructor forPOST /api/v1/access/sessionsand its companion routes. The operator can seed tenant-bound PostgreSQL provider references and SSH host/principal allowlists in the validatedpamconfig block; tenant registrars can add and disable targets through the served console, API, and CLI. An enabled broker can start without targets so the first registration is possible. PAM resolves attestors from enabled tenant-managed workload trust at request time. The target model requires an explicit PostgreSQLreadonly/writerrole allowlist, and rejects an SSH target without an explicit host, port, and principal allowlist. A Postgres DSN must be loaded by an existing tenant PostgreSQL dynamic-secret provider from a protected file or tenant secret. The injected PAM target contains only that provider ID; its issue and remove effects use the provider's durable outbox. PostgreSQL's native role deadline prevents a worker outage from extending access, while PAM's final expiry event waits for confirmed provider removal. The requester proposes an exact session withPOST /api/v1/access/session-requests; a configurable quorum of distinctaccess:approvereviewers must approve its immutable digest beforePOST /api/v1/access/sessionsconsumes that authority. An altered command or target binding cannot reuse the approval. The local browser activation, stock PostgreSQL use and expiry, and paired cold restart have been exercised. Early revocation is now requested through the tenant API and console; PostgreSQL completion waits for native role removal, while SSH completion means the served KRL contains the serial and key ID. Hosts must load the refreshed KRL to enforce rejection. A local J4 host with the current CA rejected a freshly revoked PAM certificate before its expiry and retained that rejection through host and control-plane restarts; this proves one watched host, not central fleet delivery. The provider terminal-failure path now projectsrevocation_failedwith a closed error class and exposes a fresh-key retry in the API, CLI, and console. A live PostgreSQL cross-database dependency exhausted the native removal attempts; its console retry removed the exact role and survived a cold restart. The KRL endpoint returns 503 during a catch-up it cannot finish within its read deadline, and an opted-in host retains its installed list and retries./readyzproves KRL and audit-read readiness after a restart:ssh_krlreads not ready while the revocation list cannot be brought current andaudit_readwhile audit history cannot be read, so a replica restarted behind the revocation stream or before a restored history's cutover stays out of rotation. A host agent refuses a KRL whose version does not advance its installed one (a list that revokes nothing, such as the empty onessh-keygen -kwrites when a host is prepared, yields to a list of the same version, so a new host reads current before the first revocation), and refuses any KRL, whatever its version, that would stop revoking a serial, key ID, key, or key hash the installed list revokes, so a restored or replacement control plane, which counts its own versions, cannot silently un-revoke anything on a host; before this guard a newer list that omitted an installed serial replaced it. A restored or replacement control plane takes over a lineage by adopting the list its hosts hold (trstctl ssh krl adopt): it records every predecessor revocation for any CA and moves its version past the predecessor's, and a served test proves a host refuses the replacement list before adoption and takes it after, with stockssh-keygenreading the result. One adoption reads at most 4 MiB and 100,000 entries. Per-host receipts exist for a watcher run beside an enrolled host-role agent: each heartbeat carries a receipt, signed with the agent's certificate key, of the digest and version read back from itsRevokedKeysfile and how its latest attempt ended, andtrstctl ssh krl receipts(and the SSH access console page) places each host against the served list (current, behind, refusing, failing, stale). A standalone watcher still reports nothing, a receipt is the host's own account rather than a login attempt against its sshd, and there is no control-plane push. A lab proof with real hosts moving between two control planes, and receipts from a multi-host lab fleet, remain to be shown. PAM sessions created before the verified-attestation event field was added retained an attestation ID and audit summary. Their verified selectors, claims, and verification time cannot be reconstructed from that start event. The API omitsattestationfor those sessions and the console names the evidence gap; new sessions project the full verified, non-secret facts after replay. A malformed TTL on any of the three leaves zero so the built-in bound applies — silently substituting a LONGER lifetime than the operator wrote is the dangerous direction, and zero cannot do that.MDM device correlation and per-device enrollment trace (I5):
GET /api/v1/mdm/devicesandGET /api/v1/mdm/{mdm}/devices/{id}/trace(trstctl mdm devices|trace) join Intune and Jamf device records to SCEP transactions and answer the question a status field cannot: WHICH STEP an enrollment broke at. The trace runsrequested -> issued -> installed -> renewingand the distinctions are the feature. A step nobody reported ispendingonly if an earlier step SUCCEEDED — otherwise a device that never asked would look mid-flight and sit in an "in progress" queue forever. Steps after a failure areunknown, notpending: nothing downstream was attempted, andpendingsuggests it still might happen. Only the FIRST failure is reported as the break, because reporting the last sends an operator to the symptom rather than the cause. An MDM that could not be reached yieldsunknown, never "not installed", and the list countsunobservedseparately fromfailed— merging them sends somebody to re-push a profile that is already there. Intune'sdeviceRegistrationStateis ALWAYSunknownfor certificate installation: it says that the device is registered, not that a SCEP profile or certificate installed. Intune installation evidence comes from a completedCertificatesByRAPolicyexport filtered to the exact enabled SCEP profile IDs. Jamf reads the fixedGENERAL,HARDWARE, andCERTIFICATESinventory sections. Neither provider can produceinstalled=okuntil its exact device record contains the signer-minted certificate serial with an active/valid status; a completed read with a different serial is an actionable failure, while no certificate-specific read remainsunknown. READ-ONLY IS STRUCTURAL — inventory uses fixed GET paths (/v1.0/deviceManagement/managedDevicesand/api/v1/computers-inventory). The one non-GET provider operation is Microsoft's fixedPOST /beta/deviceManagement/reports/exportJobsfor theCertificatesByRAPolicyREAD artifact; it cannot name a policy, assignment, or device mutation path, and the Graph bearer is never forwarded to the signed report-download authority. There is no MDM write route;TestNoMDMCodePathCanWriterejects PUT/PATCH/DELETE and any mutation-capable resource fragment. Devices with no matching certificate and certificates with no matching device are BOTH reported, because a correlation that showed only its successes would make an estate look covered by hiding the gaps; the join key is the hardware serial, matched case-insensitively, because device NAME would silently join two laptops an admin happened to name the same. THE SURFACE NOW HAS ITS PRODUCER:PUT/GET /api/v1/mdm/poll-schedule(trstctl mdm poll-schedule set|show) configures a per-tenant, per-MDM schedule and a leader-only ticker re-reads Intune/Jamf on that interval. An identity inventory join remains metadata only; the lifecycle trace joins the MDM hardware serial to immutableprotocol.scep.request.observedandprotocol.scep.issuance.observedfacts emitted by the actual SCEP handler, keyed by the exact CSR common name and transaction. A successful issuance fact carries the inspected signer-minted certificate serial, fingerprint, and expiry; every terminal refusal carries its failed stage and remediation. Correlation IDs and identity lifecycle rows never manufacture success. The sync is RELAY-ONLY: the leader first commits a tenant-scopedmdm.syncoutbox intent, a NETWORK relay claims it over the outbound agent channel, and that relay redeems thesecret://token for one attempt. Enabled schedules requirerequired_agent_idnaming one enrolled network relay with a fresh authenticated heartbeat. The outbox claim and credential redemption both enforce that exact agent ID. Upgrades disable older enabled schedules without an ID and retire their pending unbound jobs; the operator selects a relay and re-enables the schedule.env:references,control_planeexecution, and the oldallow_private_endpoint/private_egress_cidrsbrain-egress grant are refused. The fixed endpoint is parsed in place and only a bounded typed, signed observation returns. The control plane binds that report to the exact durable job payload, projects it before atomically completing the claim and outbox row, and performs correlation through one shared core; it has no MDM HTTP/token fallback. The in-flight pending check reads the provider from the durablemdm.syncintent: a pending Intune read stops duplicate Intune work but does not suppress Jamf (and vice versa). Each provider's schedule stamps its own waiting or completed outcome instead of reporting invented health. RENEWAL-WINDOW AWARENESS FOR OFFLINE DEVICES is served on the device list: each correlated device with a certificate carriesrenewal_at_riskand a detail naming the dates, computed from the identity's expiry against the DEVICE'S OWN last check-in (IntunelastSyncDateTime/ JamflastContactTime— never the poll time, which would make every device look fresh on every poll and defeat the check). A device inside its renewal window that the MDM has not seen since the window opened is flagged: a SCEP device renews by CHECKING IN, so nothing fails before the certificate expires in a drawer — and the count is served apart fromfailedbecause nothing has failed yet, which is the problem. The window is 30 days unless the schedule setsrenewal_window_days. Scope, stated exactly: no built-in OAuth client-credential exchange — the token reference must resolve to a bearer the MDM accepts, rotated by the operator's own pipeline; a device with no certificate gets NO renewal verdict rather than a warning. The trace'srenewingstage is now evidence-backed: a later distinct SCEP transaction supplies its request/result, while a certificate inside its renewal window with no later attempt and a stale device check-in isunknownwith instructions to bring the device online and trigger an MDM check-in. The console device row opens this complete trace and shows the first failed stage plus its remediation. Intune report export currently uses Microsoft Graph's beta report endpoint; the operator-supplied bearer must already authorize both managed-device reads and report export because trstctl does not perform an OAuth credential exchange.AD CS coexistence, first increment (F4): the
/certsrvtransport now REFUSES to send a password over plaintext. Basic is base64, not encryption, and on a plaintext hop anyone on the path reads a credential that can issue from the enterprise CA directly — the compromise is not "an eavesdropper saw a request" but "an attacker can mint certificates the whole domain trusts". The refusal is at CONSTRUCTION, so a misconfiguration fails when an operator sets it up rather than silently on the first production issuance. A domain-joined relay can supply Kerberos/NTLM through aWebEnrollmentConfig.Authenticator, an interface rather than a GSSAPI dependency because the credential belongs in the host's credential store and core must have no code path that could serialize a domain password. Configuring BOTH a password and an authenticator is refused: the authenticator would win, leaving a live domain credential in configuration that nothing reads — which is exactly what somebody later "fixes" by making it take effect. A failing authenticator fails the request and never falls back to Basic, so a transient Kerberos problem cannot put the password on the wire. AD CS certificate-database rows parse into a vocabulary that keeps PENDING distinct from FAILED and from DENIED (a pending request rendered as failed makes an operator re-submit instead of going to get it approved; a human decided a denial), an unrecognized disposition code maps tounknownrather thanfailedso a code Microsoft adds does not report healthy certificates as broken, serials normalize so the same certificate from two exports is not two inventory rows, and an unparseable NotAfter is counted as a VISIBILITY GAP rather than treated as an expiry in year zero. Scope, stated exactly: this is the transport and parsing layer only. NOTHING YET CALLS THE DATABASE INGESTION — there is no scheduled or served ingest that pulls a CA database into inventory, so issued/pending/revoked state is not reconciled by C4/XREC; no Kerberos or NTLM Authenticator IMPLEMENTATION ships (the seam exists, the relay-side implementation does not); the ACME/EST front-end that would issue a leaf through an AD CS template under a trstctl policy gate is not built; and there is no "modernize this template" console action or AD CS issuer health view.Staged agent upgrades with an automatic canary halt (A5):
GET/POST /api/v1/agents/upgrade-campaign,/pause,/resume, andPOST /api/v1/agents/upgrade-ring(trstctl agents upgrade-campaign show|start|pause|resume,trstctl agents upgrade-ring) run a rollout through canary, early, then broad rings, and a leader-only sweep advances it. The halt is the product: ONE unhealthy agent halts, with no percentage tolerance, because a tolerance on a deliberately small canary means the ring can never stop anything — which is the entire reason the ring exists. SILENCE halts too; an agent that took an upgrade and stopped answering is the most likely shape of a bad build, so it is never scored as a success. An EMPTY ring halts as well: an unassigned canary proves nothing, and advancing through it would skip the stage whose failure is supposed to stop the rollout, on exactly the fleet nobody has triaged. Resume restarts AT the ring that halted, never past it — skipping ahead would leave the agents whose failure stopped the rollout on the broken build while the campaign reported success — and a halted campaign never un-halts on its own, so a flapping agent cannot resume a rollout nobody re-approved. Pause GATES DISPATCH (the sweep returns early andfleet.Advancerefuses independently), not merely the button; a pause that grayed out UI while jobs kept flowing would be worse than none because the operator believes they stopped it.upgrade_ringempty means UNASSIGNED and is never read asbroad, and the console counts unassigned separately. Halted and paused are distinct states: one is the machine's finding, the other a person's decision. A campaign started WITH per-platform artifacts (artifacts: [{os, arch, url, sha256}]) DISPATCHES: the sweep hands every agent in the active ring its ownagent.upgradejob through the A1 ledger, each row narrowed to that one agent byrequired_agent_id(the control-plane dispatcher structurally refuses such rows — only the named agent's claim can reach them). The agent — with the-self-upgradeopt-in, which is the MACHINE operator's consent and is off by default — downloads its platform's artifact, verifies the pinned sha256 (a mismatch is refused and reported; the artifact host is a mirror, not a trusted party), swaps its binary keeping the previous one as.old, reports a SIGNED receipt, and restarts (exec on Unix; on Windows the process exits and relies on the service's recovery action — an agent that stays down reads as silence and halts the ring, which is the correct verdict). The ring is then scored against the DISPATCH LEDGER: a signed failed receipt halts immediately with no grace; an agent observed running the target version verifies (the executed receipt alone means "staged" — the reconnected version report is the proof the new build runs); an agent that neither failed nor arrived within the 10-minute grace is silent, and silence halts. Resume re-dispatches the halted ring as a NEW round, so the failed round's receipts stop counting against the retry. Scope, stated exactly: a campaign with NO artifacts is OBSERVE-ONLY — the pre-dispatch behavior, kept for fleets an external mechanism upgrades, where verification is "the agent reports the target version and has been seen since" and nothing is pushed; the console says which mode a campaign is in. The artifact sha256 is operator-supplied — the platform guarantees the fleet installs exactly those bytes, not that those bytes are good. An artifact may also carry its publisher's Ed25519 signature over that sha256: a control plane that pins the publisher key (agent_channel.upgrade_publisher_key_file) refuses to open a campaign whose artifacts it does not verify, and an agent started with--upgrade-publisher-keyrefuses to stage an artifact that is the pinned bytes but not the pinned publisher's. With neither key pinned the digest is the only integrity control, which is the default. Ring assignment is by hand per agent (POST /api/v1/agents/upgrade-ring) or by one deterministic placement (POST /api/v1/agents/upgrade-rings/auto-assign,trstctl agents upgrade-rings auto): unassigned in-service agents only, heartbeating ones first, then longest enrolled, then by name; the firstcanary_count(default one) become the canary, the nextearly_percent(default ten) the early ring, the rest broad; every ring keeps at least one agent, an operator's own placement is never moved, and a fleet with fewer than three unassigned agents is refused rather than planned into an empty ring that would halt the campaign. The rule is served with the result so it can be disagreed with before the campaign starts. A campaign opened withsoak_secondsholds the next ring until that long after the current ring was dispatched (for a campaign that dispatches nothing, after it entered the ring), and only while the verified ring stays healthy (the reason says until when; an agent that stops heartbeating during the soak is silence and halts). Recording the soak neither restarts the window nor dispatches the ring again. Zero, the default, advances a verified ring at once, so the 10-minute grace is the only wait unless an operator asks for a soak. Verification is more than a health signal: the first sweep that sees an agent on the target build queues it anagent.self_checkjob pinned to that agent (enabled together withagent.upgrade), and the new binary claims it, exercises what an upgrade can break without stopping the process — that it is the build the campaign targets, that a host executor's exec profile still names roots and commands that exist on the machine, that the relay's plugin runtime loaded — and reports a signed receipt naming each check. The agent verifies only once that receipt says every check passed; a failed check halts the ring with its own reason, and a check unanswered past the grace is silence. The campaign view and console tally the live round's checks. The check is bounded on purpose: it proves the agent still works as a job executor, not that every connector it carries deploys correctly on this estate.Provider tenant templates:
/provider/v1/templatesrecords versioned templates of customer objects,/provider/v1/tenants/{id}/templateonboards a customer from one (registering its core tenant when needed),.../template-driftreads its drift, and a rollout applies a new version across customers after a preview whose fingerprint it must match. What a template can say is what core holds: "roles" are role grants to named members, built-in or a custom role the tenant has already defined (a template does not define custom roles); a CA binding is an issuer record (a changed chain binds a new issuer and the old one stays for what it issued); connector defaults are deployment targets. A template's report schedule can carrydispatch, announcing each run on the customer's channel of that type. The Provider console does not show templates yet; they are served on the Provider API. A console view is what comes next.Provider console (L3 + AUD-58): the web app has a
/providerroute — a console for the provider's own staff, separate from the tenant plane. It lists customer tenants with their lifecycle state and drives the/provider/v1API the plane already served but no web client consumed: provision a customer, suspend, offboard, each mutation confirmed because one click changes a whole customer's world. The plane's operator is authenticated by the provider IdP (L1), so the console carries an operator bearer rather than a tenant session, and holds it IN MEMORY ONLY — never web storage — per the SPA's XSS posture (SURFACE-I01): the token lives in the tab's JS heap, is lost on reload, and an auth refusal returns the operator to the sign-in gate rather than showing an error. Each customer row expands to show its quota (max agents, certificates, secrets), reading/provider/v1/tenants/{id}/quota; an UNSET limit renders as "unlimited", never zero, because a missing cap is the absence of a limit, not a limit of nothing. Scope, stated exactly: this is the tenant-lifecycle console with per-customer quota MANAGEMENT — each customer row expands to view and EDIT its limits (max agents, certificates, secrets), a blank field saved as unlimited never zero — and per-customer WHITE-LABEL BRANDING: a brand editor (product name, custom domain, login message) that reads the current delegated brand and writes through/provider/v1/tenants/{id}/brandwithIf-Match, a route wired to the tenant-scoped provider authority event receiver and its white-label read projection, behind the same per-customer delegation gate (an operator can brand only a customer they are delegated, because a custom domain is a claim on a host and branding another's customer could seize it). A duplicate custom domain is refused before event append, and replay retains a rejected outcome for older conflicts. Customer lifecycle, quota, brand, delegation, and break-glass state now rebuild exactly from one immutable provider authority history; the six PostgreSQL views, now includingprovider_operators, expose no production mutator. Every provider mutation requires a key bound to operator + method + path + body, and the console sends one; identical sequential or concurrent retries return the original HTTP bytes, while changed commands return 409. The console also triggers the served siloed-isolation drill and shows its checks. Its Authority activity panel readsGET /provider/v1/activityand renders newest-first event id/sequence/type/time/customer/actor evidence derived from the same immutable authority history. Current delegation filters customer events before serving, global drill evidence is admin-only, and the response deliberately omits command bindings, authority payloads, and break-glass snapshots. AUD-58 adds the Provider-admin/MFA access panel. It lists SCIM operator lifecycle/source/role and exact customer-operation grants with expiry, last use, and retained revocation; it grants, revokes, and changes the Provider role through idempotent event-backed routes. SAML login is discovered from/provider/v1/auth/methods; the browser uses the separate HttpOnly Provider session and sends only the non-credential CSRF value from JavaScript. OIDC bearer input remains as a memory-only option for operator tooling. The localtrstctl provider-grantcommand remains the install-time bootstrap path; AUD-59 adds invoice evidence in the same console: an operator selects a delegated billing customer and closed period, pulls the signed JSON without impersonating a tenant, sees billable/reconciliation/digest plus independent Signature verified posture, and downloads either the signed JSON or the finance CSV whose every row retains customer, period, verdict, and digest. AUD-60 adds Customer health beside that Invoice evidence for the same selected customer.GET /provider/v1/tenants/{id}/healthrequires the exact customer'sreaddelegation beforeDirectTenantSnapshotopens the customer's forced-RLS certificate transaction. It reports lifecycle-derived health and the active-certificate count; a missing customer is 404 and an unavailable snapshot is an explicit 503. Before a pull, or after a failed read, the console says health is unknown/unavailable — it never turns missing telemetry into a healthy zero. Another customer's active certificates cannot enter the count because the query carries and is confined by the selected customer's tenant identity. The break-glass request, consent, and result-use APIs are exposed in this console (A93): the exact-customer emergency panel lets a requester open a bounded request without offering self-approval, offers consent only to a distinct approver on the selected customer, shows the narrow result only to the requester after two approvals, names a requester withdrawal apart from an approver denial, and shows the queue without emergency mutation controls in license read-only mode.AD CS certificate-database lifecycle visibility (F4, PARTIAL):
POST /api/v1/adcs/ca-database/ingestandGET /api/v1/adcs/ca-database(plustrstctl adcs ca-database ingest|listand a Posture console panel) turn certutil rows a domain-joined relay collected into a per-CA breakdown by disposition — issued, PENDING a CA manager's approval, revoked, denied, failed. Pending is kept distinct from failed and denied throughout, because the fix for a pending request is approval, not resubmission; an unrecognized disposition code isunknown, never folded into failed; an issued row whose expiry could not be read is countedunparsed; and a row carrying no request id is countedrejected, never dropped — so a collection problem cannot read as an empty or healthy CA. This wiresadcs.ParseDBRowandadcs.Summarize, which existed with NO production caller — parsing that nothing ran and a summary nobody saw. It is deliberately VISIBILITY, not control: trstctl reads the CA database, it does not approve or revoke through this surface. Scope, stated exactly: the control-plane ingestion, projection, serving and console are built and proven end-to-end (a relay posts rows, the per-CA summary reads back, a re-sweep replaces rather than accumulates); the relay-side COLLECTION is not — the CA database is read bycertutilon the domain-joined Windows relay, which this repository does not yet drive, so the rows arrive over the ingest endpoint but nothing yet runs certutil to produce them. The rest of F4 — in-domain Kerberos/NTLM authentication (anAuthenticatorinterface, not a Kerberos implementation) and fronting AD CS templates with trstctl ACME/EST — remains unbuilt.PQC readiness reports are signed and offline-verifiable (M1, PARTIAL): a cohort's handshake evidence — every targeted client's outcome WITH its handshake size and latency — is turned into a verdict by
Assessand exported as a readiness report signed inside the isolated signer under its ownpqc-readiness-reportkey, verifiable offline from the report alone. The signer key refuses every other artifact kind, so being in the shared signer chain grants it no authority over XREC or VDEC artifacts; the signature covers the whole body, so an edited verdict cannot pass verification, and the report carries the FULL evidence so a reader re-derives the verdict rather than trusting the sentence. Cost travels in the signed body on purpose — a cohort that negotiated but tripled handshake size is a different recommendation, and a report omitting cost would recommend an outage. Scope, stated exactly: this is the EXPORT and the verdict rule, not the lab. Parallel classical/PQ pilot hierarchies are served (A26/A27): a root or intermediate ceremony whosesignature_algorithmisml-dsa-44,ml-dsa-65orml-dsa-87mints that key in the isolated signer, the authority issues classical and pure ML-DSA leaves, and its CRL and OCSP responses carry its ML-DSA signature, all beside the classical hierarchy. Pilot waves with a canary halt are served (A28/A29):POST /api/v1/pqc/pilotsrecords an ordered plan of endpoint targets bound to the exact leaf each must present, dispatches the canary wave as one relay endpoint-verification sweep through the durable outbox, classifies the relay's signed transcripts (negotiated, rejected, unreachable, with handshake size and latency), takes the compat-lab verdict over everything probed so far, and halts the instant a canary refuses or cannot be reached — later waves are never dispatched — or dispatches the next wave; a finished pilot's evidence is signed inside the isolated signer as the readiness report, verifiable offline. The console's Posture page carries the lab view (A30): start a pilot, follow each wave's outcomes with handshake latency and chain size, the verdict that halted or advanced it, the skipped waves, and the signed report. The acceptance's end-to-end pilot on the lab is not yet demonstrated.Per-tenant silo isolation identifiers all key on the tenant ID (L4): a siloed tenant's Postgres schema, JetStream subject lane, and object-key prefix are each derived from the tenant's unique, immutable ID. The event LANE used to be the exception — it was derived from the operator-supplied slug, which has no charset validation and normalizes lossily, so
acme-corp,acme_corpandacme.corpall collapsed to one lane and two distinct tenants with those slugs would have SHARED a JetStream stream — one customer's events landing in another's, a cross-tenant isolation breach in the sovereignty feature. The lane now carries the slug only as a readable prefix with the ID as the uniqueness suffix, and a structural assurance test proves distinct tenants stay disjoint on all three axes (schema, lane, object prefix — with no prefix a prefix of another's) even when their slugs collide. Scope, stated exactly: this hardens the derivation and adds the assurance. The per-tenant audit stream is served (A94): a siloed tenant's lane gets a JetStream stream of its own,TRSTCTL_AUDIT_<lane>, that sources exactlyevents.<lane>.>from the live events stream (it accepts no publishes of its own, so it cannot diverge from the log), with its own name, retention, and deletion; the silo provisioner ensures and drops it with the lane, and the isolation drill'sevent_audit_streamcheck proves on live JetStream that each probe tenant's stream received its own probe event and none of its neighbour's, then removes the probe streams (event_audit_stream_cleanup). Siloed placement itself is still exercised by the drill rather than by customer provisioning. Tenant-scoped break-glass with dual consent is served (A95): a customer setsPUT /api/v1/provider-access/policy(trstctl provider-access policy set) to require its own consent and how many distinct approvals it takes; the Provider's emergency requests appear under the customer atGET /api/v1/provider-access/requests(projected from the Provider's ownprovider.breakglass_requestevent recorded under that tenant); each customer subject withaccess:writeapproves or denies a grant once throughPOST /api/v1/provider-access/consents/{grant_id}(a denial needs a reason), recorded as immutabletenant.provider_access.*events in the customer's own history. After both Provider custodians approve, the Provider'sPOST /provider/v1/breakglass/{id}/resultsis refused withtenant_consent_pendinguntil the customer's approvals are in and withtenant_deniedafter a customer denial; the Provider queue shows the customer's standing. The provider assurance readout is served (A96):GET /provider/v1/tenants/{id}/isolationreturns, under the customer's read delegation, the isolation the deployment actually serves that customer (the model, PostgreSQL schema, JetStream lane and object prefix from the live tenancy router, and the customer's audit stream with its event count as JetStream reports it, or a note saying why there is none); the Provider console shows it in each customer's setup view beside health, and the isolation drill with its lane and audit-stream checks sits on the same page. Per-tenant HSM custody is served (A123): a tenant can migrate its key domain to apkcs11wrapper the isolated signer holds. The signer creates the tenant's domain KEK, keeps it wrapped by an AES-256 key the operator provisions on an HSM and reaches over PKCS#11, and wraps and unwraps the tenant's data keys itself; the control plane persists only the wrapped KEK and never holds the KEK. Each data-key wrap or unwrap is one signer call that opens the KEK from its wrapped form inside the signer and destroys it after. Seal, unseal, rotation (including onto another HSM key) and the history rewrap work as they do for a local wrapper, and a lost HSM key fails only that tenant closed (wrapper_unavailable), never falling back to the deployment KEK. It runs in the cgo signer (thesigner-hsmimage) under the BYOK license and is proven against SoftHSM in a container; proof against a hardware HSM needs hardware and has not been run. Stated limits: the data keys themselves return to the control plane, which decrypts tenant data with them; the PKCS#11 binding hands the decrypted KEK back through a C buffer it frees without zeroing (the signer is non-dumpable and refuses ptrace, and the Go copy is wiped); and every tenant-crypto callback for such a tenant makes signer calls, each an HSM decrypt, inside the signer's in-flight bound. What comes next: one non-extractable HSM key per tenant, so the tenant's KEK is itself an HSM object and data-key wraps run inside the HSM. The product ships an S3-compatible object-store client that keeps each tenant's objects under the object-key prefix the tenancy router resolves (tenant/<id>/, orsilo/<id>/for an isolated tenant), and the doctor's operator-run isolation suite proves that confinement against the deployment's configured store (OBJ-1 to OBJ-3 indocs/operations/doctor.md; OBJ-LOCK checks the bucket has Object Lock when the deployment writes under it). The silo provisioner itself is not wired into the served binary: its planes, the object-prefix one included, have only in-memory implementations used by tests, so moving a tenant between isolation models neither creates nor clears anything in the store, and objects keep the full name they were written under.XREC reconciliation rounds are configurable (AUD-1, C4): a
reconcileblock in the config file supplies the schedules the rounds worker needs. Before this,roundScheduleswas declared and never assigned, so the worker — registered, licensed, and visible in the runtime roster — hit itslen(Schedules)==0guard on the first tick and blocked for the life of the process. Zero anti-entropy rounds ever ran, noxrec.witness.recordedevent was ever appended, andGET /api/v1/reconcile/agreementanswered "0 open witnesses" forever. A licensed operator watching a healthy worker would reasonably conclude reconciliation was running while nothing was ever compared. A schedule naming FEWER THAN TWO authorities is dropped rather than scheduled: comparing an authority to itself produces no witness and would make the report claimcollecting=trueon a deployment that still compares nothing — the same illusion in a new place. An unparseable cadence or liveness takes a sane default rather than zero, because a zero cadence busy-loops the scheduler against a customer's authorities and a zero liveness marks every authority instantly stale. With C4's adapters, two authorities are DURABLE:trstctl-selfreads the certificate inventory andtrstctl-careads the internal CA issuance ledger — independently written state under RLS — both projected onto the shared assertion vocabulary (issuer+serial identity, active-vs-revoked standing, scoped to the internal-CA jurisdiction so a DigiCert certificate in inventory is not reported as drift the ledger never claimed to know about). A round whose digests disagree hands each differing pair to the runtime's witness emitter: witness built over the exact sets the digests committed to, signed in the isolated signer, recorded with both signed digests on the ledger event, quarantine admission updated, and the rounds worker now survives a failed round (logged, retried next tick) instead of dying on the first transient error. The claim-1 acceptance is demonstrated end-to-end on the production assembly ininternal/reconcile/conformance/storebacked_e2e_test.go: seeded divergence, a real scheduled round, a witness naming exactly the differing subset, offline verification from the recorded event alone, and remediation authorized only after in-signer plan verification. Boundaries, stated exactly: a Vault or OpenBao authority is observed only when its schedule entry carries asourceblock (address, token file, KV/PKI/Transit mounts); it asserts KV metadata, PKI-issued certificates and Transit key metadata, never secret data, and pairs meaningfully only with an authority observing the same universe. A KMIP server is observed the same way when its entry carries akmipsource block: KMIP 1.4 over mutual TLS, reading object metadata and certificate values and never key values, proven against an in-process double and against a PyKMIP server (run withTRSTCTL_PYKMIP_PROOF=1); a pre-active KMIP object reduces to an unknown status. An EJBCA or step-ca CA joins with anejbcaorstepcasource block, and aninventorysource block makes an authority of the certificate inventory scoped to the same CAs. A round then compares a CA's issuance ledger with what the inventory holds. Both sides read only the CAs inissuer_certificates_fileand only certificates unexpired at the observation, and assert issuer and serial (read from each certificate's DER) and whether it is revoked. EJBCA is read with its REST v2 certificate search. The search is frozen at the observation time, and its pages must add up to the count read first; otherwise the observation fails rather than report part of the ledger. step-ca's PostgreSQL tablesx509_certsandrevoked_x509_certsare read in one read-only, repeatable-read transaction; its file and MySQL backends are not read. Proven in containers against EJBCA 9.6.3 Community and step-ca 0.30.2 (run withTRSTCTL_XREC_CA_PROOF=1). A round per CA witnessed each kind of drift: a revocation only the CA knew, a certificate the CA issued that the inventory lacked, and one the inventory held that the CA never recorded. EJBCA issued that one while its certificate storage was off; for step-ca it was signed offline with the intermediate key. AD CS ledgers need the Windows image and an x86 machine, and commercial CAs need accounts; those come next. The cloud-kms reducer still has a nil source — scheduling it fails observation closed and logs, it does not fabricate an authority — and inventory rows with no captured DER cannot participate in the comparison (no issuer bytes to derive the shared identity from) until their bytes are recovered.POST /api/v1/certificates/{id}/der-recovery(CLIcertificates recover-der, the certificate detail's Certificate bytes actions) looks for them in a mint of the tenant's own CAs that retained the certificate, found by the row's serial (ca_ledger), or reads the certificate the row'shost:portlocation serves from the control plane under the usual egress rules (endpoint; a private address needsegress:privateand its CIDR). Bytes are attached only when they hash to the row's fingerprint, only to a row that holds none, and with where they came from (certificate.der.recovered); the row then joins the comparison. An address only a relay reaches gets its bytes from a sweep withagent_channel.probe_scanned_certificateson, and fetching by serial from external CAs is what comes next.CA-key retirement (H4):
GET /api/v1/ca/keys/{id}/retirementandtrstctl ca keys retirementlist the dependents standing between a CA key and destruction. An operator withkeys:writecan submit the exact final dependency epoch throughPOST /api/v1/ca/keys/{id}/retirement, the explicit-confirmation lifecycle control, ortrstctl ca keys retire <id> --force -f <request.json>. The wire request contains onlyfinal_epochandconfirm_irreversible: the server fixes the policy class toca-signing-keyand derives the sole approval identity from the authenticated principal, so a caller cannot name a weaker class or impersonate another approver. The request only appends a frozen evidence command and its same-transaction outbox intent. The outbox worker replays that frozen event prefix and callsGatedDestroyover the signer UDS; a frozen public event segment that exceeds the signer's 1 MiB unary receive limit uses 128 KiB streamed frames. The signer keeps that segment in an unlinked, local spool with a 512 MiB total cap, hashes its exact bytes into the durable operation identity, and folds only the target key's dependency state. The API process never receives a private key or a key-backend destruction handle. Incomplete evidence produces a signed refusal and leaves the key alive. Complete evidence destroys the signer-local handle first, then mints one full public record bound to the command ID, tenant, stable key ID, final epoch, required/completion/revocation digests, quorum and audit head. A signed refusal is fsynced under the exact frozen command hash before the signer replies, so a lost response or restart returns the same refusal bytes without another signature. Both outcomes are immutable events and rebuild the tenant-RLS projection after restart. The console displays signed refusals and downloads the complete destruction record for offline verification. VDEC attaches in every build and remains fail-closed on missing evidence. The checklist now derives its CA-key dependents from the original immutable managed-leaf, PKI-secret, child-CA, re-key and public cross-sign events. It does not depend on a later best-effort registration append after signing. The indexed tenant read model is rebuilt from those same events on restart; the signer's frozen event-set check uses the same derivation, including issuance after the frozen cut. A serial-only historical issuance remains a conservativeserial:dependent rather than an invented certificate inventory row. For a host-held certificate, the effect-freePOST /api/v1/ca/keys/{id}/reprotect/plans/previewand CLI preview derive the exact public plan without recording it.POST /api/v1/ca/keys/{id}/reprotect/plansandtrstctl-cli ca keys reprotect planrecheck current state and record an immutable public plan only after checking the tenant's outstanding dependent, the exact certificate and its predecessor CA signature, one managed identity bound to that certificate and target, an active direct successor CA, and the enabled host target's current revision and assigned agent. Ambiguous identity bindings refuse. The target supplies the probe address and name; callers cannot redirect them in the plan request.GET /api/v1/ca/keys/{id}/reprotect/plans/{job_id}and the CLI plan readback recover that immutable public selection after a lost response. A plan binds the selected certificate-policy name. Preview and every later execution stage resolve its current active revision, using the configured default only when the identity has no selected or retained policy. Policy narrowing refuses before queue or signature; a legacy plan without a recorded policy name is recovered under the identity's current selection, not an unrelated default. The plan receipt proves that review was recorded, not that issuance, deployment or served-state cutover happened. The host worker dispatches the bound job to its assigned agent. The agent uses the installed subject key to make one encrypted, restart-recoverable CSR, pins the exact predecessor, installs the successor, probes the served leaf, and reports signed custody. Only that verified report can atomically supersede the old inventory certificate and change the identity's selected CA. Its source event carries the signed readback, and separate first-writer supersession and completion events account for the old CA dependent. This is an administrative continuation of the same deployed subject key and identifiers, not a new API enrollment. The identity-selected certificate profile still checks the CSR's names, key algorithm, EKU and requested lifetime. Itsallowed_protocolslist governs new enrollment and does not exclude this CA re-protection path; the profile audit labels the decisionca-reprotect. Ordinary API enrollment remains subject toallowed_protocols. The host agent cannot ask for a new name or key through this exception because the control plane rechecks the immutable plan and exact predecessor SPKI before signing. The CA hierarchy Lifecycle console now exposes the host-held certificate, direct successor, enabled target and lifetime selection. Its effect-free preview shows the server-derived agent, key, target revision and listener; it records the reviewed plan, independently reads it back, and can recover a lost response by job ID before queueing. The console never calls a queued job complete. The path is qualified on the evaluation stack against a live host target: a host-executed leaf under a predecessor intermediate was re-issued under its rotated successor by the enrolled agent with the same subject key, read back by a stock TLS client, accounted on the retirement checklist, and the predecessor key was then destroyed with a signed destruction record that the API, CLI and console all read back after a control-plane restart and a cold restart. A host-executed first issuance also resolves the identity's ownership readiness at the deploy call: an unattested owner is refused with409before any key, signature or host job exists, and a refusal that still happens after the host finished is recorded on the identity's delivery timeline aslifecycle_transition_refusedwith its cause. Retirement stays blocked until its completion evidence is present. An expired requester-held managed leaf with no managed deployment can be dispositioned throughrelease_managed_ca_leafafter two distinct custodians approve its exact public certificate digest, immutable mint event and bounded external-use risk decision. The operator previews and starts the ceremony in the API or console, then callsPOST /api/v1/ca/keys/{id}/dependencies/releaseortrstctl-cli ca keys dependent release. A durable first-writer fence survives restart and backup; its event and approval evidence rebuild the checklist. This deliberately does not claim that expiry proves the leaf was unused outside trstctl, and serial-only historical rows are not eligible. A revoked, unserved host-held leaf can use a separate bounded reason only when the exact local-issuer revocation fence is completed and the immutable revocation event matches its mint and public certificate. The same two-custodian, first-writer release flow records this risk decision. A local revocation record cannot prove that every external relying party rejected the leaf; a deployed or observed host leaf remains ineligible. If the tenant has no recorded dependency state for the requested key, the checklist returns HTTP 404 with an explanation to verify the key ID and collect dependency evidence. This does not establish whether the key itself exists; another tenant's dependency state is never disclosed. A recorded history whose dependents are all accounted for still returns HTTP 200 with its actual counts. A missing dependency report is not an empty outstanding list: an empty list reads as "this key has no dependents", which must never authorize an irreversible act on evidence nobody gathered. Served on the CA hierarchy console's lifecycle tab, where retirement belongs beside rotation. Its selector is populated only from served, signer-backed CA authorities already markedsupersededorrevoked; it sends that authority's exact ID, while the API repeats the same precondition check. A failed checklist read renders an explicit unavailable state and no destruction control, never a reassuring zero. A failedvdec.retirement.destroyoutbox row can be recovered throughPOST /api/v1/ca/keys/{id}/retirement/retry,trstctl-cli ca keys retirement retry, or the lifecycle console. The checklist includes the frozen command ID, delivery state and attempt count. The operator must bind a reason and explicit irreversible-action confirmation to the observed command ID and attempt count. The event grants exactly one more outbox attempt on the original row; it cannot mint a replacement command or change the frozen dependency evidence. A repeated idempotency key recovers the same grant after a lost response. The isolated signer returns its durable signed result for the same command if it completed before the response was lost. A changed dependent, command, payload, actor, tenant, or attempt count refuses the grant; a cold replay cannot refund an already consumed attempt.Incident fleet-reissue batch gates (H3): each batch's health gate is recomputed on read from THAT BATCH's own replacement identities, through the same verification summary D6's canary uses. It previously round-robined the run's gate list across batches (
gates[(index-1)%len(gates)]), so a batch's gate described some other batch — worse than showing nothing, because during an incident it reads as per-batch evidence and the operator deciding whether to continue is exactly who would act on it. The verdict rules are unchanged and deliberately refuse to round up: any failed replacement fails the gate, and a single UNVERIFIED replacement keeps itnot_evaluatedrather than passed — a run that is 99% verified is a run with an endpoint nobody looked at. Only receipts paired with an accepted signed agent-job receipt count. Batches that are planned, queued, or halted keepnot_evaluated, because a batch that never ran has nothing to verify and a halted one changed nothing; a read failure leaves the gate as it stands rather than inventing a verdict in either direction.Migration waves (H2):
internal/migrationis the generic ordered-cohort engine behind served CA rollover.POST /api/v1/migrations/assessremains a separate, read-only operation and reports UNKNOWNS separately from findings: a member with no observed trust store is not a member confirmed to have an empty one. A reviewed manifest then names one active signer-backed CA authority, ordered waves, exact deployed identities, exact enrolled host agents, and exact public trust-anchor paths. The server derives the public anchor from that authority; it never accepts an operator-supplied key or silently changes authorities after review.Execution is event-sourced and restart-safe. Each wave runs distribute public trust → signed readback → issue from the manifest authority against a host-generated CSR → deploy → signed live-listener handshake → advance. Both observations are gates, not progress labels: one missing receipt licenses no next action, one failed receipt stops forward publication, and later waves never enter the outbox. The failure automatically begins the newest-first inverse, including every member effect already published for the failed gate; signed receipts that arrive late are retained without restarting forward motion and their serialized inverse still runs. Pause stops new work from being published and recalls the work this run has already published: a job not yet claimed stops being claimable, a job an agent holds refuses its next lease extension and its result (the agent stops before it installs), and a signed result that still arrives is retained as a late receipt rather than applied. Resume lifts the recall and drops the cut lease, so the exact required agent claims the job afresh and the idempotent intent converges; resume continues the same gate. Rollback restores and reverifies predecessor leaves newest wave first, projects the predecessor active and successor superseded in the same event transaction, then removes successor trust. A rollback failure halts on a durable attempt cursor and a manual retry gets fresh idempotent outbox identities. The exact-agent jobs, immutable aggregate, outbox effects, receipt evidence, and rollback cursor all survive PostgreSQL projection loss and event replay.
Served through
/api/v1/migrations/runsplus/{id}/pause,/{id}/resume, and/{id}/rollback, the matchingtrstctl migrationscommands, SDKs, and the/migrationconsole. Scope: executable membership is deliberately narrower than assessment today — one active predecessor per identity that carries any mix of DNS names, IP addresses, rfc822Names and URIs, an enabled host-agent connector withverify_address, and an active host-role agent. The member binding freezes each kind of identifier on its own half (subject_dns_names,subject_ip_addresses,subject_email_addresses,subject_uris) exactly as the predecessor spelled it; the host-generated successor CSR asserts exactly those identifiers and the control plane refuses any other name, address, email or URI by name, so a client that connects by IP, matches an rfc822Name or pins a SPIFFE URI keeps working after the migration. The same binding shape serves host-generated renewal and CA re-protection, so such leaves renew and re-protect on the host path too. The predecessor is the leaf the identity's last completed deploy, verified host renewal or rollback receipt names on its target, whichever authority issued it: a leaf a public or partner CA issued through the external registry migrates onto the new private authority exactly like a locally issued one, the member binding records its inventory source and the issuing external CA id (predecessor_source,predecessor_external_ca_id), and the successor supersedes it in inventory. An identity with no such receipt falls back to its one active internally issued leaf; a receipt naming a leaf the inventory does not hold, or one no longer active, is refused at start by name. An incident run still revokes through a served internal authority, so its cohorts are the compromised internal CA's leaves, never externally issued ones. A network appliance (an f5, kemp, netscaler, a10, fortigate or paloalto target the connector catalog marksexecutes_rollback) joins a run as an appliance member: the operator names the network relay the target is assigned to (required_agent_id), and the member binding recordsexecution_model: appliance. An appliance has no trust stage of its own — a relay cannot write a trust store on an appliance and the appliance has none this control plane manages — so its trust verdict readsnot_applicablerather than verified, it takes notrust_anchor_path, and the hosts whose clients must trust the new anchor are members of their own; a cohort of only appliances opens at the leaf gate. The control plane then issues the successor under the new authority with the predecessor's exact identifiers, records it as the predecessor's successor, and queues the sealed deploy for that relay alone; the relay installs it and its signed post-deploy handshake is the live gate. Rollback re-binds the predecessor object through the same relay (the A46 re-bind families) and removes no anchor. An appliance family without an executable re-bind, a host agent named for an appliance, or a target not yet assigned to the relay is refused at start by name. Built and proven against appliance doubles; proof on a real appliance needs the operator's device. Cloud-store migrations are refused at start rather than represented as executable. An imported offline root may be named as the new authority: its key never enters the signer, so successors are issued by the root's one active signer-backed intermediate (imported through the offline-intermediate ceremony) while the root certificate is the anchor hosts receive; the member binding records that intermediate as the issuing authority. An offline root with no such intermediate is refused at start with the ceremony to run, and one with several is refused until the operator names the issuing intermediate itself, which then makes that intermediate the distributed anchor. A host-generated successor that fails its live gate keeps its sealed key on the host for the same job to resume; rolling that successor back now frees the slot, so the next host-generated job for the target generates its own key. Members whose certificate profile requires a distinct issuance approval (or a deployment withrequire_approval) no longer refuse the run: the start is the approved operation. The firstPOST /api/v1/migrations/runsrecords an immutable approval request (resource_kind: migration_run, the plan id, the requester, the canonical request digest, the new authority and each gated member's frozen profile revision and lifetime) and answers202withstatus: pending_approval, the request id and intent digest; reviewers withcerts:issueapprove it through/api/v1/approval-requests, and the same start posted again under a newIdempotency-Keyrecords the run with the consumed approval on it (approvalin the run). Incident fleet reissuance still refuses such members; it has no start approval of its own yet.Trust in the graph (H1): trust-store anchors agents collect are promoted from flat discovery findings into relationships — a
trust-storenode kind,TRUSTS(store → issuer, oriented the way impact travels) andHOSTS(resource → store).GET /api/v1/graph/trust-stores/{id},trstctl graph trust-stores, and the Graph console's issuer detail answer "trusted by N stores across M hosts"; hosts are counted distinctly, because one machine running both an OS store and a JVM cacerts is two stores and one machine to visit. A store is its own node rather than a host attribute on purpose: a machine routinely carries several with DIFFERENT contents, so "is this CA trusted on host X" has no single answer. Scope, stated exactly: an authoritative store → issuer edge requires either the same certificate SHA-256 fingerprint or the same SPKI SHA-256 public-key identity. The SPKI path deliberately recognizes a cross-signed copy of the same CA key. A subject-name-only match is exposed separately as an unverified candidate and is excluded from authoritative store/host counts, blast-radius traversal, and automation; two roots can share a subject while holding different keys. Stores nobody has scanned do not appear, which is the honest answer rather than a reassuring one; a CA showing zero trusting stores means none have been observed, not that none exist.Third-party secret scanning: CI/CD log, container-registry, Slack, and Jira artifact scanning is served through
/api/v1/secrets/scans/third-party/{provider}/ingestand the matching CLI. The contract intentionally accepts an operator-ownedartifact_path; raw logs, registry metadata, chat exports, and issue exports stay outside trstctl storage — discovery stores only redacted rule/file/line/provider metadata. The console queues an artifact, reopens the exact discovery run by ID after a restart, shows its terminal state and redacted findings, and links to the full filtered Discovery triage. This is an operator workflow over the local export; it does not collect native provider events. Provider signature verification is served: an ingest may carrysignature_path(the provider's detached RSA or ECDSA signature over the artifact bytes, raw or base64) andsigner_public_key_pem; the signature is checked before the scanner opens the artifact, a failing signature fails the run without scanning,require_signaturerefuses unsigned artifacts at ingest, and every finding recordssignature_verifiedandsigner_key_sha256. Artifact retention is automated at both ends:delete_artifact_after_scanremoves the operator's export file once the scan is terminal (one regular file; directory exports stay), and the privacy retention worker deletes redacted third-party findings older thanTRSTCTL_PRIVACY_RETENTION_THIRD_PARTY_SCANS(default 90 days) while the run record stays. Native provider API polling and provider-native annotations remain architecture shortfalls.React console scale work: cursor-aware inventory pages consume
next_cursorand accumulate additional pages on explicit operator action. Every DataGrid-backed large table switches to a bounded, overscanned DOM window after 100 loaded rows; the certificate inventory has a focused multi-page acceptance test proving the second cursor is sent and that scrolling replaces, rather than appends, rendered row nodes.
The React web console: served by the binary
The React 18 + Vite + shadcn/ui single-page app (F12) is the real embedded artifact the running binary serves:
- The shipped binary serves the real console. The release pipeline builds the
SPA into the binary's embedded asset bundle; the built bundle is committed,
so even a plain control-plane build serves the real console at
/— hashed/assets/index-*.{js,css}and anindex.htmlthat references them, not the old "not built" placeholder. A test boots the served handler over the real embedded bundle and fails if it ever regresses to the placeholder, and a release gate (TRSTCTL_REQUIRE_BUILT_UI=1) blocks a release that would embed the placeholder. - Generated frontend↔backend contract. The frontend's API types are generated from the served OpenAPI contract, not hand-duplicated: the generator emits TypeScript types from the served API spec, and the API client re-exports those types so a backend field add/rename/remove that isn't regenerated fails the TypeScript build. A CI regenerate-and-diff gate fails the build on drift, so a frontend/backend status mismatch can no longer recur silently.
- Operational console routes. First-class routes, nav entries, typed API
wrappers, and route-test coverage cover the GA operator slice: Profiles
(
/profiles), Graph (/graph, inventory + blast-radius query), Audit (/audit, event list + evidence export in JWS/NDJSON/CSV/Splunk-HEC/Sentinel), dual-control approvals from the identity table, core incident execution (/incidents— replacement issue/deploy, fleet reissuance, revocation queue, connector receipt, rollback evidence, remediation playbooks, response dispatch, sealed audit bundle), and Product help (/assistant) with grounded questions, cause investigation, and on-demand read-only MCP tools. The bounded break-glass reconciliation workflow has its own panel on the Incidents page. Deliberately API-only surfaces stay labeled until they get their own UI — a UI boundary, not a claim that the underlying API is library-only. - Console UX hardening. A destructive-transition confirmation (revoke/retire
require an explicit, credential-named confirm dialog) and
429/
Retry-Afterhandling (a concrete "retry in Ns" hint) are served and tested. Cursor-based pagination and bounded list virtualization are also served: the client carriesnext_cursor, accumulates rows, and the shared DataGrid renders only the visible window plus overscan for large result sets.
Interactive OIDC, SAML, and LDAP / Active Directory browser login & sessions: served by the binary
The OIDC authorization-code login, SAML 2.0 Service Provider login, and LDAP /
Active Directory bind login + sessions are served by the running binary
(behind auth.oidc.enabled, auth.saml.enabled, and auth.ldap.enabled).
OIDC remains in the Free core; tenant SAML and LDAP require Enterprise SSO,
included in Enterprise and inherited by Provider. An enabled tenant SAML, LDAP
or SCIM configuration without that implementation refuses startup with
SAML, LDAP and SCIM login require an Enterprise licence. Core-only builds
cannot attach it. The separate Provider operator authentication plane is unchanged.
OIDC mounts /auth/login and /auth/callback; SAML mounts
/auth/saml/login, /auth/saml/acs, and /auth/saml/metadata; LDAP mounts
POST /auth/ldap/login; all three share /auth/me and /auth/logout. OIDC
verifies the id_token's signature, issuer, audience, nonce, and temporal
claims (exp/nbf/iat). SAML verifies signed POST-binding assertions against
configured IdP metadata through the same isolated cryptography boundary. LDAP
binds the user, then performs a configured group search; production
directories should use ldaps:// — plaintext ldap:// is accepted only for
loopback development fixtures. All paths set an HttpOnly +
SameSite=Strict session cookie (marked Secure whenever the control plane
serves TLS) plus a double-submit CSRF token, authorizing API calls under the
same RBAC and per-tenant database-isolation scoping as an API token;
mutations on the cookie path require the CSRF header. When browser sign-on is
disabled the binary authenticates with scoped API tokens only, exactly as
before; an enabled-but-misconfigured OIDC, SAML, or LDAP block fails closed
at startup.
- Per-user → tenant mapping is served. Each authenticated user is mapped to
its real tenant at session issue — by a configurable OIDC claim or SAML
attribute (
auth.oidc.tenant_claim/auth.saml.tenant_claim), by an IdP or LDAP group → tenant table, or by an explicit subject/claim/group → tenant mapping (auth.*.tenant_mappings) — instead of collapsing every browser user to one tenant. A user that maps to no tenant is rejected (the login fails closed, never minting a session in a fallback tenant unless an operator explicitly opts intoallow_default_tenant). Per-tenant database isolation then confines each session to its mapped tenant, so two SSO users in different tenants see only their own data via the served API. The legacy single default tenant is retained only as that opt-in fallback. This is the served half of the defense against cross-tenant leakage; a freshly logged-in user still cannot self-issue (issuance stays behind the served RA/policy gate and the requester scope excludescerts:issue).
SCIM 2.0 provisioning: served by the binary
The SCIM 2.0 provisioning surface requires Enterprise SSO and is served by the
running binary behind auth.scim.enabled. It mounts GET /scim/v2/ServiceProviderConfig,
/scim/v2/Users, and /scim/v2/Groups; bearer tokens are loaded from
configured token files, hashed, and bound to one tenant before a request body
is trusted. SCIM user create/update/PATCH writes the same tenant-member event
path used by RBAC. SCIM active:false, DELETE, or group removal changes the
projected tenant-member roles, so the next browser-session API request sees
the new authorization result.
SCIM Bulk (RFC 7644 §3.7) is served at POST /scim/v2/Bulk (A72) and
advertised in ServiceProviderConfig (maxOperations 100, maxPayloadSize
1 MiB): the operations run in order through the same per-operation logic as
the single routes, a POST's bulkId names its created user or group for the
operations after it (in a path or in any string value of the data), each
operation reports its own status and body, failOnErrors stops the request
after that many failures, and the whole request is one idempotent mutation —
a replay under the same Idempotency-Key returns the recorded response.
Current SCIM limits are deliberate and fail closed: password management and
password-change flows are not implemented; SCIM groups never create roles. A
group grants a role only when its name is a configured RBAC role (such as
admin, operator, viewer, auditor, or ra-officer) or the token's
group_roles maps it to a configured RBAC role or a custom role the tenant
defined; any other group is refused and grants nothing. Inbound token rotation is rewriting the token file: the new
hash loads within two seconds without a restart, and a rewrite that cannot be
used keeps the last good token. The writeback bearer is still read only at
startup.
Directory writeback (A73) is served when a token also names the directory's
own SCIM 2.0 endpoint (auth.scim.tokens[].writeback_endpoint and
writeback_token_file; the bearer stays in locked memory). A change an
operator makes in trstctl to a member that token provisioned — a role grant
or removal through PUT /api/v1/access/members/{subject}, or an offboard
through POST /api/v1/access/members/{subject}/offboard — queues a
scim.writeback outbox row in the same transaction as the member event. The
worker finds the directory's User by the same attribute the token binds the
login subject to (userName, or externalId when mapped), PATCHes active
and the trstctl roles as the User's roles, reads the User back, and records
scim.writeback.recorded with the verdict; a directory answer that disagrees
with what was written is recorded as refused, a transport failure retries
through the outbox, and the directory's own words never enter the event log.
A change the directory itself pushed is never echoed back, and a member with
no SCIM identity is never written. The directory must serve SCIM 2.0 for its
own users (an identity broker, a SCIM-fronted directory, or an IdP whose SCIM
API accepts writes); an IdP that only pushes outbound SCIM cannot be written
to, and without a writeback endpoint local changes to directory-owned members
still stand until the directory's next push, as before.
Unified NHI inventory
Unified NHI inventory (CAP-NHI-02) is served as metadata, not credential
material. GET /api/v1/nhi/inventory requires nhi:read and merges
tenant-scoped identities, certificate inventory, API-token metadata, enrolled
agents, and discovery findings into one normalized inventory across
certificates, SSH keys, secrets, API keys, OAuth apps, tokens/PATs, service
accounts, IAM roles, webhooks, workload IDs, and agents. It does not return
secret values, private keys, raw API tokens, client secrets, or other
credential bytes; use the specific governed secret/issuance endpoints for
those flows.
Malicious / abused OAuth-grant detection (CAP-ITDR-03) is served from metadata
exports. oauth_grant Discovery sources emit normal oauth_grant inventory
findings and, when the export contains concrete abuse evidence, additional
oauth_grant_abuse findings tagged CAP-ITDR-03. The detector stores
provider threat signals, reason codes, evidence refs, and source event ids
only; OAuth client secrets, access tokens, and refresh tokens are rejected.
Live IdP/SaaS grant revocation and provider-side enforcement remain
connector/remediation work rather than being hidden inside discovery.
AI, RCA, and MCP surface
The AI surface — model adapter (F76), grounded RCA / NL query (F75/F77), and
the guarded MCP tool server (F78) — is served, mounted under /api/v1/ai/*
and /api/v1/mcp/* (off by default — ai.enable_api — and fail-closed when
off, so an upgrade does not silently expose it):
POST /api/v1/ai/queryreads typed, tenant-scoped evidence from owners, certificates, the graph, CBOM, and the event log (F75). A question planner reads certificate questions that combine expiry windows, expired, status, issuer, key algorithm, subject or SAN name, owner and no-owner clauses (as a list or an exact count), “who owns” a certificate, crypto-asset questions by algorithm, key size, quantum vulnerability or policy, and credential-graph inventories. Each becomes a typed query whose values are bound parameters, and the answer states its interpretation with dated citations. The planner is a fixed grammar, not a language model: a question outside it, such as one about event history or causes, returns scoped evidence marked insufficient, not a claimed complete answer. A bounded result also declares when more matches may exist.POST /api/v1/ai/rcaanswers a grounded root-cause / NL question from cited real records gathered through the tenant-then-RBAC scoping seam, preferring "insufficient evidence" to a guess (F77). A cause question needs a cited failure reason; event types alone are marked insufficient. The allowlisted renewal-failure reason is redacted before it enters the answer.GET /api/v1/mcp/tools+POST /api/v1/mcp/tools/{tool}expose the tenant-scoped MCP tools an external AI agent can list and invoke (F78). Investigation tools are read-only by default; guarded write tools (issue_certificate,rotate_certificate) appear only whenTRSTCTL_AI_MCP_WRITE_TOOLS=true, each still requiringcerts:issue, anIdempotency-Key, and emittingmcp.tool.write. They act on a lifecycle identity through the identity transition route, so the policy gate and dual control apply: under dual control an issuance is proposed (mcp.tool.proposed) and runs only when repeated after approval, and a rotation is refused. MCP has no route to direct CA issuance or to an approval decision. Evidence tools (posture_findings,weak_key_exposure,distrust_exposure,assurance_record,revocation_status,crypto_inventory) read the served routes with the caller's token, and every tool result carries findings under the published result schema. Exposure covers what discovery and the inventory have seen: hosts and trust stores no agent has scanned are not covered, and a certificate known only by its issuer name is matched by name (confidenceinferred). A result lists at most 200 findings. The rated certificate inventory behindweak_key_exposureanddistrust_exposurereads at most 500 certificates and 500 discovery findings; the rogue-certificate posture reportsread_limitedwhen it reached that, and the tool answer then says it may be incomplete.
Every route is auth-gated (API token or session, graph:read), tenant-scoped
(the tenant is the authenticated principal's, never a request field), rate
limited, and injection-inert (a hostile string in a record is inert, cited
data, and cannot by itself trigger a write). The AI model is air-gapped /
opt-in by default — no model is configured, so grounding and citations work
and nothing phones home; when an operator opts into a cloud/local model,
every prompt crosses a redactor plus a residual-entropy refuse-gate before
any egress, so no key/secret material leaves to a model (secret bytes live
only in wipeable, zeroed memory and never reach a prompt). Proven end-to-end
by acceptance tests (grounded NL-query/RCA citing real records, cross-tenant
denial, injection-inert + secret-redacted, MCP list+invoke).
Secrets and identity frameworks
Transit keys are sealed at rest and shared by every replica. Each tenant's transit (encryption-as-a-service) keyring is one sealed row in PostgreSQL: AEAD, HMAC, and signing key material is sealed under the tenant's key domain (the deployment KEK for a tenant that has not migrated), and re-locked into protected memory on load (AN-8). Moving the tenant into its own key domain, or rotating that domain, re-wraps the row with the tenant's other secrets; only its data key changes. A create or rotate is saved before it is reported, over the revision the replica last read, so two replicas changing one tenant's keys at once cannot overwrite each other: the replica that loses the save catches up and applies its change again. A failed save fails the mutation rather than handing back a key that disappears later: the unsaved key (or new version) is rolled back out of the in-memory ring and its material wiped, so nothing can encrypt under it and a retry succeeds. A replica loads a tenant's row the first time the tenant is used and reloads it when an operation names a key or version it does not hold, so a key created or rotated on one replica works on every other.
Three properties bound what this gives you:
- Persistence requires key protection. With no deployment KEK the keyring stays in memory only and keys do not survive a restart. Key material is sealed at rest or it is not written at all; there is no plaintext fallback.
transit.keyring_diris now an import source. A keyring an earlier version checkpointed there is copied into the shared keyring at startup for each tenant that has no row yet; the file is left as the operator's copy, and a file that cannot be opened still refuses the start.- A keyring that cannot be opened fails the operation. A row the tenant's key domain cannot open (a sealed tenant, a lost KEK) fails each transit operation for that tenant instead of minting fresh keys over it. The key bytes are deliberately not event-sourced, so the table restores from the PostgreSQL backup.
- Create and rotate are event-sourced facts (A126). Each is a
transit.key.createdortransit.key.rotatedevent (schema 2: key, kind, version) under a fact ID saved with that version's sealed material, appended only after the material is saved, and projected intotransit_key_facts, which rebuilds from the log.GET /api/v1/transit/keys/{name}/versionsreports each version'sfact_idand whether the log records it. A fact whose append fails leaves the key usable and is recorded by the leader's reconcile pass (at start and every ten minutes), which also gives fact IDs to versions saved before facts existed; until then the version readsrecorded: false. Facts recorded before this change are schema-1 audit records that name only the key and project nothing.
Private-egress allowlist entries must be exact network prefixes. Every
allow_private_cidrs / private_egress_cidrs surface (connectors, Rekor
code-signing, incident notifications, managed keys, secret integrations,
external CAs, ServiceNow bindings) now refuses an entry whose host bits are
set (for example 10.1.2.3/8) or that covers every address (0.0.0.0/0,
::/0) at configuration load, naming the offending value. Such entries used
to pass validation on most surfaces and were then silently ignored at dial
time, so private egress failed as SSRF-blocked with nothing explaining why.
Migration: operators whose entries currently carry host bits should write
the network address the entry already effectively meant (for 10.1.2.3/8,
write 10.0.0.0/8 — or, if a single host was intended, 10.1.2.3/32). The
dial-time guard remains as defense in depth and now logs and counts any entry
it skips.
CMP binds every CSR to the authenticated protection identity by default.
A PKIMessage's protection identity must chain to the operator-configured
anchors (protocols.cmp_client_trust_anchor_file), and — new — the CSR's
subject and SANs must be asserted by that identity, so a device credential can
renew only itself and a stolen credential stays one device in blast radius.
RFC 4210 registration-authority enrollment on behalf of third parties remains
available as an explicit opt-in: protocols.cmp_allow_ra_enrollment in the
config file or TRSTCTL_PROTOCOLS_CMP_ALLOW_RA_ENROLLMENT=true in the
environment (echoed by -check-config as
protocols.cmp.allow_ra_enrollment when CMP is enabled). Cross-identity
requests under the default are refused with a distinct audit reason
(csr not bound to protection identity), so an operator can tell an
impersonation attempt from a malformed message. Deployments that relied on the
previous unbound behavior must set the RA opt-in deliberately on upgrade.
Six of six secrets/identity frameworks are mounted on the running binary
under /api/v1/secrets/* (off by default — secrets.enable_api — fail-closed
when off, requiring a KEK when on):
- Auth-method framework (F58) backs
POST /api/v1/secrets/login— a machine presents a token, Kubernetes SAT, AWS IAM signedGetCallerIdentityrequest, GCP identity JWT, Azure workload JWT, generic OIDC token, or generic JWT and receives a scoped, tenant-scoped session (distinct from the human OIDC SSO bridge). Token credentials MAC-bind tenant, audience, principal, and expiry; JWT methods require a tenant claim or tenant-pinned config; AWS IAM is tenant-pinned through allowed account/ARN config.X-Tenant-IDis only a lookup hint — mismatched tenant headers are rejected. - Application secrets SDK (F64) backs the secret store
POST/GET/PUT/DELETE /api/v1/secrets/store/...(create, read, rotate, delete),GET /api/v1/secrets/store/{name}?resolve=true(${secret.path}reference expansion with cycle rejection),GET /api/v1/secrets/store/history/{name}?version=N(read one prior sealed version), andPOST /api/v1/secrets/store/recover/{name}(point-in-time recovery). Values are sealed at rest under the KEK.trstctl-cli run --secret ENV=path -- <cmd>wraps the same read path, injecting values only into the child process environment. Bulk import uses one event, per-name reservations, and an atomic sealed-row/receipt projection for 1–100 new names. It is served through API, CLI and console; existing or reserved names reject the whole batch. - The Vault/OpenBao compatibility shim backs the common migration paths
GET /v1/auth/token/lookup-self, KV mount-discovery preflight forsecret/,POST/PUT/GET /v1/secret/data/{path}, andPOST/PUT /v1/pki/sign/{role}for requester-generated CSRs, and the deprecatedPOST/PUT /v1/pki/issue/{role}server-keygen path for stockvaultCLI token lookup, KV v2 put/get, mount and ACL policy lifecycle, and PKI issue — deliberately a subset over the native secret store and dynamic PKI secret. Tenant-authored mount management, ACL policies, and common transit paths are also served, as are response wrapping (/v1/sys/wrapping/wrap|unwrap|lookupover the native one-time share rows, single-usetrstw_tokens with a bounded TTL) and the per-token cubbyhole (/v1/cubbyhole/{path}, sealed in the tenant key domain, keyed by the presented token, expiring with it). The native dynamic leases are served at Vault's engine paths:database/creds/<role>(username and password from a PostgreSQL, MySQL, MongoDB or Redis provider),aws/creds/<role>andaws/sts/<role>(an STS session),kubernetes/creds/<role>(a ServiceAccount token), or<provider>/creds/<role>when several providers offer the role, withsys/leases/lookup,renewandrevoke. Each read opens a native lease (an explicitIdempotency-Keyreplays it). Database leases renew up to the provider's maximum TTL; AWS STS and Kubernetes leases are not renewable on these paths because their native renewal mints a replacement credential. A synchronous revoke returns once the provider confirms removal; a Kubernetes API server can still accept a token it authenticated recently for its authentication cache TTL (10 seconds by default) after its ServiceAccount is gone. AWS cannot end one STS session early, so revoking an STS lease closes the lease and reports the session valid until its native expiry. Vault's engine configuration paths (database/config,database/roles,aws/roles,kubernetes/roles), static database credentials,revoke-prefixand lease listing are not served: providers stay operator-configured. The stock Vault CLI's KV put and PKI issue/sign commands derive a token-keyed idempotency key when no header is sent. Mount and ACL mutations also derive a token-keyed key with the exact mount or policy event generation, allowing stock CLI retries and later delete/recreate cycles. KV v2 metadata (GET /v1/secret/metadata/{path}: retained versions and write times), folder listing (LISTorGET ?list=trueon/v1/secret/metadata/{folder}: direct children, sub-folders suffixed/,404for an empty folder), object deletion (DELETEondataormetadata, the native delete with no soft-delete state),POST /v1/sys/capabilities-self(from the token's permissions and ACL policies) andPOST /v1/auth/token/renew-self(refused: API tokens are not renewable leases) are served. A re-enabled tenant KV mount starts an empty namespace; older sealed values remain under native retention and mount disable is not a physical-erasure receipt. Transit mutations require an explicitIdempotency-Key. - Dynamic secrets (F65) are served when tenant providers are configured:
POST /api/v1/secrets/leases,GET /api/v1/secrets/leases/{lease_id},POST /api/v1/secrets/leases/{lease_id}/renew, andPOST /api/v1/secrets/leases/{lease_id}/revoke— issue returns the backend credential once, later reads return metadata only, renew extends an active lease except for immutable Kubernetes TokenRequest and AWS STS credentials (renewal issues and reveals a replacement lease, then closes the predecessor), and revoke closes it; the leaseworker expires leases through an outbox-backed backend revocation queue.buildRunDepsconstructs a tenant-bound registry forpostgresql,mysql,mongodb,aws-sts(aws-iamalias),gcp-iam,azure-entra,kubernetes, andredis. Issuance is outbox-only: the pending event and sealed command commit first, provider retries reuse one stable lease identity where the provider has a native replay/lookup path; AWS STS fences an indeterminate call instead. Only the authorized issue response opens the sealed credential. Local acceptance proof logs in with each generated PostgreSQL, MySQL, MongoDB, Redis, and Kubernetes credential, rotates it, revokes both copies, and verifies rejection afterward. AWS STS is a contract-tested adapter, not a completed cloud journey: no AWS account is available in this local-only run for stock-client authentication, CloudTrail reconciliation, or rejection after native expiry. The local query double does not qualify AWS IAM/STS behavior.aws-stsand itsaws-iamalias issue expiring AssumeRole sessions; they no longer create permanent IAM users or keys. STS cannot delete one session early. A trstctl revoke closes its lease, but AWS may accept the session untilnative_expires_at; the status remains pending until then. A lost AssumeRole reply leaves an indeterminate, fenced operation rather than risking a second session. Existingaws-iamstartup attachments must migrate their policy ARN role bindings to full assumable IAM role ARNs and set an explicit 15-minute to 12-hourmax_ttl. New Kubernetes leases record the UIDs of the ServiceAccount, bound Secret, and RoleBinding and delete with UID preconditions. Leases issued by older builds have name-only provider references; revoking one now inspects every same-name object first, leaves any object that carries the lease-owner label to the newer lease that reused the name, and deletes the unlabelled legacy objects under the UIDs seen at inspection, so generated names can be reused without draining older leases first. - Honeytokens include the native
trst_bait (F39 extension) and an operator-attached AWS IAM bait-key adapter (F80). The AWS path is a local IAM/ CloudTrail contract test until a real account runs it: the pinned LocalStack profile does not emulate IAM, CloudTrail delivery, account quotas, or cross-account event visibility faithfully. No owner AWS account was available in this local-only campaign. AWS-native use detection, live-account IAM deletion, and a stock AWS client rejection after deletion remain unverified. CloudTrailLookupEventscovers only management events in configured Regions for the configured account, with delayed delivery and a 90-day lookup limit. An outbox retry/dead-letter or an expired CloudTrail reader session is a monitoring outage; it must be investigated and requeued before asserting coverage. The AWS key is deliberately longer lived bait with explicit deny, unlike the short-lived STS credentials in ordinary dynamic-secret leases. - Secret rotation (F37) backs
POST /api/v1/secrets/rotationsfor worker-queuedconnector:<target>secret-sync handoffs and for durable static-provider commands: a configured PostgreSQL, MySQL or AWS IAM dynamic-secret provider withallow_static_rotationruns stage, cutover, verify and retire as outbox-driven phases recorded by immutable events, with rollback to the sealed predecessor on a failed verification;GET /api/v1/secrets/rotations/{id}follows a command. Boundaries: the secret name must equal the provider role; verification is a connection test for databases andnot_availablefor AWS IAM access keys; arollback_failedorretire_pendingcommand needs an operator to retire the leftover credential at the provider (no retry route yet); approval-gated tenants must approve the worker's cutover write.dynamic-lease:<provider>runs the same durable command on a configured PostgreSQL, MySQL or Redis provider: stage issues a new lease ofttl_seconds(required, at most the provider'smax_ttl) through the durable lease lifecycle, cutover publishes its credential, verify opens a database session or authenticates to Redis and runs PING, and retire revokes theold_reflease, completing only once the provider confirms the removal;old_refisnoneor an active lease of that provider for the role the secret names. Rotating a MongoDB, cloud IAM or Kubernetes lease is what comes next (503today); those leases issue, renew and revoke through the lease endpoints. A connector or static request that suppliesttl_secondsreturns400. New schedules acceptconnector:<target>and static providers; scheduled dynamic-lease rotation is what comes next (creation returns503). Historical dynamic-lease schedules, and static schedules whose provider no longer allows static rotation, are markedunsupported, disabled, and make zero provider calls. Scheduled ticks bind each exact due edge to one durable, deterministic command before effects, execute at most 50 rotations while scanning at most 500 due rows, report everyapproval_pending,command_in_flight,command_claimed, orconfig_revision_unanchoreddeferral, and continue past row-local poison. A shared store/event/custody/integrity failure returns a cached503partial envelope; the same idempotency key replays it byte-for-byte without child execution, while a new key reconciles retained terminal evidence and continues.delivery_failedadvances a connector schedule only after its canonical local version committed; genericfailednever promotesnew_ref. Verified terminal command rows are purged only after the schedule advanced, the exact event remains retained, and a newer command exists; the newest lineage fence and all claimed or ambiguous rows remain. - PKI-as-a-secret / dynamic certificate leasing (F67) backs
effect-free
POST /api/v1/secrets/pki/previewand mutatingPOST /api/v1/secrets/pki. Preview and execution share the same subject, SAN, public-key, profile, and overflow-safe TTL validator. Preview signs nothing, writes nothing, makes no external call, returns no credential material, and reports structured CA/custody/revocation/audit prerequisites plus exact effects, recovery, verification, CLI/Vault parity, and a server-keyed request fingerprint. Supplying that fingerprint on execution makes a changed plan fail closed. The recommendedcsr_pemmode signs a requester-generated PKCS#10 request and returns only the certificate. The mutually exclusivecommon_namemode still returns a usable certificate and private key for compatibility, but is deprecated and fails before key generation unless itsissuance.server_side_keygenreceipt is durable. Both modes use the issuing CA in the out-of-process signer. New issuance events project the exact public leaf, subject-key custody, and responder serial together, returning the inventorycertificate_idfor normal exact revocation. A historical version-2 PKI-secret event projected only its responder serial but retained the public leaf: after an upgrade, the first lifecycle sweep checks that leaf against the issuing CA's certificate and its serial against the CA ledger, then appends oneca.certificate.inventory_recoveredevent, which projects the inventory row a version-3 mint would have had, with subject-key custody recorded as unknown (it is never guessed). The leaf is then revoked through the normal inventory workflow. A leaf the issuing CA did not sign, or one already in inventory, is left as it is. Version-1 events retain only a serial: an operator who still holds such a leaf's public certificate submits it toPOST /api/v1/certificates/pki-secret-recoveries(trstctl certificates inventory-pki-secret -f leaf.pem), which accepts it only when the issuing CA signed it and a PKI-secret mint recorded its serial, and then gives it the same recovered inventory row. A version-1 leaf whose certificate nobody kept remains serial evidence. - Secret sharing (F60) backs effect-free, value-free
POST /api/v1/secrets/shares/preview, reviewed idempotent creation atPOST /api/v1/secrets/shares, and atomic.../redeem. The server-keyed review fingerprint rejects a changed lifetime; an exact same-key retry returns the original one-time bearer rather than creating a duplicate. Redemption succeeds exactly once, and neither value nor bearer token enters the audit/event log.
Every served route is auth-gated (API token or session, secrets:read /
secrets:write) and tenant-scoped. Every mutation is idempotent (deduplicated by
Idempotency-Key) and recorded as immutable events; effect-free previews
intentionally create neither. Secret values are held
in wipeable, zeroed memory (never as a string), never logged, and never
returned beyond their design. Proven end-to-end by acceptance tests.
Secret sync, Transit, and KMIP
Secret sync external stores (F68) — served when configured. The running
binary mounts POST /api/v1/secrets/syncs and
trstctl-cli secrets syncs run. A request reads one stored secret, writes a
sealed tenant-scoped outbox row before any external write, records immutable
sync intent, and returns metadata only. buildRunDeps constructs
tenant-bound targets for AWS Secrets Manager, GCP Secret Manager, Azure Key
Vault, GitHub Actions, GitLab CI/CD variables, Vercel project environment
variables, generic CI JSON endpoints, and Kubernetes Secrets, plus
first-class Terraform Cloud/OpenTofu Variables API and Vault KV v2 targets
(Terraform writes are workspace-scoped with category/sensitive metadata;
Vault writes use the KV v2 data/metadata paths and preserve version/CAS
semantics). Only the outbox dispatcher resolves the target credential and
writes externally. The acceptance proof enforces exact target authentication,
decrypts GitHub's X25519 sealed box in a separate receiver process, and
independently reads every destination value back. GET /api/v1/secrets/syncs/targets shows the catalog and this installation's
configured targets. A sync whose receiver result was lost is reconciled by reading
the destination back where the target can (receiver-reconcile in the catalog), and
a later version never waits forever behind it: a target without readable state holds
the lane until an operator records what they saw through POST /api/v1/secrets/syncs/{id}/reconcile; GET /api/v1/secrets/syncs/{id} names that
blocking predecessor.
AWS, GCP, and Azure targets independently opt into OIDC workload identity instead
of their static target credential. The matching
aws_workload_identity, gcp_workload_identity, or
azure_workload_identity switch is false by default, and workload-identity mode
rejects the provider's static credential fields rather than keeping a quiet fallback.
The served
/api/v1/secrets/syncs/workload-identity-sources API, CLI, and Secrets console
bind a tenant JWT/JWKS trust source, exact audience/subject, provider target, and
remote-key scope. Three thin, hand-written REST exchanges use one shared
internal/cloudauth minter: AWS calls AssumeRoleWithWebIdentity; GCP calls the
RFC 8693 STS endpoint and may then impersonate one service account; Azure calls the
Entra v2 token endpoint with the configured application client ID and Key Vault
scope. Azure's federated-credential binding accepts the already validated OIDC proof,
which the form sends unchanged in its JWT-bearer client_assertion field. trstctl
does not create another assertion, sign one with a certificate, or take custody of
an Entra certificate/private key.
Proof resolution, signature and exact-claim validation, exchange, locked
short-lived credential caching, and refresh happen only inside the bounded
secret-sync outbox worker. Air-gapped mode records offline_disabled before either
the token endpoint or target can receive a request, fails that delivery once, and
does not retry forever. The resulting GCP or Azure bearer is passed to the existing
hand-written internal/secretsync pusher; no vendor SDK or provider-specific cache
is added. For Azure this replaces the azure-key-vault sync target's static
token_ref. It does not replace
TRSTCTL_MANAGED_KEYS_AZURE_BEARER_TOKEN(_FILE): those settings and
internal/kms/azurekv belong to the separate managed-key child-signer path.
TestServedAzureFederatedOutboxTenantIsolationTokenRedaction and
TestServedAzureFederatedAirGapIsTerminalWithoutNetwork prove the served Azure
exchange, target readback, tenant isolation, token redaction, and zero-network
air-gap result.
Related read-only posture routes: GET /api/v1/secrets/cloud-secret-managers
(CAP-SEC-04 — read-only cloud_secret discovery for AWS/GCP/Azure/Vault, plus
sealed-outbox sync for AWS/GCP/Azure); GET /api/v1/secrets/kubernetes-operator
(CAP-SECR-04 — the TrstctlSecretSync controller reconciles secret references
into Kubernetes Secret.data and patches pod-template annotations for
reload); GET /api/v1/secrets/workload-injection (CAP-SECR-05 —
TrstctlSecretInjection patches Deployment/StatefulSet/DaemonSet pod
templates with the shipped trstctl-agent --secret-inject sidecar, a
memory-backed shared volume, app mounts, optional valueFrom.secretKeyRef
entries, and status/content-hash annotations; the operator reads Secret
metadata and key names but does not retain values; secret bytes never appear
in API/status/audit/pod-template metadata); and
GET /api/v1/secrets/unvaulted (CAP-SECR-07 — combines
configured repository/third-party scan sources, redacted leaked_secret
findings, AWS/GCP/Azure/Vault discovery visibility, and vault-augmentation
sync targets). A webhook target delivers templated JSON signed with HMAC-SHA256
over a timestamp and the body, so a receiver can verify it and refuse a changed or
replayed delivery; generic-ci-json stays the bearer-token-only push with no
payload signature. aws-parameter-store writes SecureString parameters (proven
against LocalStack, not a live AWS account), and netlify sets secret environment
variables through Netlify's API, proven against a double built from Netlify's API
reference; a live Netlify account proof is what comes next. Terraform/OpenTofu and
Vault no longer depend on the generic JSON target. If a target is
not configured, the route returns 503 rather than attempting an external
call.
Transit/KMIP (F66) — served, with a bounded OASIS KMIP 1.4 profile. The
running binary mounts /api/v1/transit/* and the trstctl-cli transit
command group for tenant-scoped key create/rotate, complete metadata-only
version history, effect-free sealed-restore/KMIP posture, encrypt/decrypt,
rewrap, HMAC, sign, and verify. The Secrets console exposes the same workflow,
filtered immutable receipts, and explicit no-auto-retry recovery. Transit keys
never leave the process as exportable
material, request plaintext uses wipeable []byte buffers, keyrings are
zeroized on shutdown, and mutating operations emit immutable transit.*
audit events. The binary also mounts an opt-in raw KMIP mTLS listener when
protocols.kmip.enabled is true and protocols.kmip.tenant_id,
protocols.kmip.cert_file, protocols.kmip.key_file, and
protocols.kmip.client_ca_file are configured. That first served KMIP
profile is intentionally bounded: it accepts verified client certificates,
decodes TTLV with frame-size, field-count, and nesting-depth caps, serves
Query and DiscoverVersions, AES-256 SymmetricKey Create/Register/Get
(including AES-GCM wrapped Get/Register), plus Locate/Revoke/Destroy over the
wire for stock clients, records kmip.object.created, kmip.object.revoke,
and kmip.object.destroyed, and zeroizes in-memory key material on
rekey/destroy/shutdown. It also serves RSA (2048/3072/4096) and ECDSA
(P-256/P-384) key pairs: Create Key Pair (both halves Pre-Active and linked),
Activate, single-part Sign and Signature Verify (PKCS#1 v1.5, PSS, ECDSA;
SHA-256/384/512), Get of the public key, and Get Attributes, Get Attribute
List and Add/Modify/Delete Attribute for names, contact information and x-
text attributes. A generated private key is not extractable unless the
client created the pair with Extractable set, so by default it never leaves
the server; KMIP 1.4 would default Extractable to true, and trstctl
deliberately does not. Not served: Register of asymmetric keys, certificates,
Re-key Key Pair, Encrypt/Decrypt, digested or streamed signing, Sensitive
objects, templates, and the ID Placeholder (each request names its object).
PyKMIP 0.10 drives the key-pair path in an opt-in proof
(TRSTCTL_PYKMIP_PROOF=1 go test ./internal/kmip -run PyKMIP -v). With protocols.kmip.client_registrations, one
listener serves every tenant's self-service clients: a tenant registers a
client (/api/v1/kmip/clients, trstctl-cli kmip clients, the Transit page)
and gets a one-time enrollment, and the client's first KMIP request presents
its own TLS certificate with that username and password, which binds the
certificate to the tenant. The handshake proves the client holds the
certificate's key and the secret proves its tenant issued it, so no tenant
can claim another tenant's client by registering a certificate it has only
seen; a removed registration is refused at its next request. A two-tenant
proof drives this with the listener's own clients. Appliance-specific
templates (a vendor's key naming and attribute conventions) are not shipped:
each needs the appliance itself to prove it. Production
Transit durability remains conditional on the deployment KEK; the KMIP listener additionally remains conditional on its licensed runtime,
a tenant binding or client registrations, and configured mTLS trust.
Authorization policy gates and ABAC overlays: served by the binary
The RBAC guard, ABAC deny overlay, OPA/Rego default-deny policy gate, RA scope
split, and dual-control approval are enforced by the running binary, not just
in library code. The RBAC guard runs on every guarded API route. When
auth.abac.enabled is set, the ABAC deny overlay runs after RBAC on guarded
routes using request, actor, environment, and time attributes; on the served
lifecycle transition (POST /api/v1/identities/{id}/transitions) for issue,
deploy, and revoke, trstctl adds identity resource tags before the deny check.
The OPA/Rego lifecycle policy gate then gates issue/deploy/revoke fail-closed
before the orchestrator records the transition or enqueues the mint/revoke
effect. The gate is tenant-scoped under per-tenant database isolation,
recorded as immutable events, and runs on its own bounded worker lane so a
policy flood fast-rejects rather than starving other subsystems.
Registration-authority (RA) separation and dual-control approval are served.
The gate enforces the RA scope split: a privileged issue/revoke transition
requires the certs:issue authority, so a certs:request-only requester (the
ra-officer) cannot self-issue. When dual control is enabled
(ca.policy.require_approval), a privileged action is denied until a
distinct approver records an issue, rotate, or revoke approval via
POST /api/v1/identities/{id}/approvals (itself requiring certs:issue);
self-approval is rejected, backed by tenant-isolated approval-request and
approval records. This is the served half of the "loaded gun" defense — the
bootstrap token already withholds certs:issue; the served mint now enforces
the RA split and dual control too. The /request and /approvals console
pair is served for profile-bound certificate requests: it records
requester/profile/purpose metadata, keeps the request in requested, blocks
requester self-issue, accepts a distinct approval, and mints through the
signer-backed outbox. The request is a first-class object
(/api/v1/issuance-requests, trstctl issuance-requests) with its own
lifecycle: requested, then approved, denied (by a distinct approver, with
a reason), cancelled (by the requester), expired (seven days after it
opened, recorded with no decider because nobody decided), or issued; the
list keeps closed requests for audit (A117, proven on the lab: open, withdraw,
deny with a reason, refused self-approval, approve).
The OPA/Rego policy gate is default-deny on issue/deploy/revoke. With
ca.policy.enabled set, the served binary invokes the embedded policy engine
on every issue/deploy/revoke transition: the request is denied unless the
deployed Rego policy explicitly allows it (default-deny, fail-closed). The
policy input carries the action, tenant_id, the actor, and the bound
profile name, so an operator can enforce a real Rego document at runtime. A
non-compiling policy module is a hard startup error, an evaluation error
denies, and a saturated policy pool sheds with a 503 (never an allow). The
built-in base policy is default-deny, permits revocation, and requires a
bound certificate profile to issue/deploy (composing with the profile
enforcement below). Enforcement is off by default (ca.policy.enabled=false)
so an in-place upgrade does not silently start denying; the RA scope split is
enforced for privileged transitions regardless of this flag. Live policy
authoring, activation, listing, and rollback are served through
POST /api/v1/policy/versions, GET /api/v1/policy/versions,
POST /api/v1/policy/versions/{id}/activate, and
POST /api/v1/policy/versions/{id}/rollback; activation compiles the module,
records policy.version.activated, and only then installs it into the
running mutation gate.
The ABAC deny overlay is served. With auth.abac.enabled set, the served
binary compiles a package trstctl.abac Rego module at startup and evaluates
it after RBAC. It is deny-only: it cannot grant access that RBAC refused.
Every guarded route carries input.permission,
input.resource.request.method, input.resource.request.path, actor roles,
input.env, and UTC time fields; issue/deploy/revoke transitions also carry
identity metadata and flattened identity attributes such as
input.resource.env and input.resource.tags.service — supporting controls
like "prod certs may issue only during a change window." Bad Rego is a
startup error, evaluation errors deny with 403, saturated policy workers
return 503, and decisions are recorded as policy.abac.decision events.
Candidate policy dry-runs are tenant-facing through /api/v1/policy/dry-run
and the /policy workbench.
Independently of the policy flag, when a default certificate profile is bound
(ca.default_profile) the served mint validates the request against the
active profile version and rejects an out-of-profile request before signing
(an issuance.profile_evaluated deny event) — so the served mint is
profile-gated, not ungated.
Regulated CA governance mode is one coherent posture switch. Previously the
policy gate, four-eyes dual control, the bound default profile, revocation
publication, and FIPS were each enabled independently, with no single mode
that refused to start unless they were all coherently present — a compliance
deployment could half-enable the posture and silently drop a control. With
ca.governance_mode=regulated the running binary fails startup unless all of
the OPA policy gate (ca.policy.enabled), distinct-approver four-eyes dual
control (ca.policy.require_approval with a >= 2 threshold), a bound
default certificate profile (ca.default_profile), revocation publication
(ca.crl_distribution_points and/or ca.ocsp_servers), and — when
ca.require_fips is declared — an active FIPS 140-3 module are present
together, each with an actionable error. A complete regulated config boots;
the default (standard) posture imposes no coupling. The switch is enforced
in the served startup/config validation path, where the FIPS power-on
self-test already asserts the module when required. See
configuration → regulated CA governance mode.
Plugin isolation: first-party in-process, third-party sandboxed
This is a deliberate, documented trust boundary, not an accident.
- Shipped connector families run in the WASM sandbox by default, on the
control plane (
connectors.isolation: wasm) and on enrolled host agents and network relays (trstctl-agent --connector-isolation): each binary embeds one digest-pinned first-party connector module, and every connector operation runs in a fresh wazero instance of it whose only effects are the sandbox operations its grant permits, performed by the host; the instance's memory is wiped afterwards. Every family is proven to behave identically in and out of the sandbox (same effects, same errors, same optional operations).in_processremains available. An agent does not report its choice, so the connector catalog claims isolation only for families the control plane runs. The shipped CA integrations still run as trusted in-process Go code. Their blast radius if one is defective is the executing process — on the control plane, the database connection pool (confined to the tenant by per-tenant database isolation) and the signer client handle (it can request signatures), but not the CA private key, which stays in the separate signer process. They are mitigated by code review, the conformance suite, the connector SDK's capability-scoped sandbox facade, and per-subsystem bounded worker lanes. - The WASM plugin host (wazero) is real and is the isolation boundary for third-party plugins. A loaded plugin has no ambient capabilities and only the host functions its grant permits; the host holds no database pool or signer handle; and a deliberately misbehaving plugin is proven contained by test. The shipped connector families now run on it (above); moving the CA integrations onto it is what comes next. See the plugin trust model.
- Plugin extensibility is served by the binary. The WASM plugin host is
wired into the served control plane: when
plugins.enabledthe running binary loads operator-supplied CA plugins fromplugins.ca_dirand connector plugins fromplugins.connector_dir(or the legacy connector aliasplugins.dir). A signed CA plugin is listed underGET /api/v1/external-casand issues throughPOST /api/v1/external-cas/{id}/issue; a signed connector plugin routes servedconnector.deploywork through the plugin's capability sandbox (the same capability-grant model the connector SDK uses) — tenant-scoped under per-tenant database isolation, recorded as immutable events, on the plugin's own bounded worker lane. The plugin runs in its own wazero runtime with no database pool or signer handle, an operation outside its grant is denied at runtime, and the surface is off by default. The built-in connector families run in the same kind of sandbox (see above); the CA integrations still run in process. - Served plugins are signature/provenance-verified. The served loader admits
a
.wasmmodule only after its detached Ed25519 signature verifies (through the single isolated cryptography path) against the operator-configured trusted-key set (plugins.trusted_key_files), with an optional content-digest pin (plugins.pinned_digests). An unsigned, wrong-key, byte-tampered, or unpinned module is refused and the binary fails closed at startup — it never instantiates an unverified plugin. The rawHost.Load,ConformanceandConformanceUnderGrantmethods exist only in builds explicitly taggedtrstctl_conformanceand in package tests. Normal and Core production builds cannot compile a call to them. An architecture guard confines the private loader to verified admission and conformance helpers. Author-side conformance executes guest code under its grant; it checks compatibility, not publisher trust. The served surface runs the provenance gate before guest start code and keeps the wazero sandbox as defense-in-depth.
Protocols
ACME server with ARI: all three domain-validation challenges are validated for real, each failing closed — HTTP-01 (RFC 8555 §8.3), DNS-01 (§8.4, the
_acme-challengeTXT digest), and TLS-ALPN-01 (RFC 8737, theacme-tls/1id-pe-acmeIdentifierhandshake) — behind a multiplexer with an automatic method selector (wildcards → DNS-01, no inbound:80→ TLS-ALPN-01, else HTTP-01). The prior accept-everything validator has been removed from the production build (it survives only in the test binary). A DNS-01 solver with a reference provider and conformance harness ships for the publish side. A real RFC 8555 client conformance suite exercises HTTP-01 end to end (the production validator fetches the published key authorization, multi-SAN issuance, a wrong key authorization fails closed), and the same protocol-conformance routine runs as a differential against Pebble (the reference test ACME CA) in CI — so a divergence from the reference surfaces as a failure. Hosted DNS provider coverage is served through the DNS-01 provider catalog (GET /api/v1/acme/dns-01/providers): Route 53, Cloudflare, Google Cloud DNS, Azure DNS, RFC 2136, webhook, NS1, Akamai, UltraDNS, and acme-dns; the catalog exposes secret-reference fields and capability grants, not raw provider tokens. Tenant DNS-01 provider configs are served throughPOST/GET/PUT/DELETE /api/v1/acme/dns-01/provider-configs, andPOST /api/v1/acme/dns-01/preflightevaluates delegation, TXT propagation (read from live DNS unless the caller supplies its own observation, which the response labels), live CAA, method, and wildcard policy before issuance. Served ACME DNS-01 challenge acceptance checks live CAA before any DNS write, then publishes and cleans throughacme.dns01.present/acme.dns01.cleanupoutbox rows using tenant provider configs and secret-reference-backed credentials. Wildcard X.509 identity issuance requires an explicit blast-radius acknowledgment andvalidation_method=dns-01; deployed wildcard identities renew through the lifecycle scheduler'sca.renewpath with rotation evidence. The ACME server is served by the running binary: it is mounted on the control-plane TLS listener at/directory+/acme/...and brokers issuance through the orchestrator-backed path — signed in the isolated signer (so the CA key never enters the API process), tenant-scoped, recorded as immutable events, idempotent (deduplicated byIdempotency-Key), and profile-gated. A stockgolang.org/x/crypto/acmeclient with an ECDSA account key drives the served handler end to end (new-account → new-order → http-01 → finalize) and downloads a real, signer-issued certificate; a served acceptance test asserts the cert verifies and acertificate.recordedevent exists, then revokes via ACMErevokeCertand asserts the served OCSP responder returns revoked. The directory advertises the mandatoryrevokeCertandkeyChangeresources, and the server accepts ECDSA and Ed25519 account keys (not only RSA). Enable it withprotocols.acme.enabledplusprotocols.acme.tenant_id; it activates only when an issuing CA is provisioned and fails closed otherwise. The Protocols console now exposes the tenant-scoped, read-only ARI publication and scheduler-consumption posture. For certificates issued by a configured external ACME CA, the control-plane outbox worker fetches that CA'srenewalInfothrough the selected issuer's egress policy. The scheduler and posture use the retained per-certificate window; the console labels its source and fetch state. A discovered external leaf has no authenticated issuer, so it is matched to an authority by its Authority Key Identifier against that authority's operator-declaredari_issuer_key_ids(the Subject Key Identifiers of its issuing intermediates); a match gets the same upstream poll, the request and observation events, the row and the posture carryprovenance: issuer_key_idso the two bindings are never confused, and the worker re-matches the key id before every GET. A discovered leaf whose issuer no authority declares is listed with a local estimate and nothing is polled. Local Pebble evidence qualifies the supported client contract, not a public CA's production availability or network behavior. The upstream client presents an RFC 8555 §7.3.4 external account binding when theletsencrypt-type authority setseab_key_idandeab_hmac_key_reftogether (CertiNext/InCommon, HARICA/GÉANT TCS and other EAB-gated CAs); the MAC key is a credential reference held as locked, zeroed bytes, never logged or served, and one field without the other refuses to start. The binding is proven against a local EAB-requiring ACME double (registration without a binding or with the wrong key is refused, with it issuance and revocation run); a live proof against CertiNext or HARICA needs a design partner's credentials. The ACME admin console is served (A52):GET /api/v1/acme/accountslists the mount's accounts (status, admitting EAB credential, order and certificate counts),GET /api/v1/acme/accounts/{id}/ordersdrills one account down to each order's authorizations and challenges and names the issued leaf's serial, fingerprint and inventory record id, andPOST /api/v1/acme/accounts/{id}/deactivateretires an account on the operator's authority (the sameacme.account.upsertedevent a client's own deactivation writes: new requests under the key are refused, pending orders are cancelled, issued certificates stay). Revocation from that console runs throughPOST /api/v1/certificates/bulk-revokewith the named record id, so it carries the certificate path's approvals and OCSP/CRL effects rather than a second revocation path. CLIacme accounts list|orders|deactivate. An account is named by the public RFC 7638 thumbprint of its key; no served field carries key material, and a token for another tenant sees an unserved list. Client setup stays the readiness panel's credential-free command.External account bindings are authorizations, not door keys. RFC 8555 §7.3.4 EAB proves an ACME account key was pre-authorized out of band. The server used to verify that proof and discard the key id, which made every admitted account identical: nothing recorded which credential let it in, so nothing could scope what it asked for next, count what it had taken, or stop one credential without stopping all of them. An account now remembers its
kid— in its state event, so it survives a replay — and every order under that account is checked against that credential's policy. A credential may carry allowed identifiers (exact names or*.suffix, which covers the apex too), an order quota, and a validity window; an out-of-scope identifier, an exhausted quota, or a closed window refuses the order fail-closed, names the cause, and records anacme.eab.order_deniedevent. A single out-of-scope identifier refuses the whole order — issuance is granted or it is not.GET /api/v1/acme/eab-credentialsserves each credential's scope, quota, window, and live accounts-bound / orders-created / orders-denied counters, andPOST /api/v1/acme/eab-credentials/{kid}/disable(and/enable) stops or resumes new accounts and orders under one credential at runtime. Disabling is a closed tap, not a revocation: certificates already issued under the credential stay valid. The Protocols console shows the same list and carries the disable action. The policy is served as well as configured (A53):PUT /api/v1/acme/eab-credentials/{kid}/policy(CLIacme eab policy) replaces a credential's allowed identifiers, order quota, validity window and pinned profile at runtime; the change is recorded as anacme.eab.policy_updatedstate event and replayed after a restart, so aprotocols.acme_eab.keys[]entry is the starting policy, not the only one. Rotation is served without minting a secret (A50/A54): the operator writes the successor MAC key to the tenant secret store first, thenPOST /api/v1/acme/eab-credentials/{kid}/rotate(CLIacme eab rotate) withsuccessor_key_idand asecret://hmac_key_refregisters the successor under the predecessor's policy. The key is read through the secret store into locked memory, theacme.eab.credential_rotatedevent carries only the reference, and replay resolves the reference again; a reference that no longer resolves leaves the successor listed asunavailable, which admits nothing, rather than dropping it or starting the mount without it. The predecessor keeps working until the operator disables it. trstctl does not mint external account credentials over the API and accepts no key in a request body, because that would mean a shared MAC secret in an HTTP body; the HMAC key stays byte-backed in locked memory and appears in no served response, in any encoding. Per-credential profile selection is served:keys[].profilepins the tenant profile every order under that credential is issued with (the server reads that profile's authentication mode and signs under it), and without a pin the tenant'sca.default_profileapplies. The served disable verb cannot re-enable a credential that configuration disables; config is the floor.EST (RFC 7030), SCEP (RFC 8894), CMP (RFC 4210/6712), the SPIFFE Workload API, and the SSH CA issuance servers are served end-to-end by the running binary, each behind the same issuance seam as the API mint: signed in the isolated signer, tenant-scoped, recorded as immutable events, idempotent, and profile-gated:
- EST at
/.well-known/est/...(Bearer-API-token authenticated on top of TLS). The challenge is emitted by the configured authenticator: the served tenant route advertises Bearer pluscerts:requestand returns bounded RFC 6750invalid_request,invalid_token, orinsufficient_scoperesults, while BasicAuthenticator continues to advertise Basic. The conventional uppercase header spelling and a strict libest token wrapper keep the pinned challenge-driven reference client interoperable without changing ordinary raw Bearer behavior. SCEP at/scep, CMP at/cmp— mounted on the control-plane mux and exercised by served round-trip acceptance tests (a stock base64-PKCS#10 EST enroll, a CMS-enveloped SCEPPKIOperation, a CMPp10cr) that each download a real, signer-issued certificate verifying against the served CA and assert acertificate.recordedevent in the tamper-evident log. SCEP/CMP use a sealed RSA transport identity atprotocols.ra_key_filefor CMS (deliberately not the CA key, which stays in the isolated signer); keep that file on shared persistent storage in HA so cached clients survive restarts and rolling deploys. - The SPIFFE Workload API is served as a gRPC service on a Unix domain
socket (
protocols.spiffe.enabled), so aspiffe-helper/go-spiffe/Envoy-SDS client dials the socket and fetches X.509-SVIDs, JWT-SVIDs, X.509 bundles, and JWT bundles. X.509-SVIDs are signed through the isolated signer; JWT-SVIDs use the signer-backed JWT handle and the servedValidateJWTSVIDRPC validates them against the served JWT bundle. A served acceptance test drives the SPIFFE Workload API wire protocol (with the mandatoryworkload.spiffe.iometadata) over the socket and validates both SVID families. A required CI job also runs stock go-spiffe and stockspiffe-helperagainst that served socket; go-spiffe is a test-only dependency so the served binary does not take a new runtime dependency for the proof. - Host-served Workload API (B3): a host agent started with
--workload-api-socketserves the same Workload API on its OWN machine, for the workloads that run there. It attests each caller from the kernel's record of the connecting process (SO_PEERCRED on Linux; LOCAL_PEERCRED, LOCAL_PEERTOKEN and LOCAL_PEERPID on darwin) and renders SPIRE-shapedunix:uid:/unix:gid:/unix:path:selectors; a platform with no peer-credential mechanism REFUSES to serve rather than issuing without attestation. The SVID key is generated on that host and only its public half travels: the control plane signs an SVID for a key it has never seen, over theFetchWorkloadSVIDnode API on the agent channel. Authorization is per node, not per agent-fleet: a registration entry carries aParentIDnaming the node permitted to deliver it, taken from the certificate the agent authenticated with, so an agent that misreports selectors reaches only the workloads on its own machine. An entry with NOParentIDis deliverable by no agent at all and stays on the control plane's socket — so a forgotten field refuses rather than widening who may impersonate a workload, and scoping an entry moves it rather than duplicating it. On darwin the path comes from the connecting process's audit token (LOCAL_PEERTOKEN): its pid and pid version, read at connect time. The path is resolved only while that pid version still owns the pid, checked before and after the read, and LOCAL_PEERPID must agree, so a pid reused by a later process yields no path selector rather than the new process's; an entry requiringunix:path:then does not match. As on Linux, the path is the executable's at attestation time: a process that execs another binary after connecting is attested as that binary. The control plane's own socket no longer issues SVIDs to workloads unlessprotocols.spiffe.serve_local_workloads: trueis set:FetchX509SVIDandFetchJWTSVIDon it are refused with a message naming the host agent's--workload-api-socket, while trust bundles and JWT validation stay served for validators. Each refusal, and each issuance when the switch is on, emits thespiffe.workload_api.local_socket_usedaudit event, so an operator can see which workloads have not yet moved rather than assuming they all have; the SPIFFE readiness check reportslocal_workloads_served. The eval profile keeps the switch on, being one machine with no host agent. The Workloads console reports, per host, whether it serves the socket, how many SVIDs it has issued since its agent started (a counter that resets on restart, and says so), and when it last reported — with "not reported" kept distinct from "not serving", because the first is fixed by upgrading an agent and the second by changing a flag. - Kubernetes TrustBundle distribution is served for public CA-bundle
propagation: the agent reconciles cluster-scoped
TrustBundle.trstctl.comresources, validates thatspec.caBundlePEMcontains only PEMCERTIFICATEblocks, writes namespace ConfigMaps, and marks status with target count,bundleSHA256, and Ready=True. The posture routeGET /api/v1/kubernetes/trust-bundles, CLI commandtrstctl-cli kubernetes trust-bundles, and Workloads console disclose the CRD, RBAC, ConfigMap target, and residuals: the controller is event-driven — it watches its TrustBundles and the ConfigMaps it wrote, converges a change or a deleted copy within about a second, and writes nothing when a copy already holds the bundle. One bundle can be rolled out from the control plane to every reporting cluster, canary first (/api/v1/kubernetes/trust-rollouts,trstctl-cli kubernetes trust-rollouts, Workloads): each cluster's agent receives it in the answer to its posture report and applies it as a TrustBundle it owns, and a cluster reads verified only when its own controller reports the bundle distributed with the rollout's hash. A TrustBundle can also name a Secret in each target namespace and a ClusterTrustBundle that pods mount through a projectedclusterTrustBundlevolume, and the opt-in CSI node plugin (deploy/kubernetes/csi-trust.yaml, drivertrust.csi.trstctl.com) serves a TrustBundle to a pod as an inline volume; focused kind tests prove each target end to end, including a changed bundle reaching a running pod. Their limits: Secret targets outside the agent's namespace need the opt-in grant indeploy/kubernetes/rbac-trust-secrets.yaml, which lets the agent read any Secret by name; ClusterTrustBundle needs a cluster servingcertificates.k8s.io/v1beta1(off by default through Kubernetes 1.35) and the kubelet projection feature gate; the CSI plugin runs as root without capabilities to write into the kubelet's pod directories, and a changed bundle reaches pods on the kubelet's next republish. A bundle that cannot be distributed is marked Ready=False with the cause. - The SSH CA is served at
/ssh/...(protocols.ssh.enabled): cert issuance plus the OpenSSH binary KRL at/ssh/krl(sshd'sRevokedKeysconsumes it); a served acceptance test issues a user cert (verified withssh-keygen -L), revokes it, and confirms the served KRL is the binary format. The SSH workflow API and CLI also serve status, explicit-confirmation trust-rollout evidence, attestation-gated user cert issuance, KRL revocation, and host retirement handoff. The SSH CA key lives in the isolated signer under its own handle constrained to SSH-cert signing. - The RFC 3161 TSA is served at
/tsa(protocols.tsa.enabled): clients POSTapplication/timestamp-queryTimeStampReqbodies and receiveapplication/timestamp-replyTimeStampRespbodies. The timestamping key lives in the signer under its own stable handle, the TSA certificate is persisted atprotocols.tsa_cert_file, and the certificate carries the criticaltimeStampingEKU that stock OpenSSL enforces. The authenticatedPOST /api/v1/protocols/tsa/qualificationsurface and Protocols console panel review the exact tenant-bound mount, certificate, isolated signer, audit, and bulkhead posture without reading files, calling the signer/network, issuing a timestamp, or writing state. They name a secure repair for each failed gate and keep retry available. This is readiness only; stock OpenSSL over/tsaremains the wire and signature-verification proof. - The code-signing service is served by the running binary at
POST /api/v1/code-signing/signandPOST /api/v1/code-signing/keylesswhencode_signing.enabledis configured. Tenant-scoped persistent keys and one-use keyless keys stay in the isolated signer process. GitHub OIDC identity is verified from tenant-pinned JWKS before the Fulcio SAN/issuer is derived. Rekor publication uses the transactional outbox and acknowledges only an exact HashedRekord receipt whose signed-entry timestamp verifies under the operator-pinned log key.
Each protocol surface is gated by
protocols.<name>.enabledand binds a tenant viaprotocols.<name>.tenant_id. All protocol toggles default off until an operator explicitly binds the served endpoint to a tenant; if a protocol is enabled without a tenant, startup validation fails before the route is exposed (per-tenant isolation forbids minting evidence into a blank tenant). All protocols activate only when an issuing CA is provisioned. HTTP protocol namespaces stay machine-only even while a toggle is off:/directory+/acme/,/.well-known/est/,/scep,/cmp,/ssh/, and/tsareturn a stable404 application/problem+jsonresponse instead of falling through to the React console. Unknown/directory/,/cmp/, and/tsa/children are reserved too. The Protocols register does not infer availability from HTTP 200 alone; it verifies each responder's protocol-specific status, media type, and public discovery payload, so console HTML cannot masquerade as ACME, EST, SCEP, CMP, SSH CA, or TSA readiness.- Reference-implementation differentials: cross-checked against
an independent implementation in addition to our own parser. ACME: a
differential against Pebble (the reference test ACME CA) as a dedicated
CI job, plus a stock certbot CI transcript (certbot manual DNS-01
issues, renews, and revokes through the served
/directoryendpoint while CI archives public challenge records, client logs, and issued certificates). EST: a differential against the OpenSSLpkcs7parser/verifier on everymake test(so/cacertsand/simpleenrolloutput is validated by code we did not write), plus a dedicated CI job that builds a checksum-pinned libestestclientfrom source, removes its verbose access-token print before archiving transcripts, and requires it to follow the Bearer challenge and perform simpleenroll against the served EST endpoint. SPIFFE Workload API: a served stock-client differential — the real go-spiffeworkloadapiclient fetches an X.509-SVID, a JWT-SVID, and JWT bundles, and validates the JWT-SVID over the served UDS; stockspiffe-helperwrites the served X.509-SVID, key, and trust bundle to disk. CMP: a dedicated stock-client CI transcript — OpenSSLcmp -cmd p10crcreates the request, enrolls through the served/cmpendpoint, accepts the protected response, and uploads the request/response/cert/log artifacts. SCEP: a dedicated stock-client CI transcript — a SHA-256-pinnedsscepv0.10.0 build fetches the served CA-plus-RA bundle, classifies its numbered public files by exact issuing certificate and signing constraints rather than suffix order, enrolls through/scep/pkiclient.exe, and uploads both CA/RA files plus the captured PKIOperation request/response and client logs. TSA: a dedicated stock-client CI transcript — OpenSSLts -querycreates a DERTimeStampReq, CI POSTs it to the served/tsaendpoint, OpenSSLts -verifyvalidates the returnedTimeStampResp, and public request/response/log artifacts are uploaded. - SSH KRL distribution format. The SSH CA's key-revocation list is emitted
in the OpenSSH binary KRL format, the artifact
sshd'sRevokedKeysandssh-keygen -Q -fconsume — verified end-to-end by a test that has stockssh-keygenreport a revoked certificate as revoked using trstctl's KRL (and a non-revoked one as valid). A legacy JSON revocation snapshot is retained for programmatic callers. - Public-CA profile linter. Issued certificates are checked by a built-in structural RFC 5280 / CA-Browser-Forum profile linter in the issuance test suite — version, serial bounds, validity ordering/length, basicConstraints, key usage, SAN presence, SKI/AKI presence, weak-signature and minimum-key-strength checks — and the suite is red on a deliberately-broken profile. The CI gate also generates a PEM corpus for every emitted X.509 profile shape (served leaves, mTLS agent certificates, SPIFFE X.509-SVID, TSA, and the issuing CA), runs pinned zlint over the served CA plus that corpus, and uploads the generated fixtures and JSON lint transcripts as artifacts. This is a private-CA assurance gate (for your own internal PKI), not a claim that trstctl operates as a WebPKI public CA.
- EST at
SPIFFE transport (Workload API): the X.509-SVID document is spec-shaped (a single
spiffe://URI SAN, correct key usage), and the Workload API is served as a gRPC service on a Unix domain socket (protocols.spiffe.enabled). Aspiffe-helper/go-spiffe/Envoy-SDS workload dials the socket forFetchX509SVID,FetchX509Bundles,FetchJWTSVID,FetchJWTBundles, andValidateJWTSVID. The X.509-SVID workload key is minted server-side and returned in the response (per the spec); the X.509-SVID CA is the served issuing CA in the signer and the JWT-SVID signing key has its own signer handle. The Workload-API gRPC/protobuf contract is vendored verbatim from go-spiffe so the wire format is byte-identical without a build-time go-spiffe dependency.SPIRE upstream authority: the
trstctl-spire-upstream-authorityplugin puts the served CA hierarchy behind SPIRE as its X.509 upstream: SPIRE sends its local CA CSR to/api/v1/ca/authorities/{id}/intermediates/csr. The served route signs only under an approvedissue_intermediate_csrceremony for that exact CSR and the plugin sends no ceremony ID, so every SPIRE CA rotation needs custodians to approve a ceremony for its new CSR; there is no standing approval for an upstream authority. CI's SPIRE container test signs with a stand-in for trstctl; a served container test opens and approves each ceremony through the API. The plugin binary is built bymake buildand published from every release tag astrstctl-spire-upstream-authority-linux-{amd64,arm64}GitHub Release assets with a SHA-256 manifest and SLSA provenance. SPIRE loads it from its own host filesystem (plugin_cmd), so it ships as a standalone binary and is not part of the trstctl container image. The plugin also publishes each SPIRE server's JWT signing key to/api/v1/spiffe/upstream-jwt-keysand returns every unexpired key the tenant's SPIRE servers published, with updates on the open subscription, so SPIRE servers sharing a trust domain through trstctl verify each other's JWT-SVIDs (two SPIRE server containers, each verifying the other's JWT-SVID with its own bundle). trstctl keeps the public keys only; its own SPIFFE JWT authority is not added to that set, and SPIRE's optionalSubscribeToLocalBundleis not implemented. Against a trstctl that does not serve that route (an older trstctl's 404, or a 405 or 501), the plugin reports JWT key publication as unsupported, so SPIRE keeps its JWT keys in its own bundle and still starts; the next JWT key rotation publishes again.Attested issuance transport (REST):
POST /api/v1/workloads/attested-issuanceis the served proof-before-trust mint for workloads that already have their own key pair. The request carries the attestation method, base64 proof payload, public key PEM, and requested TTL; the response carries the signer-issued X.509-SVID PEM, credential id, verified subject, expiry, and attestation metadata. The SPIFFE ID is derived from the verified attestation subject, not caller-supplied text. Acceptance coverage exercises a Kubernetes projected service-account token, an AWS instance-identity document with an emulated trusted root, idempotent replay, and a forged AWS document rejection.AI-agent broker issuance (REST):
POST /api/v1/broker/agent-identitiesis served when the agent broker is configured with attestors, a policy module, trust domain, and signer-backed issuing CA. The route requirescerts:issueand anIdempotency-Key; it verifies the agent proof, evaluates policy before signing, mints a short-lived X.509-SVID, recordscertificate.recorded, emitsagent.identity.issuedoragent.identity.refused, and projects the agent-to-credential edge into the graph. The Workloads page now has an effect-free server preview, explicit issuance, unchanged-request retry after uncertain delivery, and durable broker history with search, state/method filters and detail readback. History requirescerts:readindependently of issuance. It shows original public issuance facts and projection freshness, never raw proof, task text or internal recovery bindings. Missing or privacy-erased facts remain explicitly unavailable. Clearing or reloading the form is not cancellation: inspect history and audit before choosing a new recovery key. Revocation uses shared certificate controls. Fresh-image API/CLI/browser, negative-security and preserved-volume restart qualification is still required before this source workflow can be called a complete F61 vertical slice.Agent key sets for authorization servers:
GET /agent-keys/{tenant}/{agent}/jwks.jsonis public, like the CRL and OCSP routes, and lists the public keys of the agent's valid broker certificates (an unknown agent gets the same empty set); a revoked certificate's key leaves it at once. An authorization server that reads it can authenticate the agent withprivate_key_jwt, which is how an agent with a trstctl identity obtains tokens under MCP authorization (guide); a lab test proves it against Keycloak 26.8 with token exchange. How long a revoked identity can still obtain tokens is set by the authorization server's key cache (Keycloak: thekeyscache lifespan), and tokens already issued live until they expire. What comes next: the same proof with a production identity provider.Ephemeral / JIT issuance: effect-free
POST /api/v1/ephemeral/previewand mutatingPOST /api/v1/ephemeralare served when ephemeral issuance is enabled with approval TTL/threshold, trust domain, and signer-backed issuing CA. Each tenant supplies public verification material through an enabled workload attester trust source; no process-wide trust source silently applies to every tenant. A requester withcerts:requestpresents a proof and public key; trstctl verifies the proof, opens an approval request, and enqueues the approval notification intent in the same tenant transaction. A distinct approver withcerts:issuerecords approval atPOST /api/v1/ephemeral/{id}/approvals, where{id}is the genuine queueapproval_request_idand the JSON body carriesaction: issue, the same UUID asrequest_id, and its matchingintent_digest; the requester then calls/api/v1/ephemeralwith a freshIdempotency-Keyto mint the short-TTL credential. The Workloads page now provides the same exact-preview, send-for-approval, approval-queue handoff, safe-retry, and public-certificate collection workflow. It retains only digests and public metadata after issue; the proof is cleared and private-key bytes never enter trstctl. Ephemeral API keys are served separately atPOST /api/v1/ephemeral/api-keysandtrstctl-cli ephemeral api-keys issue: callers providesubject,scopes, andttl_seconds, the raw token is returned once, and the leaseworker emitsapi_token.revokedat expiry. Headless operators can usetrstctl-cli ephemeral preview,ephemeral issue, andephemeral approve; the approval command requires the genuineapproval_request_idin both the path and body plus its matchingintent_digest. The certificate inventory retains issued JIT leaves and their public custody metadata. The health dashboard separates their intended short expiry intoexpected_ephemeral; it does not send ordinary renewal alerts for them or put them in the expiry action queue. Operators request a new attested credential from Workloads after expiry. This classification follows the retained issuance command even if a discovery scan later changes the source label.Agent ↔ control-plane mTLS gRPC channel: the agent steady-state channel is served by the running binary when
agent_channel.enabled(off by default — an upgrade does not silently open an agent port). The control plane mounts an agent-facing gRPC listener (default:9443) over mutual TLS, and an enrolled agent connects to it to (a) heartbeat its inventory/status — the server records the agent tenant-scoped and emits anagent.heartbeatevent in the tamper-evident log; (b) renew its own certificate before expiry — a fresh cert is minted through the signer-custodied agent CA, idempotently on the presented serial (deduplicated so a retry does not mint twice), recorded as anagent.cert.renewedevent; and (c) report local inventory as metadata-only discovery findings, including public OS/Java/NSS/browser/ Windows trust-store anchors and private-key-material locations/ classifications from configured roots. Inventory reports create a tenant-scoped discovery source, run, finding rows,discovery.*events, and credential-graph nodes; they do not carry private keys, PEM/DER key bytes, or secret values, and secret-looking inline metadata keys are rejected before projection. The tenant is derived from the agent's verified client-certificate SPIFFE SAN, never a request field. TheGET /api/v1/agentsresponse also publishes the servedagent.mtls.ReportInventorypath and the source kinds the shipped agent binary can actually collect —filesystem,trust-store,k8s-secret,windows-store,private-key, andssh— each with the flags that switch it on, so a capability that is listed but unconfigured is not read as coverage that is running.pkcs11ships only in a cgo-enabled agent build, because it dlopens a vendor module and the default agent binary is deliberately statically linked; the census is build-dependent so a cgo-free binary neither carries the reader nor advertises the kind. All three formerly-unbuilt kinds were previously listed here and on the API, which read as a Windows estate, token store, and Kubernetes Secrets being inventoried when nothing was collecting them. Advertised capability is derived from the agent package's own record of what it ships, anddocs/agent_advertised_capability_test.gofails the build if a kind is advertised without a constructor the agent binary calls.k8s-secretleft the unbuilt list when its enumerator was actually wired: with--inventory-k8s-secretsthe agent enumerates the TLS Secrets in its own namespace through the in-cluster service account, readingtls.crtand nevertls.key, and reports metadata-only findings over the same mTLS inventory path as every other source. It needs list access to Secrets in that namespace.--inventory-k8s-namespaces allor a comma-separated list widens one agent's inventory across namespaces (allneeds cluster-scoped list on namespaces and Secrets, shipped opt-in asdeploy/kubernetes/rbac-inventory-namespaces.yaml); a namespace the service account cannot list is reported as an error for that namespace after the readable ones, never as an empty result.windows-storeleft the unbuilt list when its crypt32 reader shipped: with--inventory-windows-storesthe agent opens the named machine or user stores read-only, walks each certificate context, and reports metadata only — it never asks the platform to export a private key, so a key held in a TPM or on a smart card is untouched and irrelevant.MY,WEBHOSTING,CA,ROOTandTRUSTEDPUBLISHERare supported;WEBHOSTINGis where IIS keeps site certificates on current Windows Server, which is the population most likely to expire unowned. An unreadable store fails the report rather than contributing nothing, and on a non-Windows build the source returns an error, never an empty inventory — telling a Linux operator their Windows estate is clean would be worse than the original defect, because it would arrive with the authority of a scan that never happened.pkcs11completed the set: a cgo build opens the configured module, walks each matching token over a read-only, non-read-write session, and readsCKA_VALUEfrom objects of classCKO_CERTIFICATE. It never searches forCKO_PRIVATE_KEY, never callsC_Sign, and never asks a token to export anything — a PKCS#11 key is normallyCKA_EXTRACTABLE=falseand could not leave regardless, but the code does not ask. Login is optional: public certificate objects are readable without one, and the PIN, when a token needs it, comes from a file rather than a flag, because process arguments are readable by anyone who can list processes — the same rule the bootstrap token follows. A cgo-free build returns an error rather than an empty token inventory, and does not advertise the kind at all. The channel is behind its own bounded agent worker lane and per-connection gRPC stream cap, so a heartbeat or renewal storm sheds withResourceExhaustedrather than starving API, protocol, outbox, or signer capacity. Agents announce an explicit protocol/capability handshake and schedule heartbeats from the server hint with bounded jitter, so rolling upgrades and fleet restarts do not synchronize a thundering beat. The agent CA key lives in the isolated signer under a stable handle, so it does not regenerate per boot — an agent's pinned CA survives a control-plane restart (the earlier in-process/per-boot stand-in is replaced when the channel is enabled, and the same signer-custodied agent CA also signs the bootstrap enrollment, so a bootstrap-enrolled agent is accepted on the steady-state channel). The shipped chart exposes the channel: whenagentChannel.enabled, the control-plane Service publishes the agent port9443(agent-grpc), the container exposes it, and the NetworkPolicy admits it (from the configuredagentChannel.allowedCIDRsplus the in-cluster peers the API admits) — so the fleet manifests (deploy/kubernetes/daemonset.yaml, the Windows MSI) that point agents at:9443reach a served port. This is distinct from the isolated signer's:9443(a signer-only Service undersigner.mode=isolated, which admits only the control plane). An untrusted/unpinned agent client is rejected at the mutual-TLS handshake (fail-closed). Proven end-to-end by acceptance tests (real signer + embedded Postgres: enroll → heartbeat → endpoint inventory report → served API capability readback → Discovery findings → graph node, plus renew → idempotent retry → reject untrusted) and rendered-chart assertions.Embedded HTTP enrollment renewal:
POST /enroll/bootstrapis mounted on the served control-plane HTTPS listener. Whenagent_channel.enabled, the binary also mountsPOST /enroll/renewalon a dedicated agent-CA mTLS HTTPS listener (agent_channel.http_renewal_addr, default:9444) that usestls.RequireAndVerifyClientCertsemantics against the signer-custodied agent CA. Bootstrap consumes a one-time token. Renewal requires the current verified agent client certificate, rejects missing or untrusted peers at the live TLS boundary, rejects expired verified peers in the handler, and deduplicates on the presented certificate fingerprint plus CSR when the served API has idempotency storage. The renewed certificate is tenant-attributed from the verified peer, never from a request header or CSR field.
Revocation
Revoking a credential through the running binary is real and recorded, not
a no-op. Transitioning an identity to revoked drives the served outbox
handler to mark the issued certificate revoked in the inventory — via a
projected certificate.revoked event, so the status is reconstructable
from the log on a read-model rebuild, and the certificate API returns
status / revoked_at / revocation_reason so the revocation is visible
on the served surface (a revoked cert reads "revoked", not silently
"active") — and project the certificate's serial into the revocation
read model from the same event, so OCSP/CRL state is rebuilt from the log
instead of from a side write.
The online revocation-distribution surface is served: the running binary
mounts an RFC 6960 OCSP responder at /ocsp/{tenant} (GET base64-in-path
and POST application/ocsp-request), an RFC 5280 full CRL endpoint at
/crl/{tenant}, a manifest at /crl/{tenant}/manifest.json, partitioned
shard CRLs at /crl/{tenant}/shards/{index}, and RFC 5280 delta CRLs at
/crl/{tenant}/delta/{base}. The freshness scheduler regenerates each
tenant's CRL set ahead of nextUpdate. Trusted issue, renewal, revocation,
protocol-enrollment, and scheduler paths publish CRLs; public CRL reads
are read-only and return 404 until artifacts are already published for a
tenant that has issued certificates.
When a local revocation has been accepted but its event has not projected,
OCSP refuses a Good answer and CRL readers refuse the predecessor artifact.
After projection, CRL readers continue to refuse an older signed artifact
until its revoked count covers the current issuer ledger. The replacement is
published by the outbox worker; clients configured to soft-fail may still
accept a leaf during this unavailable interval, so relay health and stock-client
rejection remain separate evidence. Both are proven on the running product: a
network relay reports the managed CA's CRL fresh and signature-verified with the
revocation counted, and its OCSP answer signature-verified; stock OpenSSL 4
refuses the revoked leaf with verify -crl_check (error 23), reads it as
revoked over OCSP, and refuses a TLS connection made with -crl_check, while a
TLS client that does not check revocation still connects.
A query for a revoked serial returns
revoked over OCSP and the serial appears on the full CRL, its shard, and
any applicable delta CRL within the freshness window; a query for an
issued-but-not-revoked serial returns good; an unknown serial returns a
signed unknown. The shard plan is 4-1024 partitions targeting roughly
100k revoked serials per shard, so 10-100M-row estates use bounded
shard/delta fetches while retaining the compatibility full CRL. These
endpoints are public by RFC design (relying parties check status without
credentials) but run on the API worker lane, so an OCSP/CRL flood sheds
rather than starving the rest of the control plane.
OCSP responses and CRLs are signed through the out-of-process signer: the
signing op crosses the single isolated cryptography path using the same
signer-held CA key the leaf path uses, so the CA private key never
materializes in the control plane — only the digest crosses (for an ML-DSA
authority, the TBS bytes the pure signature covers). Every query
is tenant-scoped. Each published CRL emits a ca.crl.published event that
carries the CRL DER artifact metadata, parent/base CRL number, revoked
count, and validity window, so the published-CRL read model is rebuilt
from the event log. This is exercised end to end in the local acceptance
suite: issue, revoke, assert OCSP returns revoked (and good before
revocation), assert the full/sharded/delta CRLs list the right serials
within the freshness window, and verify the signatures against the issuing
CA over real HTTP against the assembled binary and the real out-of-process
signer.
The CDP/AIA pointers stamped on issued leaves are operator-configured
(ca.crl_distribution_points / ca.ocsp_servers) because the externally
reachable URL is deployment-specific; point them at the binary's
/ocsp/{tenant}, /crl/{tenant}, and, where clients support it, the
shard/delta distribution URLs (behind your ingress) so relying parties
discover and fetch revocation status automatically. Existing leaf
certificates keep the URLs they were issued with until reissued. trstctl
revocation is both authoritative in the product's own inventory/records
and publishable to external relying parties over served OCSP/CRL.
Signer-backed CA hierarchy leaves use their own tenant-and-authority status
routes (/crl/{tenant}/authorities/{ca-id} and
/ocsp/{tenant}/authorities/{ca-id}), including after that authority is
superseded. New leaves are inventoried with their signed public bytes; their
first CRL publication is queued in the same tenant transaction as the
issuance projection. Exact-certificate revocation checks the immutable
issuance event, exact leaf signature and serial ledger before it records the
revocation and queues that authority's CRL. The authenticated
/api/v1/revocation/crls list returns the issuer-specific URL. Managed-CA
CDP/AIA pointers require the operator's ca.hierarchy_revocation_base_url;
without it, standard-mode leaves have no automatic status pointer, while
regulated mode refuses the managed-CA issue request. Existing leaves retain
their original extensions, so reissue them to add discoverable pointers.
Older managed-CA hierarchy version-2 issuance events include the public leaf
and are replayed into inventory on upgrade after exact issuer-signature
verification. Same-version responder-only events (an issuance surface recorded
the leaf for OCSP and CRL, usually beside another event that inventoried it)
stay serial evidence in the projection; when no event inventoried that leaf,
the first lifecycle sweep after an upgrade checks it against the managed CA's
stored certificate and its serial against the CA ledger, then appends one
ca.certificate.inventory_recovered event that gives it its own inventory row
(custody unknown). The inventory workflow then revokes it against that
managed CA.
The inventory identity stays stable across rebuilds and the original issuance
event remains its revocation authority. Version-1 events retain only a serial;
trstctl cannot reconstruct their missing public certificate or infer key custody
from that serial. An operator who still holds such a leaf's public certificate
submits it to POST /api/v1/certificates/managed-ca-recoveries
(trstctl certificates inventory-managed-ca -f leaf.pem), which accepts it only
when one of the tenant's managed CAs signed it, by that CA's stored
certificate, and a version-1 mint of that CA recorded its serial; the leaf then
gets its own inventory row once (custody unknown) and is revoked against that
managed CA through the inventory workflow. A leaf another CA signed, or one no
mint recorded, is refused and records nothing. A version-1 leaf whose
certificate nobody kept remains serial evidence.
CT log submission is served as an outbound side effect, not as an inline
API call: the API validates public certificate PEM and CT log URLs,
records ct.submit outbox rows in the tenant transaction, and the worker
posts RFC 6962 add-pre-chain / add-chain requests. Public HTTPS logs
are required by default; allow_private_endpoint additionally requires
egress:private and private_egress_cidrs destination grants. A request
may name each log's public key in log_public_keys (base64 DER or a PEM
block). trstctl keeps every SCT a log returns (ct.sct.recorded): one from a
log with a key is verified against it over the exact RFC 6962 signed entry
— the certificate, or a precertificate's issuer key hash and TBSCertificate
without its poison — and the log id that key implies, and is recorded
verified or invalid with the reason. One from a log without a key, or a
precertificate SCT the submitted chain cannot check (no issuer, the wrong
issuer, or a precertificate signing certificate), is kept unverified, and
the response names the logs without a key. For each verified SCT the
leader fetches the log's get-sth, checks the tree head's signature with
the same key, asks get-proof-by-hash for the entry in that tree and
verifies the audit path to the signed root (ct.inclusion.recorded): an
entry not yet merged stays pending; a tree head the log signed more than 24
hours after the SCT that still lacks the entry records it not included; a
proof or tree head that does not verify records a failed proof. The 24
hours is the maximum merge delay RFC 6962 logs publish; a log that publishes
a longer one is not read from its metadata. Logs that serve only the static
CT API (tiles instead of get-sth) are submitted to and their SCTs
verified, but their inclusion is recorded not checked; proving it from
tiles is what comes next. GET /api/v1/certificates/{id} and the
certificate detail show each log's answer.
Single sign-on
trstctl's interactive sign-on is served for OIDC, SAML 2.0, and LDAP / Active
Directory. OIDC is core; tenant SAML and LDAP require Enterprise SSO.
OIDC supports the authorization-code flow against Microsoft Entra ID /
Azure AD, Okta, Ping, Google, Auth0, Keycloak, and similar providers. SAML serves a Service Provider with
SP-initiated login (/auth/saml/login), IdP-initiated login through the ACS
(/auth/saml/acs), and SP metadata (/auth/saml/metadata). SAML assertion
verification requires configured IdP metadata and accepts signed HTTP-POST binding
responses, and the HTTP-Artifact binding (auth.saml.response_binding: artifact,
also provider.saml.response_binding): the browser then carries only an opaque
artifact to the ACS and the SP resolves it over a back-channel SOAP call to the
IdP's artifact resolution service, dialed under the same endpoint rule as every
configured URL (https, or plain http on a loopback host), so the signed response
never transits the browser; an artifact arriving at an SP that did not ask for
the binding is refused, and an IdP whose metadata declares no SOAP artifact
resolution service fails startup. Encrypted assertions are served
(auth.saml.sp_key_file, provider.saml.sp_key_file): the SP keeps an RSA
identity sealed at rest under the deployment KEK (generated on first start,
shared by replicas), advertises its certificate in the SP metadata with
use="encryption", and decrypts EncryptedAssertion elements inside the
cryptography boundary; an IdP that honors the metadata then never sends the
assertion in the clear. Without the file, assertions are accepted in plaintext
as before. Single logout is served (auth.saml.single_logout with
auth.saml.slo_url): a sign-out from trstctl revokes the local session and
then hands the browser a signed LogoutRequest for the IdP's SingleLogoutService
(the console follows the logout_url that POST /auth/logout returns); the
IdP's LogoutResponse at /auth/saml/slo must answer that browser's own
request. An IdP-initiated LogoutRequest at the same route is verified against
the IdP metadata (signature, destination, issuer, freshness), every session of
that NameID in the tenant the SAML mapping resolves is revoked, and the IdP
receives a Success LogoutResponse; a NameID the mapping cannot place in a
tenant signs nothing out and is refused, and a tenant SAML login that
overrides the subject with subject_attribute is matched by that subject, so
an IdP that names only the NameID then ends nothing. The provider plane's SAML
sign-in has no single logout yet. LDAP / Active Directory serves username/password bind at
POST /auth/ldap/login, supports direct-bind or service-account user search plus
group search, and maps directory groups to tenant roles. Password-change flows
are served (auth.ldap.password_change.enabled): POST /auth/ldap/password
proves the current password by binding as the user, then changes it through
the LDAP Password Modify extended operation (mode exop, the OpenLDAP /
389-ds / FreeIPA path) or, for Active Directory (mode active_directory,
ldaps:// required), by replacing unicodePwd as the user; the directory's
own refusal of a new password (history, complexity, minimum age) comes back
as HTTP 422 with its reason, and a wrong current password changes nothing. A
sign-in whose password the directory reports expired or "must change at next
logon" (OpenLDAP ppolicy passwordExpired / changeAfterReset, Active
Directory data 773 / data 532) is answered with
problem.auth.password_change_required instead of a session, and the
console opens the change form on it. Nested groups map roles when
auth.ldap.nested_groups.enabled is on: mode recursive walks parent groups
through group_filter ({group_dn}, default (member={group_dn})) up to
max_depth (default 5, ceiling 20) with a seen-set so a membership cycle
cannot loop, and mode in_chain asks Active Directory for the transitive set
in one search (LDAP_MATCHING_RULE_IN_CHAIN); a mapping on a grandparent
group then signs the user in. Directory writeback is served
(auth.ldap.writeback): a role change or offboarding an operator makes in
trstctl to a member whose subject is a DN under base_dn is queued in the
same transaction (outbox, AN-6) and applied by the licensed worker through
the service account to the directory's role groups (role_groups maps a
trstctl role to a group DN; the member DN is added to the groups its roles
map to and removed from the mapped groups they no longer cover, idempotently),
each application recorded as ldap.writeback.recorded with the groups added
and removed; a directory refusal (missing group, insufficient access, policy)
is recorded as refused and not retried, a transport failure is retried, and a
member outside the base is never written back. API/CI access still uses
scoped API tokens.
Kerberos/GSSAPI browser login is served (auth.kerberos, Enterprise SSO):
GET /auth/kerberos/login answers with a Negotiate challenge, the SPNEGO
ticket the browser presents is verified against the service keytab
(keytab_file, optionally pinned to service_principal and realm) inside
the cryptography boundary, and the principal (user@REALM) maps to a tenant
through tenant_mappings by subject, or by Active Directory PAC group SID
when pac_groups is on (leave it off for an MIT KDC, whose PAC carries no
logon info); a ticket under another key, an expired or replayed one,
or a principal that maps to no tenant is refused. The console offers it as
"Continue with Kerberos"; browsers negotiate only for hosts they trust for
integrated authentication, so the console host must be in that zone. Built
and proven against a local MIT KDC. NTLM browser login is served the way
non-Windows servers serve it (auth.ntlm, Enterprise SSO): trstctl holds no
NT hashes and validates no NTLM response itself; GET /auth/ntlm/login runs
the NTLM challenge sequence over the client's connection and relays each
message to a domain-joined helper speaking the squid-2.5-ntlmssp line protocol
(Samba's ntlm_auth, reached as a local helper_command or over a
helper_address socket), and only an account the helper reports as
authenticated (DOMAIN\user, normalized to user@DOMAIN, optionally pinned to
domain) maps to a tenant through tenant_mappings. A handshake is bound to
the connection that started it, expires after thirty seconds, and a refused or
completed one cannot be replayed. Built and proven against a Samba Active
Directory domain controller; a Windows forest is the remaining proof.
CA key custody
The assembled issuing CA's key is persisted, sealed at rest in the signer's
key store: a signer restart preserves the CA instead of silently rotating
it, and the key survives across restarts. Root/intermediate m-of-n
ceremonies and signer-backed leaf issuance are served. The release also
publishes a cgo-enabled HSM signer artifact that links the native PKCS#11
binding; the default static control-plane artifact does not load native
modules. The conditional managed-key path has required launched-binary
receipts for all six advertised providers, while the sealed local signer
key store remains the default when no provider is configured. Helm
externalKMS separately wraps signer key-store DEKs through the shipped
trstctl-kms-wrap adapter (AWS KMS, Cloud KMS, Azure Key Vault or Managed HSM,
and, in the HSM signer image, PKCS#11) or an operator's own adapter instead of
mounting the local signer KEK. A key store the local KEK already sealed is not
re-wrapped onto the adapter; external custody starts from an empty key store.
The chart runs the standard image, so a PKCS#11 HSM behind the adapter needs the
HSM signer image outside the chart today. Online break-glass issue,
signer-backed CA rotation with bidirectional overlap cross-certificates,
and target-CA cross-signing are production-assembled when the online
block is configured. Approvals come from authenticated immutable ceremony
events, not caller-supplied names. Break-glass bundle reconciliation
remains served at POST /api/v1/breakglass/reconcile. The credential-store
key-encryption key is a local file by default; a served rotation moves it to a
key under an operator wrapper file or an HSM key the signer holds (see the
deployment KEK passage below). See the
key-ceremony runbook,
incident response, and
disaster recovery.
In-memory custody of the reference-path CA keys: the served CA-hierarchy
path does not use these in-process reference keys — it binds each served
root/intermediate to an isolated-signer handle. The library reference
manager still holds live ECDSA signing keys in locked, wipeable secret
buffers (mlock + MADV_DONTDUMP) rather than as a bare unprotected key
on the garbage-collected heap for the lifetime of the in-process CA; the
key is reconstructed only for the instant of each signature and the
transiently parsed copy is best-effort zeroized afterward (the same
hardening the isolated signer uses). This narrows — but, given Go's
runtime, does not eliminate — the window in which an unprotected key sits
in dumpable heap; it is complemented process-wide by RLIMIT_CORE=0 /
PR_SET_DUMPABLE=0.
BYOK / HSM key lifecycle: the conditional Enterprise surface supports AWS KMS, Azure Key Vault / Managed HSM, GCP Cloud KMS, PKCS#11, TPM 2.0, and YubiHSM 2. The required wiring census reports six of six backends served through the shipped control-plane plus cgo HSM signer artifact; package reachability or a registry built only by a test is not used as evidence.
With an active BYOK license and managed_keys.enabled, the control-plane process
does not construct a provider or receive provider credentials. Its tagged EE attach
seam installs the tenant-scoped event/projection factory and durable PostgreSQL
outbox handler. A separate dispatcher delivers the lifecycle command over the
authenticated signer transport. The isolated signer constructs every listed
provider under the ee/ fence and stores provider ownership, operation outcome, and
consumed sign-authorization nonces in its fsync-backed journal. An executing operation
is not abandoned after restart: every shipped receiver finds or reconciles the same
durable operation identity. AWS stamps it atomically in CreateKey; Azure and GCP use
deterministic provider resource names; PKCS#11 and YubiHSM use deterministic CKA_ID;
TPM enumerates persistent objects and accepts only a full SHA-256 operation tag from
immutable TPM Public.AuthPolicy, probing past foreign handles without adopting or
overwriting them. Revoke and zeroize read provider/device state before and after the
accepted transition, so a lost response does not repeat the effect. For cloud KMS,
the product reports deletion_pending while the provider may retain material;
the local emulator and current QA window cannot prove the later provider retention
deadline or physical erasure. A stock provider client must check that deadline,
and a later provider readback is needed to close the destruction claim. Historical
cloud zeroized events remain immutable and appear as deletion_pending in the
API and console. The swtpm gate also
pre-occupies the first deterministic handle with a same-algorithm foreign object and
proves that object remains untouched.
The handlers at POST /api/v1/managed-keys, its dedicated JSON approval route, and
its verify-custody, rotate, revoke, and zeroize
companions return only opaque handles, public DER, algorithm, non-extractable state,
and lifecycle state. Verification uses a fresh signer-generated challenge signature
and records its last result and time independently of the lifecycle. This is a
point-in-time proof, not a promise the provider will retain the key afterward.
The default outbox retry budget can leave custody pending after the API's
short synchronous wait returns 202 Accepted with an operation ID and status
URL. Poll GET /api/v1/managed-keys/operations/{operation_id} or watch the
console's operation panel until the durable outcome is completed or failed;
refresh the exact key to read verified or unavailable. A caller whose own
deadline expires before an accepted receipt arrives should read inventory and
audit before retrying with the same idempotency key. New terminal failure audit
events carry the key ID when one existed before the action; failed generation
has no minted key ID. Historical v1 failures carry only the operation ID.
Every mutation requires Idempotency-Key; immutable events
build the tenant/RLS projection, and the provider call comes only from the sealed
outbox. Managed-key signing additionally requires a short-lived, request-bound token
from the configured content-authority command. The signer consumes its random nonce
durably before calling the provider, so token replay fails both in-process and after
restart. Provider credentials must be file-backed, are copied into locked byte
buffers, and are wiped on shutdown; the one unavoidable PKCS#11 C_Login conversion
exists only at the upstream string-only ABI edge.
| Provider | Production binding in the shipped HSM signer | Gate substrate and lifecycle proof |
|---|---|---|
| AWS KMS | Official AWS SDK v2 asymmetric KMS client | SigV4-checking emulator; atomic operation tag, ambiguous-create recovery, disable/deletion readback |
| Azure Key Vault / Managed HSM | Keys data-plane client with bearer-token file | Managed-HSM emulator; deterministic key identity, ambiguous-create recovery, revoke/delete readback |
| GCP Cloud KMS | Cloud KMS REST data plane with bearer-token file | Deterministic-resource emulator; ambiguous-create recovery, disable/destroy readback |
| PKCS#11 | cgo module session with operation-derived CKA_ID handles |
SoftHSM token; restart find-or-create plus independent pkcs11-tool state readback |
| TPM 2.0 | google/go-tpm device or swtpm socket with tagged persistent handles |
swtpm plus tpm2-tools; full operation-tag readback, forced foreign-handle collision, restart and eviction |
| YubiHSM 2 | Yubico yubihsm_pkcs11 ABI through the PKCS#11 connector |
Vendor-ABI emulator with deterministic CKA_ID; independent sign/revoke/deletion readback |
The cloud receipts are high-fidelity protocol emulation, not a claim that this test
ran in a customer's live cloud account. Likewise, SoftHSM proves the PKCS#11 ABI and
swtpm proves TPM command/lifecycle behavior; an operator still validates its exact
device firmware, module certificate, network policy, and cloud IAM. The LocalStack
demo is not the acceptance proof's receipt source. Its pinned Community 3.8.1
image keeps KMS keys in memory; the named volume and PERSISTENCE=1 do not make
them durable. A key may remain in trstctl's event-projected inventory after this
emulator loses the external key. Cold recovery of the AWS KMS demo lane is blocked
on a durable, faithful KMS service (for example, a licensed LocalStack edition or
a real isolated cloud account). SoftHSM and swtpm can prove their own provider
recovery, but cannot stand in for native AWS KMS. Static no-cgo signer builds fail
closed for native PKCS#11/YubiHSM selection; use the published HSM signer artifact. Several
providers run at once (managed_keys.providers): a key's provider is chosen per key
by the generate request or per tenant by a managed_keys.tenant_providers pin, and
every key reports its provider. The provider list is startup-static: this is ordinary
Go interface injection, not a runtime crypto plugin engine or a policy-selected
algorithm marketplace.
The managed-key runtime proof qualifies each released platform on a native
runner of that platform. make dod-gate proves the linux/amd64 control plane
and cgo HSM signer on an amd64 runner. make dod-gate-arm64 proves the
linux/arm64 builds of the same two release artifacts on a native arm64 runner,
against the same custody substrate and the same seven managed-key entries
(tools/dodcensus/manifest-arm64.json, which a census test keeps derived from
the shipped manifest); CI runs it on GitHub's ubuntu-24.04-arm runner. The
arm64 target refuses a Docker host that is not arm64: an emulated runner cannot
qualify either platform's artifact, even when its SoftHSM and swtpm component
tests pass. Neither a component test nor a locally rebuilt signer is a substitute
for that production-assembly proof. The rest of the wiring census is proved on
linux/amd64; extending the arm64 profile beyond managed-key custody is what
comes next.
The signer-held CA/issuing key's lifecycle is served through ceremony-gated
verbs in the isolated signer: generate (root, intermediate, or a key and CSR
for an external issuer), rotate and re-key, revoke (POST /api/v1/ca/authorities/{id}/revoke after a revoke_ca ceremony: the signer
refuses every signature with the key, the authority stops issuing, and its
certificate is revoked on a managed parent's CRL), import (an import_ca_key
ceremony takes a CA's existing private key into the signer, sealed on the
operator's machine to a one-time HPKE MLKEM768-P256 import key, so the control
plane relays only sealed bytes), and destroy through signed retirement. An
imported key may still exist where it came from; trstctl cannot prove other
copies were destroyed. Each tenant's secrets KEK, its tenant key domain, has
served lifecycle verbs too: generate (migrate into the domain), rotate (POST /api/v1/platform/tenant-key-domain/rotate moves the domain to a new generation
under the same or another configured wrapper, re-wrapping every stored data key
and the tenant's event history), and a reversible revoke as seal and unseal.
Rotation re-wraps data keys, not payloads, so backups, archives, and exports
taken before it still open with the previous generation. Destroying a tenant's
key (crypto-shred) and importing an operator-supplied tenant KEK are what comes
next. The deployment KEK, which protects every tenant that has not migrated and
the deployment's sealed files, rotates too (POST /api/v1/platform/deployment-kek/rotate, only by the tenant
secrets.kek_operator_tenant_id names): a new generation under a local wrapper
file or a signer-held HSM key, from which every new data key is wrapped, then a
leader re-wraps every stored data key (each tenant's stored ciphertext, the
history of every tenant that has not migrated, and the sealed files) in bounded
steps while requests are served, and a verify pass reads them all back before
the previous generation is retired. The history pass copies the event stream
once per tenant it re-wraps, so a deployment with many unmigrated tenants and a
long history takes correspondingly long; it inherits the tenant rotation's
refusal of a certificate-dependent history record, which stops the rotation for
an explicit migration. The key file remains the evidence-MAC and backup-HMAC
key and the signer key store's default seal. Wrapping the deployment KEK under
a cloud KMS key is what comes next: the signer's KMS wrap adapter now seals the
signer key store, but the deployment KEK's wrappers are still a local file or a
PKCS#11 HSM key (proven against SoftHSM). Break-glass
issue/rotation/cross-sign and offline-root public re-key/cross-sign import
have served, ceremony-gated verbs; reconciliation remains served at
POST /api/v1/breakglass/reconcile. The signer's at-rest CA key is still
sealed under a local key-encryption file by default. See the
key-ceremony runbook,
incident response, and
disaster recovery. The remaining external residual
is the product NIST CMVP certificate (see
compliance → FIPS),
a lab process software cannot perform. The validated-module path itself is
served: GET /api/v1/editions and the Platform page expose the live FIPS
POST booleans, make fips-build build target, fips-capable build (GOFIPS140) CI gate, and internal/crypto boundary as the CAP-KEY-03
operator posture.
Signer UDS peer-uid is Linux-only: the signing service's Unix-domain-socket
listener authenticates the connecting process's uid via SO_PEERCRED,
which exists only on Linux — the supported production target
(Docker/Helm). On non-Linux hosts, trstctl-signer fails closed when
process hardening, locked memory, or UDS peer credentials are unavailable.
Local developers
can opt into the filesystem-permissions-only fallback with the explicit
--allow-insecure-dev-nonlinux flag (or
TRSTCTL_SIGNER_ALLOW_INSECURE_DEV_NONLINUX=true for child signer mode), but this is
not a production control. Production deployments without reliable UDS peer
credentials should use the signer's fail-closed mTLS transport with pinned peer
certificates.
Post-quantum cryptography (issuance algorithms)
trstctl's cryptography sits behind one isolated path, and the post-quantum support lives there — ML-DSA, ML-KEM, the hybrid scheme, and SLH-DSA — all built on Cloudflare's CIRCL. What is available today:
- ML-DSA (FIPS 204;
mldsa44/mldsa65/mldsa87) — the NIST-standard lattice signature. - ML-KEM (FIPS 203;
mlkem512/768/1024) — the NIST-standard key encapsulation. trstctl can generate ML-KEM keys, encapsulate to an ML-KEM public key, and decapsulate the resulting ciphertext; all three parameter sets are checked against FIPS 203 known-answer vectors. The served HTTPS and mTLS listeners preferX25519MLKEM768for TLS 1.3 hybrid key exchange when a peer supports it, with classical TLS 1.3 groups retained for compatibility (restored in B13-8: from July until then the listeners offered only classical groups, and stock OpenSSL offering only the hybrid group was refused). Pure ML-KEM groups are not offered: Go's TLS stack implements the hybrid group. With stock OpenSSL 4.0.3, a client offering onlyX25519MLKEM768negotiates it on the HTTPS API and on the agent mTLS channel, a classicalX25519offer still connects, and a client offering only pureMLKEM768is refused for want of a shared group. - SLH-DSA / SPHINCS+ (FIPS 205;
SLH-DSA-SHA2-128s/128f/192s/256s) — the NIST-standard stateless hash-based signature. Its security rests only on the hash function, so it is the conservative choice for long-lived roots where you want assumptions independent of the lattice schemes; the trade-off is much larger signatures. Here it is a signer and algorithm-succession primitive: trstctl issues no SLH-DSA X.509 certificate and accepts no SLH-DSA CSR (the ML-DSA certificate profile of RFC 9881 is served; the SLH-DSA profile of RFC 9909 is what comes next), so stock-client SLH-DSA interoperation is not claimed even though stock OpenSSL 4 ships the algorithm. A certificate profile may list anSLH-DSA-SHA2-*label among its allowed key algorithms, but no enrollment path accepts an SLH-DSA subject key yet. - A hybrid signature (
HybridEd25519Dilithium3) — classical Ed25519 paired with ML-DSA, so breaking either component alone does not forge a signature.
Private key material is held in locked, zeroized buffers and parsed only for the moment of each operation, exactly like classical keys. The isolated signer can generate and use signer-held ML-DSA and SLH-DSA keys over its UDS or mTLS gRPC channel, and those keys are sealed in the signer key store so a restart does not silently rotate them. ML-KEM is not exposed as a signer key because it is encapsulation, not a signature; use it as the key-establishment primitive for protocol wiring rather than as an issuing CA key.
The served CA can mint a hybrid transition leaf: the certificate remains a
normal ECDSA P-256 leaf for stock TLS clients, while a signed ML-DSA-44 +
ECDSA-P256 composite binding is carried inside the certificate for
PQ-aware verifiers — deployable without forcing every client to understand
draft composite public keys on day one. The ACME, EST, SCEP, and CMP
served enrollment paths all run through that same issuer, and a CSR
carrying the hybrid proof (a classical ECDSA-P256 CSR with the
composite-binding extension) issues through all four. Pure ML-DSA CSRs are
narrower today: EST accepts and issues them (the PQC PKCS#10 parser
sits behind EST's verifier seam, proven against a stock OpenSSL 3.5
client), ACME hands the CSR bytes to the same PQC issuer without
parsing them first, and CMP p10cr carries them under a classically
protected PKIMessage: the carried CSR is verified and its names are
inspected through the same PQC-aware seam, so the binding between the
CSR and the protection identity is evaluated on the parsed names rather
than by re-parsing the CSR with the core parser (proven with a stock
openssl cmp client carrying an openssl req ML-DSA-65 request). SCEP
verifies and inspects the enveloped CSR through the same seam, so a pure
ML-DSA PKCS#10 enrolls when the client's CMS transport identity (the
self-signed certificate that signs the pkiMessage and receives the
CertRep envelope) is RSA; that transport key is separate from the subject
key in the CSR, and SCEP still cannot deliver its CMS-enveloped reply to
a signature-only transport key, a protocol limit rather than a code gap.
Stock SCEP clients (sscep) do not produce ML-DSA requests, so the SCEP
proof is a Go-built pkiMessage against the served parser, not a stock
client. The
platform (broker) issuing CA key remains classical ECDSA-P256; a signer-backed
hierarchy authority can be post-quantum: signature_algorithm ml-dsa-44,
ml-dsa-65 or ml-dsa-87 on a root or intermediate ceremony mints that key in
the isolated signer (HASH_NONE carries the TBS for the pure signature), the
authority issues classical and pure ML-DSA leaves under RFC 9881, signs its
CRLs and OCSP responses with the same key, and OpenSSL 3.5+ verifies all three;
a ceremony reviewed for ecdsa-p256 cannot mint it, and a re-key stays in the
predecessor's key family. Certificate-profile allowed_key_algorithms labels accept
the post-quantum and hybrid names in every build — a pure label
matches its exact inspected CSR algorithm, while a hybrid enrollment
carries a classical subject key and stays governed by its classical family
label. The selected profile still controls which algorithms an enrollment may use.
PQC certificate-key migration limits: New host-bound runs require an exact
CBOM leaf fingerprint, a requested identity whose subject_key_algorithm is
the run's target_algorithm (ML-DSA-44, ML-DSA-65 or ML-DSA-87; the
console offers the choice and defaults to ML-DSA-65), an active profile that
allows that subject key and DNS name, a signer-backed platform or managed
private CA, and an enabled host connector with an enrolled agent whose
installed certificate and key the agent can read back exactly. The
exact-predecessor reader covers Apache, NGINX, Caddy, Elasticsearch, MySQL,
PostgreSQL, RabbitMQ, Tomcat and Traefik targets (separate certificate and key
files), HAProxy (the certificate+key bundle at crt_path, which must hold
exactly one private key) and Postfix/Dovecot (both pairs must serve the same
leaf, since one predecessor is restored to both). Java keystores (the store
password is not available at adoption time), IIS (the key lives in the Windows
store) and Envoy (the key sits behind SDS) still have no exact predecessor
reader and are refused for host-bound certificate migration and CA
re-protection.
A native OpenSSL 3.5 probe and a server capable of serving ML-DSA certificates
are required for signed TLS readback; classical-only clients cannot be
presented as PQC verification. Under the platform CA the leaf has an ML-DSA
subject key over a classical chain; a requested identity that pins an ML-DSA
hierarchy authority (issuing_authority_source: private,
issuing_authority_id) gets a fully post-quantum chain. Historical issuance-only runs have no retained host key or
predecessor and still report rollback_unverified. A profile requiring
distinct issuance approval no longer refuses this path: the start is the
approved operation. The first POST /api/v1/pqc/migrations records an
immutable approval request bound to the canonical selection, the target
parameter set and every gated reissue's frozen profile revision and lifetime,
and answers 202 with status: pending_approval, the request id and its
intent digest; reviewers holding certs:issue approve it at
/api/v1/approval-requests/{id}/approvals, and the same start posted again
under a new Idempotency-Key consumes that single-use authority in the
transaction that records the run. A changed selection or profile edit is a new
request; a consumed, denied, expired or superseded one answers 409.
TLS posture rollout is a separate bound-target action.
The Envoy PQC posture worker queues an agent job through the outbox to one
pinned agent: the exact enrolled host assigned to a co-resident Envoy, or, for
a remote Envoy that declares vantage: network, the network relay its
required_agent_id names (A104). That agent stores the observed predecessor
encrypted before a native Envoy change, then signs a report that binds the
active configuration, its own TLS handshake with the listener (loopback on a
host, from the relay's segment for a remote Envoy), the leaf fingerprint, and
its identity. A network target that names no relay, or names an agent without
the network role, is refused before anything is queued. Rollback requests
return to that agent and restore its exact predecessor; the relay must run with
--host-rollback-dir in persistent storage. The local partner-lab Envoy xDS fixture is documented in
deploy/demo/lab/README.md. Previously queued pre-agent intents still refuse
in-process execution, and posture changed by an older build has no host-retained
predecessor to restore. These local fixtures do not qualify an external
appliance or a multi-host deployment.
The core CBOM posture recognizes ML-DSA, ML-KEM, and SLH-DSA / SPHINCS+
(and hybrid labels) when it finds them in your estate. Algorithm recognition
and posture classification attach in every build without a commercial license. Because all cryptography enters through one isolated
path, each scheme is a contained boundary implementation (a CIRCL scheme
plus known-answer tests), with no ripple into the rest of the system. The
served CBOM inventory exposes this posture through
GET /api/v1/cbom/assets: classical signing algorithms
are mapped to ML-DSA-65/FIPS 204 targets, key-establishment findings (TLS
protocols and ciphers) to ML-KEM-768/FIPS 203, deprecated DSA to
SLH-DSA/FIPS 205, and migration_progress shows how much of the observed
estate is already post-quantum-ready — pure post-quantum assets count as
future-ready, while hybrids stay migration-required until they shed their
classical component.
The core attach serves two of the former end-to-end residuals in every build: a stock OpenSSL 3.5 client creates an RFC 9881 ML-DSA-65 CSR, enrolls it through EST, and verifies the returned pure ML-DSA-65 subject leaf; the stock SPIFFE Workload API returns a two-entry response for one SPIFFE ID (the normal classical SVID and an ML-DSA-65 SVID with its matching private key). The shipped-binary CBOM TLS finding rollout against local Envoy was replayed through the host agent and outbox: stock OpenSSL observed the served TLS posture before migration, after migration, and after rollback to the exact predecessor; a signed agent readback recorded each applied state. This proves the local Envoy target, not an external appliance or every TLS client. Older runs created before the host-agent path existed remain historical evidence, not rollout proof.
Those proofs define the compatibility boundary: they do not claim every
legacy TLS client or every connector understands ML-DSA. A hybrid-to-pure
cutover for an existing hybrid certificate is served behind the relying-party
acknowledgment gate: a migration run that targets pure ML-DSA for a
certificate whose current leaf is the hybrid composite plans that reissue as
a cutover and holds its host job (finding status awaiting_acknowledgment)
until a quorum of signed relying-party acknowledgments for the identity's PCAS
retirement policy is recorded inside the policy's window. That is the PCAS-10
rule that gates retiring a predecessor key, and a policy PCAS has already
retired on that quorum still counts. A leader worker then queues the held job
unchanged and records pqc.migration.cutover_released with the acknowledging
relying parties and the ack-set digest; the host generates the ML-DSA key and
returns only its CSR, and the finding reaches applied only on the host's
signed readback. Without a policy for the identity, or without the quorum,
the cutover waits and the served leaf stays hybrid. Direct pure ML-DSA
enrollment is served through EST (and as the SPIFFE Workload API's second
SVID), and
CMP consults the same PQC parser for its carried CSR (PKIMessage
protection stays classically verified). SCEP cannot by protocol. The direct
identity API now accepts a caller-supplied CSR — supply subject_csr_pem
on the transition to issued and trstctl signs that request rather than
generating a subject key — so CSR-based enrollment, including post-quantum
subject algorithms, is no longer confined to the enrollment protocols. See
Lifecycle & PQC for operator flow and
compatibility limits.
Key custody, stated per credential kind. Whose process created a private
key and whose disk holds it is answered in one CI-checked table at
Key custody, not in prose scattered across pages. The short
version: every enrollment protocol, and the identity API when given a CSR,
generate keys in your environment and the control plane never sees them; CA
keys are created inside the isolated signer and never leave it; and every
remaining path that can generate a subject key in the control plane is named there
with what replaces it. The identity API without a CSR is one of those three:
it still works for one release train and records an
issuance.server_side_keygen event every time it runs, so you can find which
of your flows still rely on it. Each tenant can turn it off with
PUT /api/v1/tenant/key-generation, and a tenant registered since that switch
starts with it off: an issuance that would generate the key in the control plane
is then refused before anything is recorded, naming a CSR or an agent-executed
target. The switch governs PKI-as-a-secret by common name (and the
Vault-compatible pki/issue) the same way, naming the CSR route, and the
compatibility Workload API socket's X.509-SVIDs, naming the host agent's socket;
see Key custody. An identity issued from your own CSR cannot
be deployed by a control-plane connector — the key that deployment needs is
on your side, which is the correct consequence. Its renewal honors the same
CSR, and an endpoint whose target is agent-executed renews on its host
(A118, below).
And stated per credential, not only per kind. The table above is the right level for a design review and the wrong level for an audit, because an auditor is not asking about a kind — they are asking about the certificate in front of them, and a kind-level table cannot tell them whether that one took the modern path or the deprecated one. So custody is also recorded on each certificate at issuance, from what the issuing code did rather than from what the table says it should: an identity issued from your CSR records that the control plane never held the key; one issued through the deprecated server-keygen path records that it did. The certificate API returns it and the console shows it on the certificate.
Two honest gaps. Certificates issued before this shipped have no custody
recorded, and so does every certificate found by discovery — trstctl did not
witness their issuance and has no basis for a claim about it. Both read as
not recorded, which is deliberately a different value from any custody
claim rather than a default that quietly resembles the good one. And what is
recorded is the control plane's own account of what it did; it is evidence, not
an attestation. A certificate issued through ACME device-attest-01 can also
carry a claim the hardware signed: when the device's TPM certifies its key as
one it generated and will not release (sensitiveDataOrigin, fixedTPM,
fixedParent), the certificate records custody device / device_bound /
non_exportable and keeps the attestation and its binding, and anyone can
re-check it offline against TPM roots they choose (trstctl-cli certificates key-attestation then verify-key-attestation; see Key custody).
It is proven against fixture attestations and swtpm's TPM2_Certify; proof on a
hardware TPM needs one. Host agents that generate endpoint keys in their own TPM,
so agent-executed targets carry the same claim, are what comes next.
Recording it per certificate also made a fourth control-plane keygen path
visible that the kind-level table had not named: automated renewal of an
identity with no CSR, on a target the control plane deploys to, builds the
successor's CSR itself, so that successor reads control_plane. The key is not
lost (A119): it is retained in a tenant-bound sealed recovery record
(key_storage sealed_store) and travels sealed in the connector's deploy
intent, so the endpoint serves the renewed certificate; an identity with no
deployment target has no endpoint to serve it, and its key stays sealed.
Host-executed renewal (A118, B2) removes the control-plane key for an
agent-executed target: the renew-before-expiry pass and rotate queue
endpoint.renew before anything is minted, the host agent generates the key,
sends only a CSR, installs the pair, verifies the endpoint, and signs the
custody receipt (key_origin host_agent, the agent named as generator). The
batch-12 lab replay ran five such renewals on its host agent, each recorded as
host_agent/file/lab-host-1. Key custody states this in full;
the count of successors reading control_plane measures what still relies on
the deprecated fallback.
Served TLS floor and stock enrollment clients
The control plane's HTTPS listener negotiates TLS 1.3 only by default. Stock device
enrollment clients that cap at TLS 1.2 (cisco libest estclient, some SCEP agents)
cannot complete the handshake against that default. The libest conformance job
proves both sides of that policy against the control plane's own listener code
(mtls.ServerCert.ServeHTTPS, the path serveControlPlane runs): with the TLS
1.3-only default the pinned estclient fails its handshake and writes no
certificate, and with TRSTCTL_SERVER_TLS_MIN_VERSION=1.2 (AEAD suites only), the
documented opt-in for such fleets, the same client enrolls. The earlier proof
against a Go test server that silently allowed TLS 1.2 is retired. EST
/simplereenroll on the main listener still authenticates with the bootstrap
token; with TRSTCTL_PROTOCOLS_EST_MTLS_ADDR set, a second listener requires a
client certificate from the enrollment CA and authenticates renewals by that
certificate, binding the CSR to its identifiers, so devices keep no token for
renewals (DP2-037 closed; proven with a Go client and with libest estclient -r).
Found by the cold design-partner run 20260906t143500z (DP2-036, DP2-037).
ACME certificate profiles
Both directions of draft-ietf-acme-profiles are served. trstctl's ACME client (the
letsencrypt external CA type) requests the profile named by the authority's
acme_profile configuration on every newOrder, and only when the authority's
directory advertises it in meta.profiles: an unadvertised or misspelled name
fails before an order exists. With no acme_profile the authority applies its
own default, as before. trstctl's ACME server advertises its offered profiles in
meta.profiles (the mount's default plus every credential-pinned profile), accepts
a client's profile member when it names an advertised profile, and refuses an
unknown one with invalidProfile; a protocols.acme_eab.keys[].profile pins the
profile every order under that credential is issued with, and a client asking for
a different one is refused. The selected profile is recorded on the order and the
served issuance signs under it. A live profile request against Let's Encrypt's
production directory is not part of local evidence.
Upstream domain validation
trstctl is an ACME server and an ACME client, and until now only one of those could do DNS-01.
What was wrong. As a client to a public CA, the driver looked only for
http-01 and errored if the authority offered none, so DNS-01 was never
attempted and wildcards were impossible. Worse, the production constructor wired
a solver whose present and cleanup did nothing. The upstream path therefore
worked only against orders the authority had already authorized out of band.
Every test passed, because the fixture returned orders as pre-authorized — the
same shape of defect as a feature whose executor is never asked for work.
What is served now. The client negotiates challenge type from what the authority actually offers, and solves DNS-01 through the same publish path, the same providers and the same credentials the server direction already uses. A missing solver fails closed with a named reason rather than silently validating nothing. The record is retracted on every exit path, including when the context expired — which is exactly when a validation token is most likely to be left live in public DNS.
Consent is explicit, and it is two decisions, not one. The authority must
be configured with upstream_dns01, and each DNS-01 provider config must set
allow_upstream_dv. Both default to false, including on configs that already
existed. Credentials an operator supplied so
that trstctl could verify a challenge somebody else published are not consent
for trstctl to publish into that zone whenever an external authority asks —
and as validation-reuse windows compress, that publishing becomes frequent and
unattended. A migration cannot grant that permission; a content harness asserts
no existing config comes out consenting.
CAA is checked against the right issuer. The server-side check uses the
config's own CAA identifier, which is correct when trstctl is the issuer.
Upstream the certificate comes from someone else, so the check uses the external
CA's identifier, and refuses to run at all if none is configured — a CAA check
against an empty issuer authorizes everything while appearing to check.
Configuration validation requires caa_issuer_domain whenever upstream_dns01
is on, so that state is unreachable rather than merely handled. The identifier
is bound per authority: two configured ACME CAs do not share one, because a CAA
check that authorizes the wrong CA passes while being wrong, which is harder
to notice than one that authorizes everyone.
Reused authorizations are recorded, and served. An authority that already
considers an identifier authorized issues without a challenge. That is normal
and it is also the thing worth watching: automating validation removes the
human from the cycle, and with them the human who used to notice when
validation broke. GET /api/v1/acme/dns-01/upstream-authorizations and the
console's Upstream authorization freshness panel report, per identifier and
per authority, when control was last actually proved — not when a
certificate was last issued. Those two diverge silently, and the gap is the
warning. An identifier that has never been validated by this deployment is
called out by name: every issuance for it so far rode a reuse this install did
not earn and cannot repeat, and when the window closes they fail together
rather than one at a time. A reuse never overwrites the last real validation
date; that is asserted by a test, because the convenient single "last seen"
column would erase the only signal here.
What is not served. trstctl does not wait for DNS propagation before telling
the authority to validate. The record is published through the provider's API
and the challenge is accepted as soon as that call returns, so a zone whose
nameservers are slow to converge can have its authorization marked invalid and
the order retried rather than waiting. The operator-run preflight
(POST /api/v1/acme/dns-01/preflight) checks propagation for a domain, but it
evaluates TXT values the caller supplies rather than querying DNS itself, so it
cannot be reused as an automatic gate. Closing this needs authoritative-nameserver
TXT verification that does not exist in the tree yet; until it does, a slow zone
costs a retry, not a wrong answer.
Also not served: http-01 upstream — it would require an inbound listener on the validated host, which this architecture does not have and will not grow. The challenge census says so rather than claiming a type that would be selected and then fail. The publish is bounded, but by whichever deadline is tighter: the caller's context if it carries one, and otherwise the DNS-01 automation's own 30-second outbox wait. There is no separate, configurable propagation budget, and the 30 seconds is a floor for callers that set no deadline rather than a cap that always applies.
Issued, delivered, verified
Three claims about the same certificate, with three different lifetimes, and only the third is what an operator actually wanted:
- Issued — a CA produced it. Recorded in the certificate inventory.
- Delivered — a connector applied it to a target. Recorded on a
connector.deploydelivery receipt. This is trstctl's own account of what it did, and it is where every deployment surface stopped before D3. - Verified — a TLS handshake observed the endpoint serving it. Recorded on a second delivery receipt and, separately, as current endpoint state.
Delivered and verified are counted separately and never summed. A renewal can
succeed at the CA, be delivered by a connector, and never reach the listener,
with every delivery record staying truthfully green. A health surface that
counted deliveries would be reporting intentions; GET /api/v1/platform/system
counts by verified state for exactly that reason, and verified_percent is of
DELIVERED targets — "of what we have deployed, how much is confirmed live".
Verification writes a second receipt rather than editing the delivery one.
Both facts belong in the evidence chain and they have different lifetimes: "a
connector applied the credential" is true forever once it happens, and "the
endpoint was serving it" is true of the moment it was observed. A certificate
verified in June whose listener silently reverted in August shows a June receipt
still reading verified and an endpoint state reading diverged. Overwriting
the first would destroy the record that delivery succeeded, which is what an
operator needs to tell a pipeline problem from a listener problem.
unverified is the honest middle. A delivered target nobody has probed is
neither a failure nor a pass — it means nothing has looked. Folding it into
either direction would be an overclaim, and on a fresh install every target sits
here, which is the correct starting picture rather than a discouraging one.
Renewal windows, canaries and SLOs
Maintenance windows defer, they never drop. A renewal deploys to a listener and reloads a service, and there are hours in every organization's week when nobody wants that unattended. Before this the only control was switching renewal off, which trades an outage risk for an expiry risk. A window that closes now holds the sweep and records why, naming when it reopens — because a change freeze that quietly stopped renewals looks exactly like a scheduler working correctly, right up until certificates expire, and expiry is the more expensive failure by a wide margin.
An empty window list means unrestricted, never "never": an operator who configured no windows has not asked for a freeze, and defaulting to closed would turn an upgrade into a fleet-wide expiry event. A malformed window refuses to start rather than being ignored — an operator who wrote a freeze this could not read would believe production was protected while the scheduler renewed through it. Windows are expressed in named timezones so they follow daylight saving the way the person who wrote them expects, and a window whose end precedes its start wraps midnight, which is the shape most operators actually want.
Fleet gates are computed, not asserted. Until D2/D3 nothing re-read an
endpoint, so every gate trstctl filled in itself was not_evaluated — correctly,
because a verdict nobody computed is not a pass. The replacement-deployment gate
is now derived from verification receipts under two rules an operator relies on
mid-incident: failure dominates (one replacement serving the wrong
certificate fails the gate however many others passed) and absence beats
success (one replacement nobody probed keeps it unevaluated). A gate an
operator asserted is never overwritten by a computed one — they may have
inspected something this control plane cannot see. Gates evaluate at read time,
so a fleet that has since diverged stops showing a pass.
The other two gates are computed on every read as well, and appended to a run
that never named them. Graph enumeration re-reads the live store as its
completeness oracle: every identity that still holds an active certificate
signed by the compromised CA must be in the run's affected set, so a leaf the
plan missed, or one issued under that CA after the plan, fails the gate by name
(graph_enumeration_missing_identity_ids); an oracle that cannot read keeps it
unevaluated. Revocation publication is R1's freshness signal narrowed to the
compromised issuer: a stale or unreachable CRL/OCSP endpoint fails the gate, a
fresh artifact whose thisUpdate predates the run's revocations keeps it
unevaluated (it cannot carry them yet), and the gate passes only when a fresh
or expiring observation postdates the run's last change
(revocation_publication reports the counts). No observation for the issuer
means no verdict. The watermark is the run's last state change, not each
predecessor's exact revocation instant, which is the one approximation here.
Canary-first halts propagation. Starting a run persists its cursor and
publishes exactly one internal outbox command for batch one. The bounded worker
publishes replacement work only for that cursor, waits without spending its
retry budget, and accepts a verdict only when the connector verification row is
paired with an agent-job receipt whose signature was accepted. A failure stores
the explicit halt reason and leaves later batches absent from the outbox.
halted is distinct from failed: a halted batch was never attempted. Resume
reuses the same cursor and deterministic replacement ids, so crash recovery and
redelivery converge instead of reissuing or publishing a later batch twice.
The SLO counts terminal runs only. A renewal still executing is neither a success nor a failure, and forcing it into either would move the number for reasons unrelated to reliability. A window in which nothing was due reports 100%, not 0% — an estate with no renewals pending is not in breach, and paging on the absence of work is how a team learns to ignore an SLO. Error-budget burn is clamped at zero, because a figure like -340% is not more actionable than none-remaining and the raw counts sit beside it.
What is not served. The fleet state machine does not treat an unsigned
control-plane assertion as endpoint proof. A deployment executed only by a
control-plane connector therefore cannot advance this signed-receipt gate; it
remains waiting_verification until a bound agent reports the endpoint result.
Dated lifetime-reduction readiness
GET /api/v1/certificates/validity-readiness (and trstctl certificates validity-readiness, the Lifetime-reduction readiness panel on the
Certificates renewal tab) evaluates the CA/Browser Forum TLS Baseline
Requirements 2.3.1 (published 2026-10-04), section 6.3.2: 200 days for
certificates issued from 2026-03-15, 100 from 2027-03-15, 47 from 2029-03-15.
Every active certificate gets a verdict under each step (renews, at_risk
or manual, with reasons), and every public TLS profile whose max_validity
exceeds a step's maximum, or that has none, is flagged. The export names the
BR version in its file name and on every row. What it decides from, and what
it does not see:
- Publicly trusted means a reserved policy OID. A certificate counts as
public TLS when it permits TLS server authentication and asserts a CA/Browser
Forum reserved certificate policy identifier (2.23.140.1.2.1–3 or
2.23.140.1.1). trstctl's own CA is never publicly trusted, so a profile is
public TLS only when it says so (
public_tls: true). Other certificates are still evaluated, for planning, withscopesaying the maximum does not bind them. - Scheduled means the scheduler's own inventory. A certificate is
scheduledwhen it is the served certificate of a renewable identity, or its issuance evidence binds it to one — the same set the automation plan and the scheduler use. Its verdict follows that identity's delivery evidence (a delivery whose served certificate was read back isverified), its holds (failed renewal, an operator's pause, a destination conflict, a profile renewal cannot resolve) and whether each renewal waits for approval. - Client renewals are not observed. Certificates issued over ACME, EST,
SCEP, CMP, Kubernetes in-cluster issuance, attested workloads or the agent
broker are renewed by their client; trstctl does not see that client's renewal
lead, so they read
at_riskwithclient_renewal_unobserved. Measuring a client's observed renewal lead from its issuance history is what comes next. - Flagged, not refused. Issuance does not yet hold a public TLS profile to the maximum for the issuance date; the report flags it and a publicly trusted CA refuses or shortens the request. Holding issuance to the dated maximum is what comes next.
- Bounded. The report covers the 5,000 active certificates that expire first and says when there are more; ephemeral certificates, issued per use and never renewed, are left out as the expiry views leave them out.
Cryptographic risk report
GET /api/v1/reports/crypto-risk (and the crypto_risk scheduled report,
trstctl-cli reports crypto-risk) lists each weak, distrusted or
quantum-vulnerable key with its dependents, its NIST IR 8547 deadline and its
plan, and what changed since the previous scheduled report. What it decides
from, and what it does not see:
- IR 8547 is a draft. The deadlines come from the initial public draft of November 2024 (Tables 2 and 4), and the report cites it as such; a final version may move them.
- CNSA 2.0 is not encoded. The report does not state the CNSA 2.0 milestones for national security systems, and says so.
- Only what has been seen. It covers the certificates trstctl holds or has discovered and the crypto assets discovery has inventoried. A key with no recorded dependents may be one nothing has scanned around.
- Name matches are matches by name. A certificate trstctl knows only by its recorded issuer name is matched to a distrusted CA by that name.
- Bounded. It rates up to 50,000 certificates and reads up to 100,000 rated discovery findings, and says when it read less; the on-demand report lists up to 1,000 keys and the scheduled one up to 20,000, while its summary and changes count every key.
- A certificate without bytes is sized from its recorded algorithm name
(
RSA-2048,ECDSA P-256); when that names no size, the 112-bit rule applies and the report says it assumed it.
Credential assurance records
GET /api/v1/certificates/{id}/assurance-record signs what trstctl already holds about
one certificate; it collects nothing new. Read it with these limits:
- Independent observation is what scans and endpoint checks saw. Discovery findings and endpoint observations (local or relay) are independent of trstctl's record of its own actions; a connector's delivery receipt is not, so a delivery counts as observed only when an endpoint was then seen serving the certificate. External observers' reports (see External observers) are listed apart and count the same way when they match.
- Endpoint observations are the latest per listener and vantage. Earlier observations of the same listener are not retained, so the record cites the newest one; the verification receipts carry the history of each delivery's check.
- A profile appears only when the issuing command bound one. Scheduler renewals that bound no profile, imported and discovered certificates, and history older than reviewed profile bindings have no profile entry, and the rendering says so.
- Bounded. Each evidence list holds up to 200 entries and the audit references up
to 100 (
truncatednames any list that hit its bound); an evidence pack carries up to 50 records, those whose lifecycle changed in its window. - Verify against keys you pinned. The response's
public_jwksis a convenience;trstctl certificates verify-assurance-recordrequires a JWK set you obtained separately and never trusts the file's own copy.
Outcome pages
The three outcome pages say how strongly each step is proved and name the test:
- Kept by hand. The proof column is maintained with the features it describes; no guard checks it against the test suite, so a page can trail a change until it is updated.
- Starting screens, not guided steps. The console home's Outcomes section opens each workflow's first screen; it does not walk the steps in order.
Evaluation kit
trstctl evaluate runs one certificate journey against an endpoint you choose
and writes a signed evidence pack. Read the pack with these limits:
- Discovery is a handshake, not a sweep. The control plane connects to the endpoint named for the run (a CBOM TLS scan); finding endpoints nobody listed is what a network relay's discovery scan does, and the journey does not run one.
- One vantage. Verification and the renewal check are the command's own TLS handshakes from the machine it ran on.
- OCSP, not CRL. Revocation is checked over OCSP; CRL publication is asynchronous and not checked. The issuing CA certificate for the OCSP check comes from the issuance result or the chain the endpoint served; an endpoint that serves only its leaf, with a host-generated key, cannot be checked.
- Keys from the same server. The run checks the audit export with the keys
the server publishes;
trstctl evaluate verifychecks the pack and its export again with a key set obtained separately. - One run, and it changes the endpoint. The journey creates an identity, deploys two certificates and revokes the first. It is not a load, failover or long-running test, and belongs on an endpoint under evaluation.
Customer assurance reports
GET /api/v1/reports/assurance and the customer_assurance scheduled report put
a tenant's credential assurance into plain sentences in its white-label brand.
What the report decides from, and what it does not establish:
- Counts are exact; lists are bounded. Each list names at most 10 items (unowned and soon-expiring certificates, failures, incidents); the count beside it is exact.
- Endpoint results are the latest observation. The report counts each endpoint's latest check from each vantage, not every check made in the period.
- Only failures this deployment recorded. Failed renewals are renewal runs that ended failed; failed deliveries are connector deliveries that failed or failed their verification. Work done outside trstctl is not seen.
- Changed credentials are classified up to 50. Whether a changed credential's deployment was independently observed comes from its assurance record, read for the 50 most recently changed credentials; the report says when there were more.
- Large incident evidence is named, not carried. An evidence bundle over 64 KiB appears as its SHA-256 digest; read it from the incident.
- Only a brand that embeds safely. A logo is embedded only as a PNG, JPEG,
GIF or WebP data URI of at most 256 KiB, never SVG, and the accent only as
#rgbor#rrggbb; anything else is left out and the report says so. The HTML rendering never names trstctl; machine-readable format identifiers keep their schema names. - The comparison is with the previous signed report. A scheduled run compares with its schedule's previous completed run, read back from the archive and verified first; an on-demand report compares with the tenant's newest completed one, whatever period that covered. With no archive or no earlier run there is nothing to compare, and the report says so.
- Verify against the key you pinned.
trstctl-cli compliance report-runs verifyneeds the signing key obtained separately and never trusts the key inside the file. A run signed before a key rotation verifies under the key that signed it. - The Provider reads reports on its API. The Provider console does not show
a customer's reports yet;
/provider/v1/tenants/{id}/assurance-reportslists and downloads them. A console view is what comes next.
Endpoint verification
Every other record in this product reports what trstctl DID. An outbox row
delivered, a connector returned success, a certificate was issued — all of them
can be true at once while the listener serves something else entirely, because
a connector's reload is one exec call inside its own Deploy method and nothing
downstream observes whether it took effect.
The failure this exists to catch. A renewal succeeds at the CA. The connector writes the file. The reload fails, or the service ignores it. Every delivery receipt stays green, the inventory correctly describes the new certificate, and clients keep getting the old one until it expires. Inventory-based expiry alerting cannot see this, because the inventory is right but does not describe what is being served.
Two vantages, and the difference is not redundancy. The host agent's check
runs inside connector.deploy, in the only window where it can: after the
reload, before the redeemed material is destroyed. It verifies against the exact
bytes it deployed rather than a description of them. The relay's check is a
separate endpoint.verify job — network vantage, no credential redeemed — and
it is the only witness for an appliance, because nothing runs on an F5. A local
pass means the box thinks it is fine; only a relay pass means a client could get
it. The two are stored as separate rows and never merged.
A verdict never claims more than it checked. Each record carries whether the name set and chain were compared in addition to the fingerprint. An expectation that supplied no SAN set yields a verification that does not claim to have checked names.
Unreachable is not a pass. It is stored as its own state, with no mismatch class, no comparison flags and no observed fingerprint — enforced in three places (the agent's transcript validation, the projector's decode, and a database CHECK constraint) because an endpoint nobody could connect to, recorded as verified, would be a worse false assurance than the blindness this replaces.
last_good_at is never erased by a failure. The gap between it and
last_checked_at is how long an endpoint has been failing, and losing it on the
first failure would destroy the only measure of the outage's age. An endpoint
that has never once been observed serving what it should reads as never
verified, which is a stronger statement than "not recently" and renders as one.
Re-verification is a loop, not an event. A post-deploy check proves the
reload took effect at that moment; it says nothing about the weeks afterward,
and a listener can start serving the wrong certificate long after a deploy — a
failover to a node that never got the file, a config reload elsewhere, a restored
backup. Sweeps run hourly by default (EndpointVerificationInterval), batched at
50 endpoints per job so one tenant cannot hold a relay indefinitely. Each sweep's
expectation comes from the control plane's own record of what should be there,
never from the last observation — re-probing against what was last seen would
re-verify a divergence as correct on the next sweep and silence its own alarm.
When a queued relay sweep returns after a host deployment has changed that
endpoint's expected fingerprint or address, its signed transcript and job
receipt remain historical evidence. The obsolete comparison does not replace
the current endpoint row or send a divergence alert. A fresh network check is
still needed to prove the replacement from the relay's vantage; the next
scheduled sweep obtains the latest local deployment expectation. Until it
does, the relay row reads not_checked for the current identity, the summary
counts it separately, and the dashboard says it is awaiting a relay recheck.
Divergence alerts route by severity, not by finding score. A served-identity
divergence is critical; an unreachable endpoint is warning, because a probe
that could not connect may be a firewall or a maintenance window and paging at
critical for that teaches people to ignore the channel. The alert kind is
distinct from credential.drift deliberately: drift drives a file-repair
workflow keyed on a filesystem path, and a served-identity divergence usually has
a perfectly correct file that a process never reloaded.
Automatic rollback is opt-in, per target, default off. Setting
auto_rollback_on_verify_failure on a deployment target closes the loop: a
verify_failed deploy queues D4's executed re-bind to the predecessor. It fires
only on verify_failed and never on plain failure — a deploy that failed did not
change the target, so rolling it back would undo something that was never done.
Absent flag means off; a target whose config predates this feature never starts
changing itself because a new version shipped. The four appliance re-bind
families and all 14 host-local restore families can execute it. Host work is
pinned to the agent that retained the predecessor; a first deployment still has
nothing to roll back to.
Manual and automatic rollback share a tenant-scoped inventory check: the
predecessor must be known and unrevoked, and a bound identity must not be revoked
or retired. The control plane repeats that check before handing out queued work;
the updated agent requests fresh authorization before opening its predecessor
store or redeeming an appliance credential. Upgrade both components to obtain
all three checks. A control-plane refusal is recorded as rollback_refused,
without claiming agent execution. An operator can cancel a target's in-flight
connector work (POST /api/v1/connectors/targets/{id}/cancel): queued commands
are withdrawn, agent-leased commands are recalled so the agent is refused at its
next check-in and its late result is retained rather than applied, and a command
the control-plane worker is executing inside a connector call is reported and
left alone rather than interrupted mid-write. Cancellation cannot undo an effect
that already landed. Compromised material already installed on a host target is
removed by POST /api/v1/connectors/targets/{id}/purge (endpoint.purge): the
exact enrolled agent proves the installed leaf, overwrites and removes the
certificate and key files the target names, wipes the rollback ledger of that
fingerprint and runs the bound stop action; only a purged receipt claims
that. A purged receipt drops that target from the rogue listing's
installed_on at once, and an identity whose leaf is then served by no target
moves from deployed back to issued (the certificate exists; nothing serves
it); an identity still served elsewhere keeps its state. Retiring or revoking
the identity is the operator's separate next step. Appliance and cloud targets, java-keystore, IIS and Envoy are not covered:
their material is not an addressable file on an enrolled host.
Configuring it. Local post-deploy verification runs when a deployment target
carries verify_address (and optionally verify_server_name) in its config —
for example {"cert_path": "/etc/nginx/server.crt", "verify_address": "api.example.test:443"}.
The address is never derived or guessed: on an appliance target
endpoint is the MANAGEMENT API, and the connector's target string is a routing
label, so an F5's management plane and the virtual server it fronts are
different sockets. trstctl suggests one from evidence instead:
GET /api/v1/connectors/targets/{id}/verification-suggestions (CLI connector target verification-suggestions, the destination's Verification address
panel in the console) lists the host:port addresses where a discovery scan saw
a certificate trstctl delivered to that target (a delivered or verified deploy,
not purged since), each with the findings that saw it. Nothing changes until an
operator confirms one with POST .../verification-address (CLI connector target confirm-verification-address), which refuses an address no finding suggests and
a target that changed since the suggestions were read; the confirmed address is
written to the target as a new revision and later deployments verify the listener
there. Any other address is set by updating the target.
What is not served. Verification only covers endpoints with a listener address an operator gave or confirmed; an endpoint with no address is never verified and never claims to be. Suggestions come only from TLS discovery findings, so a listener no scan has seen is not suggested; addresses read from connector bindings (a load balancer's virtual server, a cloud listener) are what comes next.
External observers
POST /api/v1/observations takes what an external observer saw at an endpoint and
compares it with what trstctl expects there; POST /api/v1/observers registers
observers, which also receive rotation and rollback events. What it relies on:
- The expectation is trstctl's own endpoint verification. An observation of an
address no trstctl verification checks is recorded as
unexpected, with nothing to compare. Report endpoints by thehost:porttrstctl verifies. - The observer's report is taken as reported. trstctl does not probe the endpoint
to confirm it; the observer's token (scoped to
discovery:write) is what authenticates the report, and the record names the observer. - Events are at least once. An observer drops a redelivered event by its
id. A projection rebuild does not resend events older than a day. - What comes next. Observer management is served through the API and CLI; console pages for it, and probectl's own client for this contract, come next.
GET /api/v1/issuers/capabilities and trstctl issuers capabilities serve one
row per authority kind: discover, issue, renew, revoke, whether trstctl can
satisfy its domain validation unattended, where the private key is generated,
and what the authority validates before issuing. The console shows revocation
and domain validation beside each configured issuer.
Unattended DV is checked against the source, in both directions. It is true
only where the issuer's package wires a challenge solver into the ACME driver —
one authority kind today — and an authority that reads false must say why, because
"there is no challenge to solve" (an internal CA) and "a human completes DCV in
the vendor's console" (public OV/EV) are opposite operational situations. It is a
property of the build; whether a given authority is actually configured for it is
the upstream_dns01 flag on that authority.
Revoke is the field that matters, and it is deliberately conservative. It is true only where this build ships an implementation that contacts the authority and is proven against that authority's protocol by a test:
| Issuer | Revoke | How |
|---|---|---|
letsencrypt |
yes | RFC 8555 §7.6 revokeCert, JWS-signed with the account key |
vaultpki |
yes | POST {mount}/revoke by serial number |
ejbca |
yes | REST PUT /certificate/{issuer_dn}/{serial}/revoke |
Everything else reports false with a note saying where to revoke instead.
DigiCert, Sectigo, Venafi, AWS Private CA, Google CAS and step-ca all document a
revocation API; trstctl does not drive any of them, and a documented endpoint
nobody has implemented is a plan rather than a capability. AD CS revocation runs
through the CA's own management interface, and Azure Key Vault disables
certificates rather than revoking them.
No silent no-ops. Exact-certificate revocation against an authority that
cannot revoke through this build returns a failed item with the unsupported
reason and queues no upstream effect. The X.509 identity-transition preview
marks complete revocation not ready and names the affected certificate. For a
non-compromise or non-serving identity, a privileged operator can still record
the local lifecycle transition to stop automation; that state does not revoke a
leaf at the external CA. For a serving X.509 keyCompromise, ordinary lifecycle
execution refuses before an event or outbox intent and directs the operator to
the reviewed CA-and-host command. Without an exact managed host, the operator
must revoke the exact certificate at its issuing CA and independently stop the
workload. The product does not infer host containment from the revoked state.
The revocation.publish command attributes every bounded leaf to the authority
that signed it from retained issuance evidence: platform-CA leaves publish on
the tenant CRL, leaves from a private CA hierarchy authority (host-executed
identities whose issuing_authority_source is private) are revoked in that
authority's issued-certificate ledger and CRL, external-CA leaves at that CA, and
a superseded predecessor whose managed CA key was destroyed by signed retirement
is skipped rather than blocking the live successor's publication.
This is the specific failure the epic exists to remove: an
operator revoking a compromised key and being told it worked, while the
authority still considers the certificate valid, is worse off than one told
plainly that trstctl cannot do it — the second sends them to the vendor console,
the first sends them home.
ACME is worth one further note. The protocol identifies the certificate to revoke by its DER, not by serial, so trstctl cannot revoke an ACME certificate it does not hold a copy of. A request carrying only a serial is refused with the reason rather than sent as something the protocol cannot express.
The local Pebble 2.10 partner-lab issuer is also a limited revocation witness.
Its issued leaf in the U5 QA run had no CRL Distribution Point and no OCSP URI.
The issuer accepted trstctl's authenticated ACME revokeCert requests for the
exact retained certificates, but a stock relying party cannot discover a
signed CRL or OCSP responder from that leaf and independently reject it on
revocation grounds. The lab can prove upstream acceptance and can use the
host agent plus a fresh TLS probe to prove the compromised leaf is no longer
served. It cannot claim stock client revocation enforcement for that Pebble
certificate. The same upstream ACME path is proven with an independent issuer
that does publish revocation status (A114): the lab's step-ca 0.28.3, an ACME
CA validating dns-01 through the lab's own DNS-01 webhook, regenerating its CRL
on revoke and naming it in every leaf. trstctl ordered a certificate from it,
revoked it through its ACME revokeCert, and stock OpenSSL then refused the
leaf by step-ca's CRL (openssl verify -crl_check: certificate revoked; a TLS
handshake with CRL checking fails the same way), while a client that does not
check revocation still connects. That proves the trstctl side for any
upstream ACME CA that publishes status; each issuer's own publication is its
own to qualify.
After exact host containment, the endpoint replacement preview reads the
target's latest host recovery check: a read-only endpoint.recovery_check job
the enrolled host answers in a signed receipt with the SHA-256 of its pinned
recovery action's executable and command line and the listener state one local
TLS probe saw. The preview reads ready, and the replacement can be created,
only when a check requested after the stop shows a recovery action the
target's connector runs while deploying and a listener that is not serving the
contained leaf. The check reports what the host profile pins, not what the
action does: whether it re-enables only that listener and refuses every revoked
predecessor is still the host operator's to test. The agent's verified
deployment receipt and a fresh stock TLS readback are the completion evidence;
the preview and accepted binding alone are insufficient. The NGINX procedure is
documented in docs/features/deployment-connectors.md.
A CI guard parses every issuer package and fails the build if the matrix and the code disagree in either direction — a claimed capability with no implementation, or an implementation the matrix does not advertise.
Coverage, provenance and blind spots
A certificate count is not an inventory, and the difference is the whole of this section. "We found 4,312 certificates" answers how many discovery happened to turn up. An auditor asks a different question — how much of the estate did you look at, and when — and a system built only from findings cannot answer it, because what it never looked at leaves no trace in what it found.
Coverage is measured against a declaration. An operator names the segments they own, with the ranges, an exclusion flag and reason where a segment is deliberately out of scope, and a staleness window that is theirs to set — a DMZ and a lab do not deserve the same answer. Coverage is then the share of declared, non-excluded segments swept inside their own window. Excluded segments are removed from both halves rather than counted as covered; a number that rose when somebody excluded something would reward exactly the wrong behavior.
The consequence is worth stating plainly: an estate with nothing declared reports no coverage, not full coverage. That reads as unhelpful on day one and is the only defensible answer — the alternative is a system that declares itself complete because nobody told it what it was missing.
Every certificate carries provenance. Which source last observed it, of what
kind, and when. last_seen_at is deliberately distinct from created_at: the
first says something confirmed the certificate still exists, the second only
says trstctl once recorded it. Conflating them makes a stale inventory look
freshly verified. A certificate this control plane issued that nothing has since
scanned has no observation at all, and is reported that way — it is evidence
of an issuance, not of a deployment.
The blind spots are named, not implied. The register lists segments never swept, segments outside their own staleness window, segments declared out of scope with the reason, asset classes no configured source can ever see, the count of inventory rows with no observation behind them, and the addresses trstctl's own sources saw outside every declared segment. Each carries the action that closes it, or says plainly that nothing does.
Addresses outside the declared segments. GET /api/v1/discovery/coverage/candidates (trstctl-cli discovery coverage candidates, and the coverage panel in the console) lists the network addresses
recorded in the last 30 days — certificates' deployment locations, endpoints
trstctl verified, endpoints external observers reported, and destinations'
verification addresses — that no declared segment covers, by the same rule a
sweep's targets are held to (an exact hostname or address, a CIDR, or an
explicit range; an excluded segment counts as declared). Nothing is resolved,
and file paths and other location labels are not addresses. Each is a candidate
for a declaration, which the console can make in one step. The list is never
complete: an address none of these sources has recorded cannot appear, so a
network nobody has touched still has to be declared by the operator.
What this does not do: it does not verify that a certificate found at an address is the certificate that address serves to a real client — that is verification, and it is a separate claim.
Kubernetes deployment
The control plane ships a production-shaped Helm chart
(deploy/helm/trstctl): the API/UI with the signing service isolated (its
own locked-down, network-unreachable sidecar), external PostgreSQL and NATS
as the default, a default-deny NetworkPolicy, and TLS.
Kubernetes Operator scope: a focused CRD-driven operator ships today. The
trstctl-operatorbinary (it rides inside the same multi-binary control-plane image and is run bydeploy/operator/operator.yamlvia an entrypoint override) reconcilesTrstctlControlPlanecustom resources into a managed control-plane Deployment (replicas, image, the PostgreSQL DSN Secret reference, NATS URL and replica knobs, sidecar-signer socket/volumes and managed-key provider enablement; its manifest documents the leader-elect and coordination.k8s.io controls that keep the reconcile path bounded) and the network surface around it, the way the Helm chart renders them:- a Service on the API/UI port (
spec.service: type, port, annotations); - a default-deny
NetworkPolicy(spec.networkPolicy, on unless disabled) that admits the API port only from the named ingress controller pods, the namespaces you list, or the same namespace when you opt in, and lets the control plane reach DNS, PostgreSQL and NATS (optionally only within the CIDRs you list) and its signer; - an optional Ingress (
spec.ingress: class, host, TLS Secret, annotations) whose backend is the HTTPS port; - with
signerMode: isolated, the signer as its own pod with its own Service on 9443 and aNetworkPolicythat admits only this control plane, over the mutually pinned mTLS channel. Unless the resource names the two Secrets you provide, the operator generates a cross-pinned pair into<name>-signer-mtlsand<name>-cp-signer-mtls, mints a new pair a month before it expires or when the server name changes, and rolls both pods onto it through a pin fingerprint on their templates.
Signer authorization follows the chart:
spec.signer.authTokenCommand(required with production NATS) or, for a sidecar evaluation,spec.signer.allowCoResidentAuthorizer. Combinations the control plane would refuse at startup set the resourceBlockedwith the reason instead of rolling out pods that cannot start. Every object the operator creates carries its managed-by and owner labels and an owner reference, so deleting the resource removes them; it deletes only objects carrying both labels, and refuses to take over an object of the same name that it does not own. Status reports each surface (create,update,delete,noneorabsent) with phaseReady,ReconcilingorBlockedand a message. The operator also reconcilesTrstctlSecretSynccustom resources into Kubernetes Secrets plus reload annotations, andTrstctlSecretInjectioncustom resources into no-code sidecar/env/file injection patches for opted-inDeployment,StatefulSet, andDaemonSetworkloads. For SecretSync resources it resolves values through the served secret-store API, writesSecret.data, recordsstatus.contentHash, and patches pod-template annotations instead of deleting pods. For SecretInjection resources it reads source Secret metadata and key names, rejects missing requested keys before patching, then patches the shippedtrstctl-agent --secret-injectsidecar, app mounts, and optional env references, and recordsstatus.injectedWorkloads. It is a real, level-based reconcile loop (poll, diff, converge), not a stub; it speaks the Kubernetes API directly (no client-go/controller-runtime). The shipped operator manifest runs two replicas and--leader-elect; the replicas coordinate with a realcoordination.k8s.ioLease so exactly one reconciles while the other remains a hot follower. Thekindjob in CI drives the reconciler against a real API server with only the RBAC the manifest grants: the API server accepts every object it renders, the generated pair completes the pinned handshake, a converged pass writes nothing, a spec change patches in place and removes what the spec dropped, and an object it does not own is refused. What comes next: the operator does not yet generate the KEK and signer-authorization Secrets or render the chart's PodDisruptionBudget, ServiceMonitor and external-KMS custody arguments (the chart covers those today), it polls rather than watches, and a control plane serving through an operator-managed isolated signer has been proven at the object and channel level, not yet as a running pair on a cluster.- a Service on the API/UI port (
Kubernetes certificate CRDs: the Kubernetes agent ships a real trstctl
Issuer/ClusterIssuer/Certificatecontroller. It marks trstctl issuer resources Ready, signs matching cert-managerCertificateRequests through a served trstctl issuance endpoint using a mounted API token, signs approved native KubernetesCertificateSigningRequests fromcertificates.k8s.io/v1, and also fulfills a trstctl-nativeCertificatedirectly into akubernetes.io/tlsSecret.GET /api/v1/kubernetes/certificate-signing-requestsandtrstctl-cli kubernetes csrexpose the CAP-K8S-04 posture, supported signer names, RBAC, status fields, and residuals. The cert-manager path is proven in CI onkindwith real cert-manager fromCertificateto TLSSecret; the native trstctl path is proven by the served controller acceptance test from trstctlCertificateto local CSR, signer, Secret, and Ready status; native Kubernetes CSR support is proven by a controller test that writesstatus.certificatewhile preservingApprovedonly after Kubernetes approval (native CSRs do not define a Ready condition). It is an informer/work-queue controller: each resource is listed and then watched (following bookmarks and relisting when the server expires a version), and a de-duplicating, backed-off work queue converges one object per key as its event arrives, with a full re-check every--k8s-resync(default ten minutes) for what no watch reports. The kind job proves it with the agent's own service account and the re-check ten minutes away: a TrustBundle copied within half a second, a deleted copy restored, an approved CSR signed, a trstctlCertificateissued and renewed. trstctlCertificates are renewed with a new key two thirds of the way through the validity they had when issued (status.renewalTime), and reissued when their spec changes or their Secret is deleted. cert-manager is optional: without its CRDs the native CSR and TrustBundle paths run. Because the agent runs as a DaemonSet, the cluster-scoped controller elects a single reconciler through acoordination.k8s.ioLease (trstctl-agent-issuer-controller, 30s duration, identity from the pod'sPOD_NAME): one pod reconciles, the others idle, and a follower takes over within one lease duration if the holder dies. If the Lease RBAC is absent the agent logs once and reconciles anyway — duplicated idempotent work beats no controller. The namespaced cert-manager bridge does not contend. CSR approval stays a Kubernetes approver's by default. The opt-in--k8s-csr-approver(withdeploy/kubernetes/rbac-csr-approver.yaml) has the agent ask the issuing CA about each pending native CSR for a trstctl signer name —POST /api/v1/ca/authorities/{id}/issue/evaluate, the checks issuance makes, without signing — and write Approved, or Denied with the CA's reason (condition reasonTrstctlProfilePolicy); an approved request is signed in the same pass. A post-quantum subject key is evaluated on its names and lifetime. cert-managerCertificateRequests keep their own approver. The kind proof of the approver runs in the Kubernetes e2e job.Multi-replica HA: the Helm chart runs the control plane multi-replica by default (
replicaCount: 2,RollingUpdate maxUnavailable: 0, PodDisruptionBudget, pod anti-affinity), and running >1 replica is safe: leader election (a PostgreSQL session-scoped advisory lock) gates the continuous background workers — the outbox dispatcher, audit retention, idempotency/outbox GC, the projection tailer, the CRL scheduler, and the read-model snapshot worker — to exactly one replica so they never double-apply, with automatic failover to a follower on leader loss; all replicas serve reads. A shared signer key store (persistence.signerKeysAccessMode: ReadWriteMany) means every pod's locked-down sidecar signer (the isolated key-holder process) loads the same sealed issuing-CA key, so all replicas are the same CA (first-boot provisioning is serialized by an advisory lock). For a single signer pod that serves all replicas independently, setsigner.mode: isolated: the signer runs as its own pod reached over a cross-node mTLS gRPC channel — TLS 1.3, AEAD-only, with the control plane and the signer each pinning the other's certificate (an untrusted or merely CA-signed-but-unpinned peer is rejected). Thetrstctl-signerbinary serves--mtls-listenand the control plane dials it withsigner.mtls_address; the chart renders the signer Deployment/Service/NetworkPolicy on:9443when you supply thesigner.mtls.*certificate material. The default co-located sidecar (UDS) topology remains the simplest single-pod option and is not required to change for the HA above. See disaster recovery → High availability. (The agent, separately, runs as a DaemonSet across all nodes.)Cross-cluster federation is passive read-state replication: a passive cluster can import a peer event log, keep a durable per-peer cursor, and project the imported tenant/trust/certificate/audit read state locally for failover. It is intentionally not an active-active write conflict resolver — keep one writable region for a tenant at a time, stop or fence primary writes before promotion, and use
TRSTCTL_FEDERATION_RPO/TRSTCTL_FEDERATION_RTOas measured runbook targets.
Non-functional targets: what is measured vs. aspirational
We separate NFRs that have executable evidence from ones that are aspirational and not yet measured in CI, so neither is silently over-claimed.
- Performance & scale NFRs are measured: the hot-path latency/throughput
SLOs and the capacity model are pinned to committed measurement receipts
by an executable smoke gate (
make perf-smoke) and a served realistic/peak live-load gate (make perf-live), and sustained-load endurance is pinned by a soak gate (make soak) that fails on a leak slope or an SLO breach. These are local eval/self-test scale denominators, not a substitute for a customer-specific multi-hour load test at your own capacity tier. - Usability outcome NFRs are evidence-gated.
USABILITY-SLO-001has automated wizard timing evidence: thescripts/usability/first-run-receipt.jsonreceipt is generated byscripts/usability/measure-first-run.mjs, which walks the first-run wizard contract (internal CA confirmation, first certificate issuance, served connector deployment, upstream-CA issuance, dynamic-secret lease issuance, enrollment-token minting, and agent detection) and keeps that assisted path inside a 15 minute time-to-first-certificate budget. The scope is intentionally narrow: it measures the browser journey and served API-client contract in CI, not human reading time, package download time, real network latency, or the physical agent installation step.USABILITY-SLO-002for operator-satisfaction / NPS has a receipt gate but no numeric NPS claim: the currentscripts/usability/operator-study-receipt.jsonisno_numeric_claim, and release tooling fails closed if release notes publish NPS, CSAT, or operator-satisfaction numbers before a real measured external-operator study receipt replaces it. See Usability outcome SLOs.
How to read the roadmap against this
The source-checkout README.md capability table
describes what is built and tested; this page tells you what is served by
the binary today. When the two differ, this page is the authority for what
you can rely on at runtime.
Non-production entitlement boundary
The signed non-production entitlement is an offline deployment binding, not a
traffic classifier. Version 2 licenses enumerate one production ID and at most
three non-production IDs; the binary refuses an operator-declared runtime ID or
environment that does not match. It cannot independently prove that a host
called non_production carries no real-user traffic, because doing so would
require surveillance or phone-home state that the product explicitly avoids.
Version 1 license files predate environment binding. They remain loadable for upgrade continuity but are production-only and cannot claim the bundled non-production entitlement. Issue a v2 file before moving a legacy deployment into a staging or test slot.