A ctlplne studio product

Issuance & certificate authorities — how trstctl mints and governs certificates

What it is

Issuance is the act of creating a certificate: a machine asks for one, an authority signs it, and the machine gets back a signed ID it can present. This page covers issuing through any authority, running your own CA hierarchy, the rules that constrain what may be issued, telling clients when to renew, taking certificates back early, and where the private key physically lives.

The mental model: trstctl is a passport office. A CA prints and signs passports; a profile is the rulebook for what a valid passport may say; a registration authority checks your paperwork but can't print the passport itself; revocation is the bulletin of canceled passports; and the HSM is the locked vault holding the official seal.

Why it exists

Certificates expire on purpose and must be re-minted constantly, so issuance has to be automatic, governed, and auditable. Without a real issuance layer, three things go wrong: the wrong certificate gets minted (too-long validity, weak key, a name the requester shouldn't control); the signing key leaks and forges everything; or a compromised certificate keeps being trusted because nobody can pull it back. trstctl's issuance layer exists to make each of those hard.

How it works

One issuance path, any CA (F4)

Every certificate trstctl issues goes through a single, uniform interface — a CA with one real method, Issue(request) — no matter who actually signs. The built-in signer-backed CA, a CA in your own hierarchy, and 14 third-party authorities (Let's Encrypt/ACME, DigiCert, Sectigo, Microsoft AD CS, AWS Private CA, Azure Key Vault, Google CAS, EJBCA, Smallstep, Venafi TPP/TLS Protect, Vault PKI, GlobalSign, Entrust, and the shell CA escape hatch) all implement that same interface. On every one of these paths a requester's key that fails a structural weak-key check, or that the tenant has recorded as compromised, is refused before it is signed or sent upstream, naming the check (see Weak keys). The running binary exposes configured upstreams as a served registry at GET /api/v1/external-cas; callers issue through one selected CA with POST /api/v1/external-cas/{id}/issue using a PEM CSR, DNS names, and an Idempotency-Key. The CA Hierarchy page drives the same route from the browser: pick a configured external CA, submit CSR/DNS/profile/TTL, watch outbox-pending while the issue intent is recorded, then see external-ca-issued evidence — never the certificate PEM.

The same binary exposes read-only CA discovery at GET /api/v1/ca/discovery: one response normalizing configured public/private upstream CAs and imported private hierarchy authorities, with counts, source ids, status, and served path pointers — never certificate PEM or private key material.

That single path wires the intended guarantees to receipts. Each issuance carries an Idempotency-Key. For an upstream CA, the request first commits one tenant-scoped external-ca.issue outbox intent; only the normal bounded dispatcher calls the provider, waiting for its exact tenant/key result rather than draining unrelated tenants. Before calling a provider without a native request token, the worker durably claims that operation once; providers whose receiver enforces the supplied token (AWS PCA, Azure Key Vault, Google CAS) use the reconciled retry lane, while every unproven adapter stays at-most-once. A completed result replays byte-for-byte from the certificate.recorded projection, so a changed command gets 409 before provider I/O, and an interrupted tokenless submission is never resent — it stays explicitly indeterminate rather than guessing the CA did nothing. An ACME authority that pre-authorizes accounts out of band (CertiNext, HARICA and other EAB-gated CAs) is configured with eab_key_id plus eab_hmac_key_ref; the upstream client then presents the external account binding at registration and the MAC key never leaves locked memory. acme_profile names the draft-ietf-acme-profiles profile the client requests on every order; the request is made only when the authority advertises that profile in its directory.

For ACME, a live attempt that stops before submitting the CSR for finalization can prove that signing never began. The worker may then retry the same issuance request after the directory, account, order, or domain-validation service recovers. Account or order preparation may repeat, within the existing bounded outbox retry policy. Once finalization starts, a lost response remains uncertain: trstctl tries to read the original order's result, and refuses a new issuance if that cannot be verified. A process crash supplies no proof of which step ran; an already-pending uncertain claim stays blocked. Do not delete that claim to force another certificate. After a definite result the worker emits certificate.recorded, rebuilds the certificate inventory, and records the separate ca.issue evidence row. The identity-transition issuance retry path (POST /api/v1/identities/{id}/transitions to issued) is exercised by the shipped-stack Compose E2E gate: the gate repeats the same request with the same key, waits through the real PostgreSQL/JetStream/outbox/ isolated-signer path, and asserts certificate inventory remains exactly one before revoking that certificate and verifying OCSP and CRL state. This bounded proof does not turn tokenless or unreconciled upstream adapters into exactly-once receivers. The request's CSR is inspected through the single isolated cryptography path, and the active profile is enforced before signing, with an issuance.profile_evaluated event recorded either way.

If the CA returns a certificate but local recording fails, retries retain that result instead of asking the CA to sign again. Recording recovery checks each retained certificate event's complete receipt before doing projection work again. A receipt must match the original event; a sequence watermark alone is not proof. Missing recordings still recover in order, under the tenant and privacy barriers. Recovery still reads retained history, so its cost grows with that history; this does not establish a fixed recording-latency guarantee.

Upstream CA credentials are configured by the control-plane operator, not tenant JSON: file references load into locked byte buffers for one outbox attempt and are wiped afterward. Azure CA private-key operations and Let's Encrypt account JWS signatures stay in the isolated signer, and the API exposes only the non-secret registry row (id, type, name, status). A reused idempotency key after completion returns the original certificate without re-signing, even after garbage collection. A crash before the provider claim resumes through the outbox. After a tokenless provider claim commits, recovery needs the original result or a live protocol proof that signing was not submitted; an uncertain crash stays blocked. The production external_cas JSON shape, file:/absolute/path credentials, private-endpoint allowlist, custom trust-root, and mTLS fields are documented in Configuration.

Kubernetes CRD-native issuance

trstctl ships Issuer, ClusterIssuer, and Certificate CRDs in the trstctl.com API group. The Kubernetes agent reconciles them, marks issuers Ready, signs cert-manager CertificateRequests only when they target an existing trstctl issuer, signs approved native CertificateSigningRequests from certificates.k8s.io/v1, and can fulfill a trstctl-native Certificate directly into a Kubernetes TLS Secret. With the opt-in --k8s-csr-approver the agent also approves or denies pending native CSRs for trstctl signer names by the issuing CA's answer (POST /api/v1/ca/authorities/{id}/issue/evaluate, the checks issuance makes without signing; trstctl-cli ca authorities evaluate-leaf), writing Denied with the CA's reason. The read-only GET /api/v1/kubernetes/certificate-signing-requests / CLI trstctl-cli kubernetes csr report the served CAP-K8S-04 surface, supported signer names, required RBAC, and residuals.

A cert-manager Certificate references trstctl with issuerRef: {name: trstctl, kind: ClusterIssuer, group: trstctl.com}; a workload can also use trstctl's native API directly:

apiVersion: trstctl.com/v1alpha1
kind: Certificate
metadata:
  name: web
spec:
  secretName: web-tls
  dnsNames: [web.apps.svc.cluster.local]
  issuerRef: {name: trstctl, kind: ClusterIssuer, group: trstctl.com}

The agent forwards only a CSR to the configured trstctl issue endpoint, adds a stable Idempotency-Key, and authenticates with a token mounted from a Kubernetes Secret: cert-manager gets the normal kubernetes.io/tls Secret; a trstctl-native Certificate gets a locally generated workload key written to Secret/<secretName> (transient buffers wiped) and marked Ready; a native CertificateSigningRequest needs Kubernetes or a separate approver to set Approved — the agent never approves its own requests. cert-manager CertificateRequests for this external issuer also need a separate approver on every renewal. Without one, a short-lived Secret can expire while the next request waits. The cert-manager approval runbook shows a scoped approver policy, RBAC binding, denial checks, and served-TLS readback. For native Kubernetes CSRs, the agent accepts spec.signerName values such as trstctl.com/trstctl or trstctl.com/<issuer-name>, optionally disambiguated with the trstctl.com/issuer-{name,kind,group} annotations, and writes the PEM chain to status.certificate while preserving the Approved condition — completion is status.certificate being present, not a custom Ready. CI proves the cert-manager path against a real kind cluster; served controller acceptance proves both the trstctl-native path and CAP-K8S-04 native CSR support. The shipped ClusterRole grants sign only for trstctl.com/trstctl; a named signer such as trstctl.com/payments needs that resource name added rather than granting every Kubernetes signer.

For cert-manager, the enrolled agent also reports each trstctl-issued CertificateRequest's exact UID, controller-owned parent Certificate UID, and issued leaf fingerprint over the authenticated agent channel. The event-projected GET /api/v1/kubernetes/cert-manager-certificate-requests and trstctl-cli kubernetes cert-manager-requests expose the latest controller observation. Inventory list/detail responses retain kubernetes_provenance for an exact issued fingerprint even after cert-manager garbage-collects that request. The Certificates console links that observation to Workloads & Machines without calling an externally controlled renewal manual. This is issuance provenance; verify the Kubernetes Secret and the served listener independently before claiming deployment.

The same agent serves CAP-K8S-07 public trust-bundle distribution. Configure the controller's exact --k8s-trust-namespaces allowlist before applying a cluster-scoped TrustBundle.trstctl.com. The empty default allows no namespaced targets. The controller writes canonical public certificates to new or matching agent-owned ConfigMaps and records the target count, bundle hash, and Ready condition. GET /api/v1/kubernetes/trust-bundles, trstctl-cli kubernetes trust-bundles, and the Workloads console show the controller's reported posture and required grants.

The control plane can schedule a rollout to reporting clusters, canary first. Each participating controller must enable --k8s-trust-rollouts=true locally. Stored remote TrustBundles require a valid rollout identity and hash, and can distribute to ConfigMaps. The separate CSI trust plugin needs its own opt-in to serve remote TrustBundles. Read the intended revision and its live target bytes when verifying delivery. The canary gate uses the reported hash and bundle name. A newer rollout of the same name replaces it cluster by cluster. See the Kubernetes trust guide.

Running your own CA hierarchy (F48)

trstctl can be your private PKI: a root CA, intermediates beneath it, end-entity certificates beneath those — the usual tree where the root is kept offline-precious and the intermediates do the day-to-day signing.

The dangerous operations are gated by an m-of-n key ceremony: nothing happens until m of n named custodians approve. Root and intermediate creation are served today: open a ceremony, collect distinct custodian approvals, then create or import the CA. Each operation consumes one pending ceremony whose purpose matches the reviewed resource: root:<sha256-of-ca-spec>, intermediate:<parent-ca-id>:<sha256-of-ca-spec>, offline-root:<sha256-of-root-cert-der>:root:<sha256-of-ca-spec>, or offline-intermediate:<parent-ca-id>:<sha256-of-ca-spec>. Existing CA import uses import-existing-ca:<signer-handle>:<sha256-of-chain-der>:root:<sha256-of-ca-spec>, binding the reviewed chain to the exact signer-held key handle.

For a key that trstctl must create, use prepare_existing_ca with the public external issuer chain and child CA profile. After two distinct custodians approve, POST /api/v1/ca/authorities/imported/csr reserves a tenant-bound signer key before returning its public CSR. The external issuer signs that CSR; GET /api/v1/ca/authorities/imported/csr/{ceremony_id} recovers the same CSR after a lost response or restart. Import the signed child certificate followed by the exact reviewed issuer chain with the same ceremony, signer handle and profile. The server checks the public key, issuer chain and optional registered parent before consuming the ceremony. An already-held signer key uses the import_existing_ca ceremony above.

A post-quantum pilot hierarchy runs beside the classical one through the same ceremonies: set signature_algorithm to ml-dsa-44, ml-dsa-65 or ml-dsa-87 on a root or intermediate spec and the signer mints that key (the ceremony purpose binds the key family, so approvals reviewed for ecdsa-p256 cannot mint it). The authority issues classical and pure ML-DSA leaves under RFC 9881, its /crl/{tenant}/authorities/{id} and /ocsp/{tenant}/authorities/{id} responses carry its ML-DSA signature, each authority reports its key_algorithm, and OpenSSL 3.5 or newer verifies the chain, CRL and OCSP responses. Offline-root-signed intermediate CSRs and external-issuer import CSRs stay classical; a re-key keeps the predecessor's key family.

CA renewal/re-key uses ca-rekey-v2:<sha256-of-authority/spec/reason> to create fresh signer-held CA material for the selected authority with the reviewed lifetime and profile. Online intermediate creation, external intermediate CSR signing, cross-signing, and intermediate re-key require an active signer-backed issuing CA. If the parent is an offline public-only CA, ceremony preview and start refuse the online operation before asking custodians to approve it. To replace an intermediate under that parent, use prepare_existing_ca to obtain a public CSR for a new signer-held CA key, have the offline issuer sign the CSR outside trstctl, import the signed successor with its reviewed chain, then preview and activate the predecessor-to-successor rotation. Short approvals return ErrQuorumNotMet; an opener approving their own ceremony, or a ceremony already used or opened for a different resource/spec, fails closed before the CA mutation commits — stopping one compromised admin account from minting a rogue root or intermediate, and stopping one valid ceremony from being replayed against a different CA request.

The console and CLI do not ask an operator to approve a blind change. The effect-free POST /api/v1/ca/ceremonies/preview route validates the exact ceremony body and returns a non-secret receipt before any approval record or signer action exists. For zero-downtime rollover, POST /api/v1/ca/authorities/{id}/rotate/preview applies the activation eligibility rules to the exact predecessor and successor, then explains the routing changes, trust risks, and proof steps without changing either CA. Both previews return an exact request fingerprint and explicit empty write/external-effect lists; the later mutation rechecks authorization and state under lock instead of trusting the preview.

The served hierarchy API lives at /api/v1/ca/ceremonies/preview, /api/v1/ca/ceremonies, /api/v1/ca/authorities, /api/v1/ca/authorities/offline-roots, /api/v1/ca/authorities/imported, /api/v1/ca/authorities/imported/csr, /api/v1/ca/authorities/{id}/offline-intermediates/csr, /api/v1/ca/authorities/{id}/offline-intermediates, and /api/v1/ca/authorities/{id}/issue, with zero-downtime successor activation at /api/v1/ca/authorities/{id}/rotate/preview and /api/v1/ca/authorities/{id}/rotate, signer-backed renewal/re-key at /api/v1/ca/authorities/{id}/rekey, cross-signing at /api/v1/ca/authorities/{id}/cross-sign, offline-root successor/cross-certificate import at /api/v1/ca/authorities/{id}/offline-rekey, and offline-root-produced cross-certificate verification at /api/v1/ca/authorities/{id}/offline-cross-signs. Online root/intermediate private keys live only in the isolated signing service, referenced by signer handles; the control plane stores certificates, chains, metadata, and ceremony state, never the CA private key. Existing-CA import verifies a public chain's first certificate against the supplied signer handle and the chain/profile before serving normal leaf issuance. A durable, global first-writer fence binds that handle to one tenant and reviewed ceremony before the authority event; a conflicting tenant cannot reuse the same signer-held key, including after the first tenant offboards. Offline-root import accepts exactly one public certificate PEM (never a private key) and an optional possession_signature: the root's signature, made on the offline system, over the import ceremony's challenge. With it the authority records possession as offline_signature; without it, as not_proven (see the key ceremony runbook). It then generates a signer-held intermediate CSR for the operator to sign outside trstctl, and imports the result only if it chains to the offline root, matches the reviewed CASpec, and carries the signer-held public key. Rotation and re-key activations both promote a signer-backed successor (re-key from a fresh ca-rekey-v2 ceremony), mark the predecessor superseded, record replaces_id, and keep both issue URLs live while new certificates chain to the successor; offline-root re-key works the same way but stays an operator ceremony since the offline key never enters trstctl.

External intermediate CSR signing is also a protected workflow: the console reviews the public CSR and spec, opens an exact two-custodian ceremony, refreshes its quorum and signs only after approval. API and CLI callers supply that ceremony ID with the same CSR and spec; the external private key stays with its requester. The operator verifies the returned chain and CSR public key with a stock X.509 client. Every served step (ca.ceremony.started, ca.ceremony.approved, ca.ceremony.cancelled, ca.root.created, ca.authority.imported, ca.intermediate_csr.issued, ca.intermediate.created, ca.authority.rotated, ca.authority.rekeyed, ca.cross_signed, ca.endentity.issued) is a tenant-scoped event recorded immutably in the tamper-evident log. Cross-signing uses a purpose-bound cross-sign:<ca-id>:<sha256-of-target-cert-der> ceremony: the signer-backed route signs inside the signer, while offline-root routes verify validity, key usage, EKU, DNS, path length, subject/public key, and both chain directions first. The full public cross-certificate is recoverable by ceremony ID from the event-projected GET /api/v1/ca/cross-signs/{id} receipt after a restart. The CA hierarchy console exposes effect-free review, quorum readback, execution, and that receipt for both signer-backed cross-signing and public-only offline-root cross-sign import. The offline import form accepts only public certificate PEM blocks; two distinct custodians approve the exact issuer, target and cross-certificate package. Preview and ceremony start verify the offline root's status, the public signature and the certificate constraints before requesting approvals; execution repeats the checks before recording the second trust path. Offline-root succession has a separate, indexed, tenant-scoped receipt at GET /api/v1/ca/offline-rekeys/{ceremony-id} containing the successor lineage and both public cross-certificates. It is projected from the same immutable re-key event as the authority state and can be recovered by ceremony ID after a lost response or restart. The Imports workspace in the console previews the exact public package, starts the bound two-custodian ceremony, reads the quorum, activates the successor, and recovers that receipt. Offline private keys never enter the browser or control plane. On a fresh browser session, the console labels a recovered completed receipt as a server result and says that the original public inputs were not compared in that browser. The full operator procedure is the CA key-ceremony runbook.

On upgrade, read-model catch-up checks older version-2 managed-hierarchy ca.endentity.issued events that were previously recorded only in the CA responder ledger. The hierarchy payload includes a subject field, even when empty; same-version responder-only payloads omit it and keep their original meaning. When the hierarchy event contains public DER and its exact fingerprint, trstctl verifies the leaf against the named issuing CA and recovers one stable certificate inventory row. The original event remains the mint; this does not sign a replacement or claim where the leaf's private key was held. A same-version responder-only payload keeps its serial meaning in the projection; if no other event inventoried its leaf, the first lifecycle sweep verifies that leaf against the managed CA's stored certificate and records a ca.certificate.inventory_recovered event, which gives it one inventory row that revokes against the same CA. Version-1 serial-only events cannot supply a public certificate for this replay. For those, use a supported certificate import to record an independently obtained public PEM; an import is an observation and does not by itself prove trstctl issued or can revoke that leaf. A failed replay leaves startup closed with an error rather than guessing a certificate or CA.

A CA key retirement checklist also accepts a narrowly reviewed disposition for a revoked, unserved host-held leaf. The release binds the immutable mint, completed local-issuer revocation event, public digest, and two distinct custodian approvals. The signer independently folds both events before accounting for the dependent. Event-only recovery rebuilds the completed revocation authority receipt from that retained event if the independent first-writer table is absent. This is a risk decision about possible external use, not proof that every relying party enforced revocation. See the key-ceremony runbook for the operator path.

For a still-served host certificate, CA succession uses a recorded re-protection plan and a retained-key CSR from the assigned host agent. This continues the same deployed subject and key under the direct successor CA. The identity's selected active certificate profile, or the configured default when the identity has none, constrains names, key strength, EKU and lifetime. Preview, registration, queueing, worker dispatch and signing recheck that policy. A later policy edit that excludes the certificate refuses the cutover; its allowed_protocols list governs new enrollment, so it does not turn this administrative continuation into API enrollment. The audit marks the profile decision ca-reprotect. The host's signed served-state report, independent TLS readback and updated retirement checklist are needed before treating the old CA dependent as cleared. The cutover event updates the identity's selected authority ID and displayed authority name together, using the tenant's successor CA record. A warm restart repairs names projected by older builds from the retained cutover events without changing a later CA selection or recreating an offboarded identity.

Recovering a failed CA-key retirement delivery

The CA Hierarchy lifecycle tab shows the exact frozen retirement command and its signer-delivery state. If delivery is failed after its attempt budget, review the failure and enter an audited reason to grant one more attempt. The retry uses the same tenant-bound command and outbox row, so it cannot silently alter the dependency evidence or sign a new destruction request. If the signer already handled the command before its response was lost, its durable journal returns the same signed refusal or destruction record. Read the checklist again to see the result; pending means the signer has not yet returned an accepted result. The CLI command is trstctl-cli ca keys retirement retry <id> -f retirement-retry.json; its body uses command_event_id, expected_attempts, reason, and confirm_irreversible, with an Idempotency-Key header.

Executing a CA migration in waves (H2)

CA rollover is served as a durable trust-before-leaf workflow, not as a batch loop. A run lists its cohorts explicitly. To build them from the estate instead of by hand, ask the read-only cohort planner (P3-L1): GET /api/v1/migrations/cohort-plan selects the deployed X.509 identities bound to an enabled deployment target by profile, by issuer_ca_id (the trstctl CA that issued each one's current certificate, the CA a planned switch revokes it through), by owner_id or by connector, and cuts them canary first: a canary of canary_size (1 to 10, default 1), then cohorts of cohort_size (1 to 500, default 25), in name order, each member with its target's host agent. Identities the selector matched but a run could not take (not deployed, no enabled target, a current certificate whose issuer cannot be read) are listed as skipped with the reason. A plan places at most 1,000 members and says when it was truncated; it records nothing. On the Migration page, Plan cohorts from a selector on the waves step does the same by profile or issuer, and Use these cohorts fills the waves for review.

trstctl migrations plan-cohorts --profile web-tls --canary_size 1 --cohort_size 25

Then run the read-only assessment:

trstctl migrations assess -f assessment.json

Then start a reviewed manifest with a fresh Idempotency-Key (the CLI supplies one unless you override it):

{
  "plan_id": "root-rollover-2026",
  "new_authority_id": "11111111-1111-4111-8111-111111111111",
  "waves": [
    {
      "id": "canary",
      "ordinal": 1,
      "members": [
        {
          "identity_id": "22222222-2222-4222-8222-222222222222",
          "agent_id": "33333333-3333-4333-8333-333333333333",
          "trust_anchor_path": "/etc/trstctl/roots/next.pem"
        }
      ]
    }
  ]
}

new_authority_id must name an active signer-backed hierarchy authority in the same tenant. The control plane reads its public certificate and freezes both the authority ID and anchor fingerprint into every member binding. Private keys never enter the manifest or control-plane process. When the trust receipt arrives, the successor is signed by that exact authority; a later CA rotation does not silently reroute an already-reviewed run.

Each agent_id must be the active host agent saved as required_agent_id on that identity's enabled deployment target. Another enrolled host in the same tenant is not interchangeable: Start rejects the mismatch before queuing trust or successor work. Keep the target assignment and manifest aligned when moving a workload to a new host, then submit a fresh reviewed run.

A cloud certificate store joins a run with no agent (A166). Where the tenant has no network relay enrolled, the control plane's own client carries the cloud stores, so a member whose target is one omits both agent_id and trust_anchor_path:

{ "identity_id": "44444444-4444-4444-8444-444444444444" }

The binding is frozen when the run starts: a network relay enrolled while the run is under way does not take the member (A166-L5). The control plane still installs the successor, reads the store back for the live gate and executes and reads back a rollback; only a new run must name the relay.

The control plane issues the successor under the new authority with the predecessor's exact identifiers, creates its fingerprint-named certificate and points the target's certificate map entry at it, then reads the entry back: the live gate passes only when the certificate the entry serves is the successor. Rollback re-points the entry at the predecessor and reads it back the same way. The store serving a certificate is not yet the load balancer serving it: the provider propagates a change on its own schedule. When the target's config names the endpoint its load balancer answers on (verify_address, with verify_server_name for SNI, and verify_private_egress_cidrs when that endpoint is on a private range), the gate also needs a TLS handshake there to present the certificate. Until it does, the member waits, checking every 30 seconds without spending its retries, for up to 30 minutes from when its row was queued; an endpoint still presenting another certificate after that fails the gate as a wrong answer would. Without verify_address the store's readback alone closes the gate, and the member's evidence in the console says so. A store that cannot be reached is retried. If it stays unreachable until the control plane stops retrying, the member fails its gate as though the store had answered with the wrong certificate: the run rolls the member back on its own, and if the rollback cannot reach the store either, the run halts and its halt reason names the member. Once the store answers again, roll the run back once (Review rollback in the console, or POST /api/v1/migrations/runs/{id}/rollback) to re-bind the predecessor. A GCP Certificate Manager target qualifies when it names certificate_map_entry and trstctl delivered the current certificate to it. AWS Certificate Manager re-imports in place and Azure Key Vault serves its newest version, so neither can restore a predecessor; Start refuses both with that reason. Once a relay is enrolled, cloud-store work runs on the relay: assign the target to it and name the relay as the member's agent_id. In the console, choose Control plane (cloud certificate store, no agent) as the member's executor.

The read-only assessment counts trust stores reported by that exact assigned agent. A deployment target's display name does not identify a machine. If a trust-store finding predates the receiver-stamped agent ID or its authenticated completed run, run inventory again from the enrolled host; the historical finding remains visible in discovery but cannot make a migration appear ready.

The phase order is fixed: install and read back the public anchor, issue against a host-generated CSR, deploy, handshake the configured listener, then release the next wave. Agent receipts are signature- and lease-verified before they can move a gate. Any failed receipt halts publication of later work and automatically starts the newest-first inverse. Every effect published for the failed gate is settled, including work that was already leased and reports after rollback began. pause prevents new publication while retaining receipts for already-leased work; resume reconstructs only the unfinished actions. The manual rollback control uses the same inverse: restore and verify predecessor leaves newest wave first, restore inventory truth from the same migration event, and remove successor trust only after the leaf inverse passes. A failed inverse halts with its durable attempt cursor; a later manual retry uses fresh outbox identities rather than replaying a terminal job.

Starting and resuming require both keys:write and certs:issue, because either operation can release a successor mint. Pause and rollback require keys:write.

Closed-loop proof (X3). A run started with "require_observed_traffic": true holds each wave at its live gate until two proofs agree for every member: the signed live receipt that the endpoint serves the successor (the probe, which also reads the chain a wire sensor cannot see on TLS 1.3), and an observer agent's report of new connections at the member's verify address (or its server name on that port) after the successor went live — presenting the successor on TLS 1.2. Behind NAT or a load balancer the observer sits beside the backends, so the member's deployment target can name them (X3-L2): "observed_addresses": ["10.0.3.17:8443"] in the target's configuration (host:port, at most 16, refused otherwise when the target is saved) is frozen into the run, and connections seen there, or under the member's server name on one of their ports, count as the member's. The leader checks every 30 seconds, reading every one of the tenant's observer TLS findings a page at a time and keeping only those at the endpoints it is waiting on (X3-L4), so no number of findings elsewhere hides the one that decides. A member still unobserved observed_traffic_window_seconds (default 3600, at most 604800) after going live pauses the run with the reason; the run never advances on the probe alone, and a resume releases the gate only once the wire evidence is there. Missing wire evidence is never recorded as a failed member. When a member has both proofs, the observer's findings for its predecessor certificate at that endpoint are marked remediated, naming the live receipt and the observed connections.

Wire evidence for every replacement (X3-L1). A plain renewal's deployment, or any other replacement, waits for the same wire evidence without a migration run. When a verification of an endpoint (GET /api/v1/endpoints/verifications, Connectors → Health, retries, and rollback) finds it serving another certificate than it last observed there, the endpoint records which certificate was replaced and when, and its traffic reads awaiting. On the same 30-second leader pass, an observer agent's new connections at the endpoint's address, or at one of its target's observed addresses (X3-L2), after the replacement — presenting the replacement on TLS 1.2; on TLS 1.3 any new connection, since the verification already proved the certificate — move it to observed, naming the observer finding (traffic_finding_id), and mark the observer's findings for the replaced certificate at that address remediated, naming the endpoint, the replacement and the observed connections. Nothing closes on the probe alone. An endpoint still unobserved an hour after its replacement reads unobserved from then (traffic_unobserved_since); later evidence still closes it. The Connectors page shows each endpoint's wire evidence.

Planned CA switch (P3). A run started with "revoke_predecessors": true (on the Migration page, "Revoke each predecessor once its successor is verified live") moves its members to the new issuer and retires what they served before. Each cohort runs the same gates — the new root distributed and read back first, then the successor issued and deployed, then the signed live receipt that the endpoint serves the new chain (and the observed-traffic proof when the run requires it) — and only then is each member's previous certificate revoked, reason superseded, through the trstctl CA that issued it, which publishes its CRL before the cohort counts as done. A switch away from a vendor CA (P3-L2) revokes a previous certificate the external CA registry issued at that vendor, through its integration (by serial, or from the certificate for an ACME authority): the revocation is recorded, and the member counts as done, only once the vendor accepted it. A vendor that holds the request for an approver leaves the member waiting and the request is sent again until it is accepted; a vendor whose integration cannot revoke pauses the run with the reason. The assessment names the vendor that would revoke each predecessor (external_ca_id, ca_name), and the cohort planner's issuer_ca_id selects by an external CA too. A switch can also move to a vendor CA (P3-L2): start it with new_external_ca_id (an available CA from the external CA registry) and new_trust_anchor_pem (that vendor's root) instead of new_authority_id, or choose it under Vendor CAs on the Migration page with its root pasted. Every host member trusts and reads back the vendor root first; then the host generates its key and the vendor signs its certificate request, the endpoint is verified serving the vendor's chain, and the predecessor is revoked as for any switch. Only host members can move to a vendor, since the vendor signs a request the host makes. A member whose current certificate neither a trstctl CA nor a vendor integration can revoke is refused at start rather than left valid. The first cohort is the canary: a failed trust readback or live check in any cohort pauses the run, naming the member, with nothing revoked and no later cohort started. Resuming retries exactly the failed members as the same jobs (the host agent reuses the key and certificate it kept for that job, so no second successor is signed); rolling back restores the current cohort, never a cohort whose predecessors are already revoked. Starting the same switch again with the same Idempotency-Key returns the existing run and changes nothing.

Before starting, assess the switch as one (P3-L4): POST /api/v1/migrations/assess with "revoke_predecessors": true lists, member by member, the certificate the switch would revoke (its id, serial and fingerprint) and the CA that would revoke it, read by the same rules as the start. A member whose current certificate no trstctl CA can revoke is listed with the reason and is an unknown (predecessor_not_revocable) that blocks the start. The Migration page assesses a switch this way and shows Certificates this switch revokes in the review; an assessment that was not a switch's does not open Start migration for one. Assessing changes nothing.

trstctl migrations start -f run.json
trstctl migrations list
trstctl migrations show RUN_ID
trstctl migrations pause RUN_ID
trstctl migrations resume RUN_ID
trstctl migrations rollback RUN_ID

The REST equivalents are POST /api/v1/migrations/runs, GET /api/v1/migrations/runs, GET /api/v1/migrations/runs/{id}, and the three action routes below that run. The executable boundary is X.509 identities deployed by an enabled host-agent connector with a configured verification listener, by a network appliance through its relay, or into a cloud certificate store through the control plane; the successor asserts exactly the predecessor's names and addresses. The new authority may be a signer-backed CA or an imported offline root; for an offline root the successors are issued by its one active signer-backed intermediate and the root certificate is the anchor distributed to hosts. Every unsupported membership is rejected, with its reason, before any trust job is published. The run pins each identity's active certificate-profile revision and effective validity into the immutable member and successor job. A member whose profile requires a distinct issuance approval makes the start itself the approved operation: the first start answers 202 with an approval request, reviewers approve it by intent digest, and the same start posted again records the run with that single-use approval consumed. Incident fleet reissuance still refuses an approval-required profile before Start. A profile label alone never authorizes signing.

Profiles and the registration-authority split (F53)

A certificate profile is a versioned, tenant-scoped rulebook: allowed key algorithms and minimum sizes, extended key usages, maximum validity, DNS suffixes, and protocols. Editing a profile creates a new version; old versions stay queryable, so you always know which rules a past certificate was issued under. On every issuance, enforceProfile fetches the active version, validates the request, and emits an audit event for the allow-or-deny decision. A leaf's extended key usages come from its profile, never from the CSR alone: a CSR may ask only for usages the profile allows, a profile that lists none allows serverAuth and clientAuth, and no leaf ever carries ocspSigning or anyExtendedKeyUsage. An ocspSigning leaf from an issuing CA would be a delegated OCSP responder for every certificate that CA issued; trstctl's own responder certificate is made separately. A CA hierarchy's any constraint is not copied onto its leaves either. The platform issuing CA also never signs a timeStamping leaf: its own TSA certificate, minted separately, anchors the audit trail, so a timestamping leaf comes from a tenant-managed CA. For an identity lifecycle issuance, preview and execution check the pinned profile's known protocol, validity, and subject-name rules before recording identity.issued. The signing worker still checks the actual CSR, including key strength and requested usages, before signing. A ready lifecycle preview therefore cannot turn an ACME-only profile into a queued API issuance.

Each AD CS issuer's health is on Posture → AD CS issuers (and GET /api/v1/adcs/issuers/health): healthy, degraded, failing or idle over the last seven days, from the certificates it issued and the refusals trstctl diagnosed, with the latest refusal's cause (for example the template's permissions not allowing the requester to enroll) and what to do about it.

A profile can also choose who signs. With upstream_external_ca set to a configured external CA's registry ID, ACME, EST, SCEP and CMP enrollments under that profile pass the profile's checks in trstctl and are then signed by that CA instead of trstctl's own: for example ACME in front of an AD CS template, with one adcs external CA entry per template. The client receives the upstream's leaf and chain, and the certificate is recorded in inventory like any other. Saving a profile that names an external CA the tenant does not have is refused (422). An ACME revoke of an upstream-issued certificate is refused, because trstctl's CRL covers only its own CA; revoke it through trstctl's revocation (the Revocation center or POST /api/v1/certificates/bulk-revoke), which goes to the upstream CA.

Revoking at AD CS. Web Enrollment has no revoke call, so an adcs issuer revokes through a domain-joined network relay named in its revocation_relay. A revocation of a certificate that issuer issued is handed to that relay, which runs certutil -revoke with the reason, publishes a new CRL with certutil -CRL and reads the CA database row back; trstctl marks the certificate revoked only when the relay reports the row revoked and the CRL published. A relay that has not answered yet leaves the revocation waiting (it does not count against its retries); a failed attempt is retried on the revocation's own backoff. AD CS takes the reasons unspecified, keyCompromise, cACompromise, affiliationChanged, superseded and cessationOfOperation. With dual control on (require_approval), a revocation waits for its approvals before the relay is asked, like every other revocation.

Approving and denying held AD CS requests (N1). A template that requires CA manager approval holds a request as pending. Through the same relay, trstctl approves or denies it at the CA — behind its own approval, never on one person's word. POST /api/v1/external-cas/{id}/adcs-requests/{request_id}/approval-requests asks for an exact decision (approve or deny, with a reason); reviewers who may issue approve it under /api/v1/approval-requests (the requester cannot approve their own); then the requester executes exactly what was approved with POST .../decisions, which spends the approval on one relay job. The relay reads the request's row first and acts only while the CA still holds it as pending — so an approval can never re-issue a request a manager already denied — then runs certutil -resubmit (approve) or certutil -deny and reads the row back. GET /api/v1/external-cas/{id}/adcs-requests lists the requests the issuer's CA holds, newest first, from the relay's latest read of its database (adcs.ca_database, N1-L2): each one's request id, template, requested name and requester, and whether a trstctl issuance is held on it. truncated says the CA held more than the newest 500, and read_at says when it was read; a later read replaces the list. The CA Hierarchy page shows the same list under each AD CS authority. CLI: external-cas adcs-requests. GET /api/v1/external-cas/{id}/adcs-decisions lists each decision as queued, decided with the CA's disposition (and, for an approval, the issued serial), or failed with the relay's reason. Status: served.

Resuming an issuance the CA held (N1-L1). An issuance through an AD CS template that requires CA manager approval waits for the request for 60 polls (two minutes at the default poll_interval), then gives up with the request still held. trstctl records the CA's request id with the issuance. GET /api/v1/external-cas/{id}/held-issues lists each one as held, issued or denied, newest first. Once a CA manager has approved the request, POST /api/v1/external-cas/{id}/held-issues/{request_id}/resume reads it back from the CA by that id. trstctl then records the certificate under the held issuance's own idempotency key. The issuance's caller gets the certificate by replaying its request, and the issuance's delivery finishes without contacting the CA again. While the CA still holds the request, resuming answers 409 and records nothing; a denied request resumes as denied. Reading a request back cannot issue a second certificate, so resuming again is safe and gives the same answer. CLI: external-cas held-issues and external-cas held-issues resume. When the issuer's relay approves a request a trstctl issuance is held on (N1), recording that decision queues the resume in the same transaction, so the issuance finishes on its own. A manager's approval made at the CA itself is resumed by an operator. The CA Hierarchy page lists each AD CS authority's held issuances (request id, names, when they stopped waiting, and whether they were issued or denied) with Resume. Status: served.

The registration-authority (RA) model is a role split that prevents the classic PKI abuse of one person approving and fulfilling their own request. The built-in ra-officer role can read/write profiles and request certificates but doesn't hold certs:issue — only an operator/admin can issue. The split is enforced by RBAC, not convention, and a test asserts it. Authoring profiles is covered in the certificate-profile guide.

The self-service requester path is served end to end for X.509 requests. /request lists active profiles and tenant-visible owners, makes the requester choose the accountable owner by name, and sends the exact proposed body to the effect-free POST /api/v1/issuance-requests/preview route. The same server admission rule used by submission checks the owner, validates an optional public CSR, and resolves an active profile name to its exact immutable version. The answer names blockers, key-custody posture, independent approval authority, and the writes that a later submission would perform. Preview itself appends no event, writes no projection, creates no idempotency record, and contacts no CA. The console fails closed: it enables submission only while a green preview still matches every reviewed field. Operators and automation can ask the same question with trstctl issuance-requests preview --body request.json.

Once the exact approved issuance has a recorded certificate, the signing worker marks the request fulfilled without requiring the requester to keep a browser open. The completion API can also reconcile a historical result after deployment, revocation, or retirement. This changes only the request receipt: the credential's current state stays intact. issued_by comes from the original issuance event (and remains empty when that historical event has no actor); issued_at is the certificate's retained recording time. Missing or ambiguous signing evidence, a different owner or profile, and an unrelated issuance key still refuse completion.

Submission sends the normalized body to POST /api/v1/issuance-requests. The server re-runs that shared admission rule before appending issuance.request.opened; missing or malformed values return 400, while absent and other-tenant owner UUIDs return the same 422 answer. The console links directly to Profiles, Owners, and CA hierarchy when a prerequisite needs configuration. This is important: oidc|dev-1 names an authenticated caller, while an owner UUID names a tenant database row. They are different identifiers and are never substituted for one another.

The requester's /request history and the /approvals inbox read the same first-class issuance-request projection. A submission therefore remains requested; it does not create an identity or mint a certificate. A distinct principal can approve or deny the exact request, and approval is still not issuance: the request becomes issued only after the authorized issuance outcome is linked. If issuance is interrupted after approval, the approved row remains visible and the console offers Retry safely. Preparation reuses the request's deterministic identity and stable issuance key, so retry continues the same operation instead of minting a second certificate.

The built-in ra-officer can read owners, author/read profiles, and request certificates, but still cannot write identities or hold certs:issue.

Profile recovery is served as an append-only operation. POST /api/v1/profiles/{name}/versions/{version}/restore/preview returns an effect-free receipt for copying a known-good historical spec into the next version. The receipt pins the current active version, semantic spec digest, reason, request fingerprint, issuance risks, and proof steps; it emits no event and performs no external call. POST /api/v1/profiles/{name}/versions/{version}/restore rechecks that active-version fence under the projection lock, creates one new event-sourced active version, and leaves every historical row untouched. Reusing the idempotency key returns the same result. A newer concurrent version fails closed with 409, and profiles governed by requires_approval retain non-requester dual control. The console exposes the same flow after a version diff; the CLI commands are profiles restore-preview and profiles restore. See the profile-authoring guide for a complete example.

A profile create or edit that falls under requires_approval answers 202 with an approval_id and state: awaiting_approval instead of a new version. A distinct reviewer holding profiles:write finds the parked request with GET /api/v1/profiles/approvals (or GET /api/v1/profiles/approvals/{id}) and reviews the exact proposed spec, its spec_sha256 digest, requester, expiry, and any restore source/reason in the console's Requests waiting for approval page or in the API/CLI. The console counts these requests alongside other pending approvals. A reviewer with only profiles:read and profiles:write can use this inbox without access to certificate, secret, or managed-key decisions; the console disables the decision for the requester. Approval takes effect immediately: the reviewer confirms it with POST /api/v1/profiles/approvals/{id}/approvals; quorum (one non-requester approval) applies the queued spec as the new active version and the record reads state: issued with the created profile_id. The requester's own approval is refused with 403 (dual control). Identical retries of the approval with the same Idempotency-Key replay the one decision. The parked request is event-sourced (profile.edit_approval.requested / .approved, projected into profile_edit_approvals), so it survives a control-plane restart and is visible to every replica; it expires 24 hours after the 202 (state: expired, approval answers 409) and the requester resubmits the change to open a new request. The digest is SHA-256 of the compact JSON spec bytes returned by the review record, so an API or CLI reviewer can verify what was displayed before deciding.

Telling clients when to renew: ARI (F46)

If thousands of clients renew at the same fixed "30 days before expiry," they stampede — and if a certificate must be replaced early (a mass revocation), there's no way to tell them. ACME Renewal Information (ARI, RFC 9773) fixes both: the CA publishes a suggested renewal window per certificate, and clients renew within it.

trstctl's ACME server computes the window as the last third of the certificate's life. An ACME client can spread its renewals across that window. If trstctl's CA flags a certificate for early renewal, its published window opens immediately; clients that poll ARI can act on that signal.

The local window is served by trstctl's ACME server at GET /acme/renewal-info/{certid} and used by its lifecycle scheduler for deployed X.509 identities. Certificates can renew when that local window opens, even before the fixed renew_before fallback. For certificates issued by an external ACME CA through trstctl, the control plane polls that CA's renewalInfo and the scheduler uses the retained window. A leaf that arrived through discovery has no authenticated issuer; set ari_issuer_key_ids on the letsencrypt-type authority to the Subject Key Identifiers of its issuing intermediates and a discovered leaf whose Authority Key Identifier matches is polled the same way, with its posture labeled as matched by issuer key id rather than by an issuance fact.

Operators inspect the same chain through the read-only GET /api/v1/acme/ari/posture route, the trstctl-cli acme ari posture command, or the ARI posture panel on Protocols. The authenticated route requires lifecycle:read and PostgreSQL RLS limits every certificate and rotation-run row to the caller's tenant. It reports whether ARI publication is served for that tenant, the locally computed or published suggested window for each affected certificate, and whether the lifecycle scheduler is pending, running, succeeded, or failed for that window. It never returns certificate bytes, fingerprints, tenant IDs, account/order data, or private-key material.

The public ACME route and the operator route answer different questions: /acme/renewal-info/{certid} tells an ACME client when it should renew; /api/v1/acme/ari/posture tells an authenticated operator what is being published and whether trstctl's scheduler consumed it. If ACME is not mounted for the tenant, the posture says not_served rather than pretending the certificate is published. An empty items array honestly means that the tenant has no affected deployed certificate rows.

Revocation: OCSP and CRLs (F47)

When a certificate must stop being trusted before it expires, you revoke it and publish that fact two ways. A CRL is a signed list of revoked serials, regenerated periodically; OCSP answers "is this one revoked?" live, one certificate at a time. For its own hierarchy trstctl does both: Revoke(serial, reason) emits ca.certificate.revoked to the tamper-evident log, and GenerateCRL bumps the CRL number, signs a fresh list through the isolated cryptography path, and emits a v3 ca.crl.published event (CRL DER, artifact kind, shard metadata, delta base, validity window) so CRL state rebuilds from the log. Small estates use the plain /crl/{tenant} full CRL; at scale the same publication also serves /crl/{tenant}/manifest.json, /crl/{tenant}/shards/{index}, and /crl/{tenant}/delta/{base}, so relying parties fetch bounded or RFC 5280 delta CRLs instead of a 10-100M-row monolith. The OCSP responder signs with a delegated responder certificate (OCSPSigning EKU + ocsp-nocheck) rather than the CA certificate; rotations emit ca.ocsp_responder.rotated, and the responder runs in its own bounded lane so a flood can't starve the API.

RFCs 6960 (OCSP), 5280 (CRL).

Revocation is typed and batchable: requests use an RFC 5280 named revocation reason such as keyCompromise, cessationOfOperation, or privilegeWithdrawn (unknown raw integers are rejected), and bulk revoke at /api/v1/certificates/bulk-revoke / /api/v1/identities/bulk-revoke returns matched, revoked, skipped, and failed counts so a wide incident response is explicit about partial success. OCSP responses echo a valid OCSP nonce when the request carries one and sign with the delegated responder. Each response is valid for ten minutes; a nonce-free response is reused only in the first half of that lifetime, then signed again, so no relying party is handed one about to expire. If the certificate route's durable event-ID index is catching up, it returns 503 with Retry-After: 1; repeat the exact request with the same Idempotency-Key rather than creating a second revocation command. CRL serving returns weak ETag validators and honors If-None-Match with 304 Not Modified so relying parties don't refetch an unchanged CRL. GET /api/v1/revocation/crls / trstctl-cli revocation crls and the Certificates console expose the same distribution state (full CRL, shards, delta base, freshness window).

Bulk identity revocation requires both identities:write and the privileged certs:issue authority. Every selected identity must pass the lifecycle policy and attribute-based restrictions before any selected identity is changed. A denial is not partial success: no revocation intent is queued for the selected batch. When dual control is required, the bulk request cannot supply the exact one-use approval for each identity and is refused. Use the individual reviewed revocation workflow below; a standing approval does not authorize a bulk action.

To select actual inventory certificates rather than lifecycle identities, send certificate_ids with 1–100 UUIDs and a named reason. An empty array is invalid. Do not combine this field with ids, identity_ids, owner/issuer filters, kind, or status. The same authorization, policy and approval restrictions apply, even when no certificate matches. removeFromCRL is an unhold operation, not a valid reason for this exact-certificate revocation command.

The exact path verifies the public certificate's signature against the served issuing CA and requires an existing tenant-scoped issuer record. It does not guess from an issuer name or serial, revoke same-owner siblings, or substitute the internal CA for an external issuer. For external leaves, immutable issuance evidence binds the exact public certificate to its configured CA adapter. The request queues that authority's revocation command through the outbox. Its item status is queued, counted in total_queued; it is not counted in total_revoked. Missing verified authority or an unavailable adapter returns an explicit per-item unsupported result. An adapter that cannot revoke through trstctl also fails that exact item before an upstream outbox command is queued. HTTP 200 can contain queued or failed items, so always inspect the returned counts and items.

One retained certificate.revocation.batch.applied command records the exact selection and outcome. Its projection updates certificate inventory, the CA's revocation record and a revocation.crl.publish outbox intent in the same database transaction. If inventory and issuer disagree, it reconciles them while preserving the issuer's first revocation time and reason. revoked therefore includes a reconciled record; it is not a count of newly revoked serials at the CA. skipped means inventory and the issuer already agree that the certificate is revoked. External items use event schema version 2 and a revocation.publish outbox intent. They change certificate status only after their recorded issuer accepts revocation; their serials never enter the platform CA ledger or CRL. Issuer errors retain the command for retry and leave revocation unconfirmed. The console automatically checks the selected certificate; GET /api/v1/certificates/{id} provides the same current result. An idempotent replay still returns the original queued receipt.

Reuse the same Idempotency-Key for the same authenticated route and request. Retries recover the original result from retained events even after the HTTP result cache expires or an append succeeds before the database transaction fails. Changing the selection or reason under that key returns 409. CRL publication is asynchronous: a committed result is not proof that publication or a relying-party check succeeded. Follow the delivery outcome, verify signed CRL/OCSP data, and test a client configured to enforce revocation before declaring containment complete.

The local CA denies a Good OCSP answer as soon as an exact revocation command is accepted, even if its event append or SQL projection must be recovered after a crash. CRL readers likewise refuse an older signed list while that command is pending or while the current issuer ledger contains more revoked serials than the latest full signed CRL. The outbox worker publishes the replacement; an unavailable CRL during this window is an explicit distribution failure to monitor, not evidence that a relying party has rejected the leaf. A revoked or retired identity also stops queued local signing, deployment, and host renewal work from receiving fresh authority while its terminal event is reconciled. Certificate list and detail responses expose revocation_pending; identity list and detail responses expose terminal_pending and the accepted terminal_target_state. The console labels these pending states so an operator can distinguish accepted authority removal from completed projection.

For one managed certificate, Certificates → Revocation & CT → Revocation center is the safe operator path. It is a three-step journey: choose the X.509 identity and factual RFC 5280 reason; fetch an effect-free server preview; then type the exact credential name to execute. The preview is bound to the identity's current lifecycle version and request fingerprint. It explains the event, same-transaction outbox intent, asynchronous revocation.publish work, graph-derived affected systems, and verification steps while writing no event and contacting no external system. Execution echoes that reviewed version, so a certificate that changes after review fails with 409 and must be reviewed again. After success, the center links directly to the immutable identity.revoked audit evidence and the affected-systems graph. The lifecycle event and outbox remain the only mutation authority; the console does not invent a second browser-only revocation path.

For reason keyCompromise, a serving managed X.509 identity takes the compound response in deployment connectors. The preview binds the issuing CA and served leaf before one confirmation queues separate CA revocation and host containment. The console reads both statuses by the original request key; revoked is only the lifecycle projection and a queued host receipt does not mean the listener stopped. Check each certificate serial through signed CRL/OCSP and verify the live host with a fresh stock TLS client. The ordinary single-effect transition is refused for a serving X.509 keyCompromise even if target metadata is missing: its CA-only success would leave endpoint containment unproven. It remains available for other factual revocation reasons and for an X.509 identity that has not entered a serving state. If no exact managed host target exists, revoke the exact certificate with its issuing CA and stop the host through an independently verified procedure.

A key is recorded as compromised in the tenant from the moment the compromise is reported, not from the moment the revocation publishes. Revoking an identity with reason keyCompromise (the ordinary transition or the compound response) records, in the same transaction, the keys of the certificates that revocation will revoke, one certificate.subject_key.compromised event per certificate, before revocation.publish runs. An exact certificate revocation for keyCompromise that is queued at its external CA records the key when it is queued, whatever the CA answers later, and a certificate revoked for keyCompromise directly records its key as the revocation is recorded. From then on no certifying path in that tenant will certify the key again (CA/B BR 6.1.1.3, condition 4). The ACME, EST, SCEP, CMP, REST, Vault-compatible PKI, SPIFFE, agent enrollment, CA authority, break-glass and external CA paths refuse a request for that key with the subject key is recorded as compromised in this tenant, so the replacement must use a new key. The check stays inside the tenant; another tenant's records are never consulted. Each record is the certificate's SubjectPublicKeyInfo SHA-256, and each issuance makes one indexed lookup by tenant and hash. There is no cap, so a mass keyCompromise revocation never blocks the replacement issuance it calls for. At startup the server records the keys of revocations an earlier build projected. If the server stops between an identity's revocation and its key records, the keys are recorded when the revocation publishes instead. A certificate with no retained DER has no key to record, and once a key is recorded as compromised it stays recorded, even if the CA's revocation later fails.

For an X.509 identity, the preview checks the exact unrevoked certificates tied to its issuance or verified deployment, including superseded leaves that could be restored. If their recorded issuing CA cannot revoke through this integration, ready is false and the warning names the certificate ID and directs the operator to that CA. An identity transition can still stop local automation, but an identity.revoked state does not prove that the external CA revoked any leaf. Confirm the exact serial upstream and verify signed revocation evidence before calling the incident contained. The preview checks one bounded page of certificates; a selection above 100 is marked not ready for complete verification.

External distribution health is monitored separately from trstctl's own publication state. An hourly leader derives distinct CDP and AIA OCSP URLs from the certificate inventory and queues bounded revocation.probe work for a network relay, so private PKI endpoints are checked from the network that uses them. An operator with certs:write can also probe now (POST /api/v1/revocation/health/probe, trstctl-cli revocation health probe, or Probe endpoints now in the console), for example right after fixing a relay's trust bundle; requests inside one minute share one probe. The relay verifies CRL/OCSP signatures and exact certificate context, then returns a signed lease-bound report. GET /api/v1/revocation/health and Certificates → Revocation & CT expose fresh, expiring, stale, unreachable, and invalid answers with latency, nextUpdate, relay identity, and evidence digest; non-fresh answers create notifications. With agent_channel.probe_scanned_certificates on, a certificate known only from a relay's network scan or a cloud listing is also checked for its own serial, against the issuer the scan saw served with it or the provider returned with it (else a known issuer that signed it); its certificate detail shows those answers (whether the CRL lists the serial, the OCSP status) with the relay and evidence, and the endpoint list stays one row per endpoint. An answer is expiring when its nextUpdate falls in the last quarter of its own signed lifetime, capped at 24 hours: a seven-day CRL warns in its last day and a ten-minute OCSP response in its last two and a half minutes, so a responder that keeps signing short-lived answers reads fresh. A missing observation is shown as unknown, never healthy. This proves the relay's observation, not every relying party's fail-closed or soft-fail configuration. Controlled CRL/OCSP responders cover repository CI; a real Windows AD CS lab run remains external infrastructure evidence.

Dark-segment serving is a separate local-cache path. A network-role agent can load a bounded --revocation-cache-config with multiple issuers and explicit local CRL and OCSP routes. It verifies CRL issuer signature, number, and signed time window before caching. For OCSP it also binds the request to the configured issuer and validates the responder signature, requested serial, nonce, thisUpdate, and nextUpdate. Only nonce-free answers are reused, and stale or invalid objects produce 503 with no protocol bytes. Each mTLS heartbeat signs metadata per segment/issuer/protocol; GET /api/v1/revocation/caches and Protocols → Revocation cache by segment expose fresh, stale, empty, and error without exporting upstream URLs or cached objects. An unreported relay is unknown, not healthy.

CT submission is served at POST /api/v1/revocation/ct-submissions / trstctl-cli revocation ct-submit: the outbox queues a precertificate and final certificate to configured RFC 6962 CT logs, recording ct.submission.queued then ct.submission.delivered after add-pre-chain/add-chain. Name each log's public key in log_public_keys and every SCT that log returns is verified against it and kept (ct.sct.recorded: verified, or invalid with the reason); the leader then proves the entry is in the log by checking an audit path against a tree head the same key signed (ct.inclusion.recorded: verified with index, tree size and root; pending until the log merges it; not included once a tree head signed 24 hours after the SCT still lacks it; or a failed proof). A log without a key keeps its SCT unverified, and a log that serves only the static CT API is not checked for inclusion. The certificate detail (GET /api/v1/certificates/{id}, transparency) shows each log's answer.

Rogue and non-compliant certificate posture is served at GET /api/v1/revocation/rogue-certificates / trstctl-cli revocation rogue-certificates and the Certificates console: the tenant's own certificate_posture rules from its active policy module (see the policy engine in policy and governance), unexpected CT findings from monitored logs combined with policy violations such as weak keys, expired active certificates, over-long public-TLS lifetimes, and missing owners/issuer metadata — metadata and projection references only, never certificate PEM or private-key material.

Revoke a rogue certificate at its issuer. On the Certificates page, choose a revocation reason and use Revoke at issuer on a finding. A certificate in inventory goes through the exact revocation, which reaches its recorded issuer. A Certificate Transparency finding — a certificate trstctl never asked for, such as one issued in your own vendor account outside trstctl — is revoked by serial at the external CA you name; CAs whose revocation takes a serial are offered (DigiCert, Sectigo, AWS Private CA, Google CAS, step-ca, EJBCA, Vault PKI), and so is an ACME CA (N10). Each authority in GET /api/v1/external-cas says which way it can revoke a certificate trstctl found (revokes_by_serial, revokes_by_certificate; both false while it is unavailable), and the console offers authorities from those. An ACME CA identifies the certificate by its bytes, which trstctl never held: trstctl reads them back from the Certificate Transparency log that recorded the finding (through that CT source's own egress settings), checks them against the finding's fingerprint and serial, gets its own ACME account authorized for every DNS name in the certificate through the issuer's DNS-01 provider (an order it never finalizes), then asks the CA to revoke, as RFC 8555 §7.6 allows an account authorized for all of a certificate's names. A finding at a precertificate entry revokes its issued certificate (N10-L1): trstctl looks for another CT finding of the tenant's under the same serial whose log entry is the issued certificate from the same issuer, reads that entry back from its own log the same way, and sends it. Until a watched log has recorded the issued certificate, the revocation waits with that reason. An entry that is not the finding's certificate, a log the source no longer names, or a certificate with an IP identifier is refused without contacting the CA. The finding then shows the revocation queued, and revoked once the CA confirms; a CA that has no certificate with that serial, or that refuses the account, leaves it unconfirmed with the reason. Automation uses POST /api/v1/revocation/rogue-certificates/revoke with discovery_id, authority_id and reason (identities:write plus certs:issue), or trstctl-cli revocation rogue-certificates revoke -f req.json.

Where the private key lives: HSM/KMS (F26)

A CA's private key is the system's single most valuable secret — anyone who has it can forge any certificate, so trstctl keeps it in hardware or a cloud key service that signs without revealing it. An HSM/KMS backend implements one interface (Backend → GenerateKey → a Signer that signs via the device); trstctl supports PKCS#11 HSMs, TPM 2.0, YubiHSM 2, AWS KMS, Azure Key Vault, and GCP Cloud KMS. Adding one is a single change because all cryptography goes through one isolated path: key material never leaves the device — it lives in a separate isolated signing service, wipeable in memory, and only signatures/public keys cross the wire. Every backend must pass a conformance harness (ConformBackend) that signs a probe, verifies it, and confirms a wrong message and a tampered signature both fail.

The release includes a dedicated cgo HSM signer artifact: its PKCS#11 adapter opens the configured native module using stable token CKA_ID values across restarts, TPM 2.0 uses google/go-tpm persistent handles, and YubiHSM 2 uses Yubico's yubihsm_pkcs11 ABI. The launched-binary gate proves SoftHSM/swtpm lifecycle behavior with independent command-line readback; the default control-plane artifact stays static and never loads a native module, keeping provider credentials and private-key operations inside the separate signer.

CAP-KEY-05 — Multiple algorithms (RSA / ECDSA / Ed25519) + PQC — is served: the profile path is POST /api/v1/profiles and trstctl-cli profiles create -f profile.json. The profile API validates allowed_key_algorithms through internal/crypto, then stores the accepted policy as profile.created evidence. It accepts classical RSA, ECDSA, and Ed25519 labels and, with PQC in the core since 2026-09-20, the PQC signature labels Hybrid-ML-DSA-44-ECDSA-P256, ML-DSA-65, and SLH-DSA-SHA2-128s. Unknown labels fail closed, and ML-KEM stays out of certificate-signing profiles because it's a key-encapsulation mechanism, not a signing algorithm. internal/server/crypto_agility_served_test.go's TestServedCryptoAgilityProfilesValidateBoundaryAlgorithms proves the served profile create/list round trip; the PQC issuance proofs live under internal/pqc and internal/pqcmigration.

The managed-key API spine is configuration- and license-gated for AWS KMS, Azure Key Vault/Managed HSM, GCP Cloud KMS, PKCS#11, TPM 2.0, and YubiHSM 2 custody: once managed_keys.enabled is true and managed_keys.provider selects aws, azure-key-vault, gcp-kms, pkcs11, tpm2, or yubihsm2 (or managed_keys.providers lists several to run at once, with optional per-tenant pins), the control plane exposes:

  • GET /api/v1/managed-keys/custody — return a secret-free startup plan for all six providers, the default and every running provider, the caller tenant's pin, signer attachment, and exact blockers;
  • POST /api/v1/managed-keys/preview — validate provider and algorithm without writing state or calling the signer/provider, then name the later execution effects and verification evidence;
  • POST /api/v1/managed-keys — create a non-extractable KMS/HSM-resident signing key (extractable: false; no private material returned);
  • GET /api/v1/managed-keys — read the tenant's event-projected key inventory with bounded cursor pagination, including keys from providers used before a deployment configuration change. A cloud KMS key (AWS KMS, Azure Key Vault/Managed HSM, GCP Cloud KMS) carries the provider's own dates (A168-L3): provider_created_at, when the provider created the key, and provider_deletion_at, when it will destroy the key's material once deletion is scheduled. The isolated signer reads them from the provider after each lifecycle action (DescribeKey; the key version's createTime and destroyTime; the key's created attribute and the deleted key's scheduledPurgeDate) and records them with the result — a provider that cannot be read leaves the action to be retried — so trstctl's record holds the provider's dates rather than the moment its own operation completed (created_at). A custody proof records the creation date of a key created before this release; a result that reports no dates never erases recorded ones. The console's managed-key inventory shows both;
  • GET /api/v1/managed-keys/{provider}/{key_id} — read the current state and public material of one exact provider/handle pair after a lost response or restart;
  • GET /api/v1/managed-keys/operations/{operation_id} — read a tenant-scoped provider command's durable queued, completed, or failed state. A mutation still running after its short synchronous wait returns 202 with this opaque ID and status URL, also when a read inside that wait runs out of its time; replaying the same idempotency key returns the same receipt;
  • POST /api/v1/managed-keys/verify-custody — ask the isolated signer to sign a fresh random challenge with the exact owned key, then verify the signature against the saved public key. The response and inventory show the last proof status and time separately from lifecycle state. A historical active state does not prove the provider still has the key; a new key reads not_checked until this challenge succeeds. The console's Refresh status updates both the list and the selected key. When provider retries are exhausted, the key reads unavailable while its lifecycle state remains historical; new failure audit events include the exact provider and action, plus the key ID when one already exists. A failed generation has no key ID to name. Older v1 failure events remain searchable by their operation ID;
  • POST /api/v1/managed-keys/approvals — record a distinct custodian's approval for an opaque key handle and rotate/revoke/zeroize;
  • POST /api/v1/managed-keys/rotate — mint a successor key;
  • POST /api/v1/managed-keys/revoke — disable the current key at the provider;
  • POST /api/v1/managed-keys/zeroize — destroy a local device key, or request provider-side deletion for a cloud key. Cloud KMS responses and inventory read deletion_pending after the provider accepts the request. That state means signing is blocked; it does not claim the provider has erased retained key material. Read the provider's deletion deadline with its stock client. Local PKCS#11, TPM, and YubiHSM keys read zeroized after confirmed device removal. Older cloud events that reported zeroized remain immutable; the API and console present their historical projection as deletion_pending rather than making an unsupported physical-destruction claim.

The Certificate authorities → Key custody console turns those first two routes into a three-step ELI5 journey: choose a provider and algorithm, review proof that nothing changed, then generate and manage the key. It lists configuration variable and file-reference names but never accepts provider credentials, file contents, or private-key bytes in the browser. A provider mismatch, disabled lifecycle, missing signer attachment, non-effect-free preview, or any server blocker keeps generation locked. The inventory remains visible after a browser refresh; selecting a key performs a fresh tenant-scoped read. Keys from a provider no longer attached to this deployment remain visible for audit, with actions disabled until that provider is restored. For a 202 receipt, the console keeps the selected key visible and shows the operation ID. It checks for a terminal result while the tab is visible and refreshes on return; Refresh status checks immediately. Copy the ID to continue with the CLI after leaving the page.

Rotate, revoke, and zeroize require two distinct custodians to approve one exact request before the requester retries the action. The console retains that request's Idempotency-Key in first-party browser storage, scoped to the signed-in tenant, requester, key, and action, through navigation and browser reloads. Return to the same browser profile after approval and click the action again. The console clears the saved request only after a terminal provider result. If site storage is cleared or the requester switches browser profiles, a fresh action key opens a new approval request and supersedes the old one; the custodians must approve the new request. API and CLI callers must likewise retain and reuse their original Idempotency-Key across the approval wait. A different key is a new intent.

The CLI mirrors those verbs under trstctl managed-keys, including list, get <provider> <key_id>, and approve. operation <operation_id> reads a pending command without issuing it again. Approval requires keys:approve; lifecycle mutation requires keys:write; the requester never counts as an approver; and every request is tenant-scoped, idempotent, and recorded as a key-material-free event before its PostgreSQL outbox command reaches the signer. A required gate launches the shipped control plane and cgo signer, exercises all six providers end to end against faithful cloud emulators, SoftHSM, or swtpm, and stops the signer mid-rotation to independently verify the resulting state (see Pitfalls & limits for what that proves and doesn't).

Rotate, revoke, and zeroize first open an exact, immutable approval request when dual control is required. A pending request returns HTTP 403 with code=managed_key_approval_pending, approval_request_id, and intent_digest in the problem body. The key is unchanged. The Key custody console shows this as a waiting state and links to Requests waiting for approval; an independent custodian approves there, then the requester retries the action. A denial, expired request, missing approval authority, or ordinary permission failure is a refusal and does not use the pending code. The requester cannot approve their own request.

The same posture includes the served CAP-KEY-03 FIPS path: GET /api/v1/editions and the Platform page expose the live FIPS POST booleans, make fips-build build target, fips-capable build (GOFIPS140) CI gate, and internal/crypto boundary, keeping the NIST CMVP product certificate as the external lab-certification residual.

Use it

Issue and govern through the served API and CLI:

# create a versioned profile (RA officer or admin)
trstctl-cli profiles create -f tls-server-90d.json

# list active profiles
trstctl-cli profiles list

A profile spec looks like this — note the explicit, enforced constraints:

{
  "name": "tls-server-90d",
  "spec": {
    "allowed_key_algorithms": ["ECDSA"],
    "min_ecdsa_bits": 256,
    "allowed_ekus": ["serverAuth"],
    "max_validity": "2160h"
  }
}

A hybrid transition profile allows the hybrid key label and binds it to the protocols allowed to request it:

{
  "name": "hybrid-web-30d",
  "spec": {
    "allowed_key_algorithms": ["Hybrid-ML-DSA-44-ECDSA-P256"],
    "allowed_protocols": ["acme", "est", "scep", "cmp"],
    "allowed_ekus": ["serverAuth"],
    "max_validity": "720h"
  }
}

Issuance happens through the enrollment protocols (ACME, EST/SCEP/CMP), the private-CA hierarchy API, and the external CA registry API, each of which calls the one issuance path with an Idempotency-Key. Revoke from the incident flow in Incident response.

Pitfalls & limits

  • Private-key custody is a deployment boundary. All six managed-key backends are census-served through the separate HSM signer artifact, but the operator must still provision an Enterprise BYOK license, the providers it runs (one, or several with per-tenant pins), credential files, IAM, network egress, and device/module trust — see configuration for the startup contract.
  • Emulator proof is not deployment certification. The cloud gate uses faithful vendor-protocol emulators, PKCS#11/YubiHSM use a SoftHSM-backed ABI target, and TPM uses swtpm — stronger than an author-injected registry or unit test, but not a live cloud account, a physical customer HSM, or the device's FIPS certificate. The native bindings ship in the cgo HSM signer artifact, not the default static control-plane artifact.
  • ARI scheduling covers trstctl-issued deployed X.509 identities. Certificates discovered from another CA can still be inventoried and risk-scored, but renewing them needs a configured issuer path that can replace that outside certificate.
  • External CA registration is operator configuration. Tenants can list and use configured upstream CAs, but provider credentials aren't created through the tenant REST API.
  • Revocation covers trstctl's own hierarchy. Third-party CA certificates are revoked through those CAs.

Reference

  • CLI groups: profiles, issuers, external-cas, certificates, and acme ari posture.
  • Served routes: POST|GET /api/v1/profiles, GET /api/v1/profiles/{name}/versions/{version}, the effect-free POST /api/v1/profiles/{name}/versions/{version}/restore/preview, append-only POST /api/v1/profiles/{name}/versions/{version}/restore, POST /api/v1/certificates, GET /api/v1/external-cas, POST /api/v1/external-cas/{id}/issue, GET /api/v1/acme/ari/posture (lifecycle:read), POST /api/v1/ca/authorities/{id}/rotate, POST /api/v1/ca/authorities/{id}/rekey, POST /api/v1/certificates/bulk-revoke, POST /api/v1/identities/bulk-revoke.
  • Upstream CA adapters: AD CS, AWS Private CA, Azure Key Vault, DigiCert, EJBCA, Entrust, GlobalSign, Google CAS, Let's Encrypt/ACME, Sectigo, shell CA, Smallstep, Vault PKI, and Venafi TPP/TLS Protect.
  • Key ceremony: StartCeremony → ≥threshold × Approve → CreateRoot / ImportExisting / CreateIntermediate. See the runbook.
  • Events: ca.issue, issuance.profile_evaluated, ca.root.created, ca.authority.imported, ca.intermediate.created, ca.authority.rotated, ca.authority.rekeyed, ca.cross_signed, ca.certificate.revoked, ca.crl.published.
  • RFCs: 5280 (X.509/CRL), 6960 (OCSP), 9773 (ARI).

See also

ACME & DNS · Enrollment protocols · Certificate-profile guide · CA key-ceremony runbook · Signing-service design · glossary: CA, CSR, OCSP, CRL, HSM/KMS

Covers: F4, F48, F53, F46, F47, F26

Rendered live from github.com/ctlplne/trstctl — found a mistake? edit this page.