Issuance & certificate authorities — how trstctl mints and governs certificates
What it is
Issuance is the act of creating a certificate: a machine asks for one, an authority signs it, and the machine gets back a signed ID it can present. This page covers issuing through any authority, running your own CA hierarchy, the rules that constrain what may be issued, telling clients when to renew, taking certificates back early, and where the private key physically lives.
The mental model: trstctl is a passport office. A CA prints and signs passports; a profile is the rulebook for what a valid passport may say; a registration authority checks your paperwork but can't print the passport itself; revocation is the bulletin of canceled passports; and the HSM is the locked vault holding the official seal.
Why it exists
Certificates expire on purpose and must be re-minted constantly, so issuance has to be automatic, governed, and auditable. Without a real issuance layer, three things go wrong: the wrong certificate gets minted (too-long validity, weak key, a name the requester shouldn't control); the signing key leaks and forges everything; or a compromised certificate keeps being trusted because nobody can pull it back. trstctl's issuance layer exists to make each of those hard.
How it works
One issuance path, any CA (F4)
Every certificate trstctl issues goes through a single, uniform interface — a CA
with one real method, Issue(request) — no matter who actually signs. The built-in
signer-backed CA, a CA in your own hierarchy, and
14 third-party authorities (Let's Encrypt/ACME, DigiCert, Sectigo, Microsoft AD CS,
AWS Private CA, Azure Key Vault, Google CAS, EJBCA, Smallstep, Venafi TPP/TLS Protect,
Vault PKI, GlobalSign, Entrust, and the shell CA escape hatch) all implement that same
interface. On every one of these paths a requester's key that fails a structural
weak-key check, or that the tenant has recorded as compromised, is refused before it
is signed or sent upstream, naming the check (see
Weak keys). The running binary exposes configured upstreams as a served registry at
GET /api/v1/external-cas; callers issue through one selected CA with
POST /api/v1/external-cas/{id}/issue using a PEM CSR, DNS names, and an
Idempotency-Key. The CA Hierarchy page drives the same route from the browser: pick
a configured external CA, submit CSR/DNS/profile/TTL, watch outbox-pending while the
issue intent is recorded, then see external-ca-issued evidence — never the
certificate PEM.
The same binary exposes read-only CA discovery at GET /api/v1/ca/discovery: one
response normalizing configured public/private upstream CAs and imported private
hierarchy authorities, with counts, source ids, status, and served path pointers —
never certificate PEM or private key material.
That single path wires the intended guarantees to receipts. Each issuance carries an
Idempotency-Key. For an upstream CA, the request first commits one
tenant-scoped external-ca.issue outbox intent; only the normal
bounded dispatcher calls the provider, waiting for its exact tenant/key result rather
than draining unrelated tenants. Before calling a provider without a native request
token, the worker durably claims that operation once; providers whose receiver
enforces the supplied token (AWS PCA, Azure Key Vault, Google CAS) use the reconciled
retry lane, while every unproven adapter stays at-most-once. A completed result
replays byte-for-byte from the certificate.recorded projection, so a changed command
gets 409 before provider I/O, and an interrupted tokenless submission is never
resent — it stays explicitly indeterminate rather than guessing the CA did nothing.
An ACME authority that pre-authorizes accounts out of band (CertiNext, HARICA and
other EAB-gated CAs) is configured with eab_key_id plus eab_hmac_key_ref; the
upstream client then presents the external account binding at registration and
the MAC key never leaves locked memory. acme_profile names the
draft-ietf-acme-profiles profile the client requests on every order; the request
is made only when the authority advertises that profile in its directory.
For ACME, a live attempt that stops before submitting the CSR for finalization can
prove that signing never began. The worker may then retry the same issuance request
after the directory, account, order, or domain-validation service recovers. Account
or order preparation may repeat, within the existing bounded outbox retry policy.
Once finalization starts, a lost response remains uncertain: trstctl tries to read
the original order's result, and refuses a new issuance if that cannot be verified.
A process crash supplies no proof of which step ran; an already-pending uncertain
claim stays blocked. Do not delete that claim to force another certificate.
After a definite result the worker emits certificate.recorded, rebuilds the
certificate inventory, and records the separate ca.issue evidence row. The
identity-transition issuance retry path (POST /api/v1/identities/{id}/transitions to
issued) is exercised by the shipped-stack Compose E2E gate: the gate repeats the
same request with the same key, waits through the real PostgreSQL/JetStream/outbox/
isolated-signer path, and asserts certificate inventory remains exactly one before
revoking that certificate and verifying OCSP and CRL state. This bounded proof does
not turn tokenless or unreconciled upstream adapters into exactly-once receivers. The request's
CSR is inspected through the single isolated cryptography path, and
the active profile is enforced
before signing, with an issuance.profile_evaluated event recorded either way.
If the CA returns a certificate but local recording fails, retries retain that result instead of asking the CA to sign again. Recording recovery checks each retained certificate event's complete receipt before doing projection work again. A receipt must match the original event; a sequence watermark alone is not proof. Missing recordings still recover in order, under the tenant and privacy barriers. Recovery still reads retained history, so its cost grows with that history; this does not establish a fixed recording-latency guarantee.
Upstream CA credentials are configured by the control-plane operator, not tenant
JSON: file references load into locked byte buffers for one outbox attempt and are
wiped afterward. Azure CA private-key operations and Let's Encrypt account JWS
signatures stay in the isolated signer, and the API exposes only the non-secret
registry row (id, type, name, status). A reused idempotency key after completion returns
the original certificate without re-signing, even after garbage collection. A crash
before the provider claim resumes through the outbox. After a tokenless provider
claim commits, recovery needs the original result or a live protocol proof that
signing was not submitted; an uncertain crash stays blocked. The production
external_cas JSON shape, file:/absolute/path credentials,
private-endpoint allowlist, custom trust-root, and mTLS fields are documented in
Configuration.
Kubernetes CRD-native issuance
trstctl ships Issuer, ClusterIssuer, and Certificate CRDs in the trstctl.com
API group. The Kubernetes agent reconciles them, marks issuers Ready, signs
cert-manager CertificateRequests only when they target an existing trstctl issuer,
signs approved native CertificateSigningRequests from certificates.k8s.io/v1, and
can fulfill a trstctl-native Certificate directly into a Kubernetes TLS Secret. With
the opt-in --k8s-csr-approver the agent also approves or denies pending native CSRs
for trstctl signer names by the issuing CA's answer
(POST /api/v1/ca/authorities/{id}/issue/evaluate, the checks issuance makes without
signing; trstctl-cli ca authorities evaluate-leaf), writing Denied with the CA's
reason. The
read-only GET /api/v1/kubernetes/certificate-signing-requests / CLI
trstctl-cli kubernetes csr report the served CAP-K8S-04 surface, supported signer
names, required RBAC, and residuals.
A cert-manager Certificate references trstctl with
issuerRef: {name: trstctl, kind: ClusterIssuer, group: trstctl.com}; a workload can
also use trstctl's native API directly:
apiVersion: trstctl.com/v1alpha1
kind: Certificate
metadata:
name: web
spec:
secretName: web-tls
dnsNames: [web.apps.svc.cluster.local]
issuerRef: {name: trstctl, kind: ClusterIssuer, group: trstctl.com}
The agent forwards only a CSR to the configured trstctl issue endpoint, adds a stable
Idempotency-Key, and authenticates with a token mounted from a Kubernetes Secret:
cert-manager gets the normal kubernetes.io/tls Secret; a trstctl-native Certificate
gets a locally generated workload key written to Secret/<secretName> (transient
buffers wiped) and marked Ready; a native CertificateSigningRequest needs Kubernetes
or a separate approver to set Approved — the agent never approves its own requests.
cert-manager CertificateRequests for this external issuer also need a separate
approver on every renewal. Without one, a short-lived Secret can expire while
the next request waits. The cert-manager approval runbook
shows a scoped approver policy, RBAC binding, denial checks, and served-TLS readback.
For native Kubernetes CSRs, the agent accepts spec.signerName values such as
trstctl.com/trstctl or trstctl.com/<issuer-name>, optionally disambiguated with
the trstctl.com/issuer-{name,kind,group} annotations, and writes the PEM chain to
status.certificate while preserving the
Approved condition — completion is status.certificate being present, not a custom
Ready. CI proves the cert-manager path against a real kind cluster; served
controller acceptance proves both the trstctl-native path and CAP-K8S-04 native CSR
support. The shipped ClusterRole grants sign only for trstctl.com/trstctl; a named
signer such as trstctl.com/payments needs that resource name added rather than
granting every Kubernetes signer.
For cert-manager, the enrolled agent also reports each trstctl-issued
CertificateRequest's exact UID, controller-owned parent Certificate UID,
and issued leaf fingerprint over the authenticated agent channel. The
event-projected GET /api/v1/kubernetes/cert-manager-certificate-requests and
trstctl-cli kubernetes cert-manager-requests expose the latest controller
observation. Inventory list/detail responses retain kubernetes_provenance
for an exact issued fingerprint even after cert-manager garbage-collects that
request. The Certificates console links that observation to Workloads &
Machines without calling an externally controlled renewal manual. This is
issuance provenance; verify the Kubernetes Secret and the served listener
independently before claiming deployment.
The same agent serves CAP-K8S-07 public trust-bundle distribution. Configure the
controller's exact --k8s-trust-namespaces allowlist before applying a
cluster-scoped TrustBundle.trstctl.com. The empty default allows no
namespaced targets. The controller writes canonical public certificates to
new or matching agent-owned ConfigMaps and records the target count, bundle
hash, and Ready condition. GET /api/v1/kubernetes/trust-bundles,
trstctl-cli kubernetes trust-bundles, and the Workloads console show the
controller's reported posture and required grants.
The control plane can schedule a rollout to reporting clusters, canary first.
Each participating controller must enable --k8s-trust-rollouts=true locally.
Stored remote TrustBundles require a valid rollout identity and hash, and can
distribute to ConfigMaps. The separate CSI trust plugin needs its own opt-in to
serve remote TrustBundles. Read the intended revision and its live target bytes
when verifying delivery. The canary gate uses the reported hash and bundle
name. A newer rollout of the same name replaces it cluster by cluster. See the
Kubernetes trust guide.
Running your own CA hierarchy (F48)
trstctl can be your private PKI: a root CA, intermediates beneath it, end-entity certificates beneath those — the usual tree where the root is kept offline-precious and the intermediates do the day-to-day signing.
The dangerous operations are gated by an m-of-n key ceremony: nothing happens
until m of n named custodians approve. Root and intermediate creation are served
today: open a ceremony, collect distinct custodian approvals, then create or import
the CA. Each operation consumes one pending ceremony whose purpose matches the
reviewed resource: root:<sha256-of-ca-spec>,
intermediate:<parent-ca-id>:<sha256-of-ca-spec>,
offline-root:<sha256-of-root-cert-der>:root:<sha256-of-ca-spec>, or
offline-intermediate:<parent-ca-id>:<sha256-of-ca-spec>. Existing CA import uses
import-existing-ca:<signer-handle>:<sha256-of-chain-der>:root:<sha256-of-ca-spec>,
binding the reviewed chain to the exact signer-held key handle.
For a key that trstctl must create, use prepare_existing_ca with the public
external issuer chain and child CA profile. After two distinct custodians
approve, POST /api/v1/ca/authorities/imported/csr reserves a tenant-bound
signer key before returning its public CSR. The external issuer signs that CSR;
GET /api/v1/ca/authorities/imported/csr/{ceremony_id} recovers the same CSR
after a lost response or restart. Import the signed child certificate followed
by the exact reviewed issuer chain with the same ceremony, signer handle and
profile. The server checks the public key, issuer chain and optional registered
parent before consuming the ceremony. An already-held signer key uses the
import_existing_ca ceremony above.
A post-quantum pilot hierarchy runs beside the classical one through the same
ceremonies: set signature_algorithm to ml-dsa-44, ml-dsa-65 or ml-dsa-87
on a root or intermediate spec and the signer mints that key (the ceremony
purpose binds the key family, so approvals reviewed for ecdsa-p256 cannot
mint it). The authority issues classical and pure ML-DSA leaves under RFC 9881,
its /crl/{tenant}/authorities/{id} and /ocsp/{tenant}/authorities/{id}
responses carry its ML-DSA signature, each authority reports its
key_algorithm, and OpenSSL 3.5 or newer verifies the chain, CRL and OCSP
responses. Offline-root-signed intermediate CSRs and external-issuer import
CSRs stay classical; a re-key keeps the predecessor's key family.
CA renewal/re-key uses ca-rekey-v2:<sha256-of-authority/spec/reason> to create fresh signer-held
CA material for the selected authority with the reviewed lifetime and profile.
Online intermediate creation, external intermediate CSR signing, cross-signing,
and intermediate re-key require an active signer-backed issuing CA. If the
parent is an offline public-only CA, ceremony preview and start refuse the
online operation before asking custodians to approve it. To replace an
intermediate under that parent, use prepare_existing_ca to obtain a public
CSR for a new signer-held CA key, have the offline issuer sign the CSR outside
trstctl, import the signed successor with its reviewed chain, then preview and
activate the predecessor-to-successor rotation.
Short approvals return ErrQuorumNotMet; an opener approving their own
ceremony, or a ceremony already used or opened for a different resource/spec, fails
closed before the CA mutation commits — stopping one compromised admin account from
minting a rogue root or intermediate, and stopping one valid ceremony from being
replayed against a different CA request.
The console and CLI do not ask an operator to approve a blind change. The
effect-free POST /api/v1/ca/ceremonies/preview route validates the exact ceremony
body and returns a non-secret receipt before any approval record or signer action
exists. For zero-downtime rollover,
POST /api/v1/ca/authorities/{id}/rotate/preview applies the activation eligibility
rules to the exact predecessor and successor, then explains the routing changes,
trust risks, and proof steps without changing either CA. Both previews return an
exact request fingerprint and explicit empty write/external-effect lists; the later
mutation rechecks authorization and state under lock instead of trusting the preview.
The served hierarchy API lives at /api/v1/ca/ceremonies/preview,
/api/v1/ca/ceremonies, /api/v1/ca/authorities,
/api/v1/ca/authorities/offline-roots, /api/v1/ca/authorities/imported,
/api/v1/ca/authorities/imported/csr,
/api/v1/ca/authorities/{id}/offline-intermediates/csr,
/api/v1/ca/authorities/{id}/offline-intermediates, and
/api/v1/ca/authorities/{id}/issue, with zero-downtime successor activation at
/api/v1/ca/authorities/{id}/rotate/preview and
/api/v1/ca/authorities/{id}/rotate, signer-backed renewal/re-key at
/api/v1/ca/authorities/{id}/rekey, cross-signing at
/api/v1/ca/authorities/{id}/cross-sign, offline-root successor/cross-certificate
import at /api/v1/ca/authorities/{id}/offline-rekey, and offline-root-produced
cross-certificate verification at /api/v1/ca/authorities/{id}/offline-cross-signs.
Online root/intermediate private keys live only in the isolated signing service,
referenced by signer handles; the control plane stores certificates, chains,
metadata, and ceremony state, never the CA private key. Existing-CA import verifies a
public chain's first certificate against the supplied signer handle and the
chain/profile before serving normal leaf issuance. A durable, global first-writer
fence binds that handle to one tenant and reviewed ceremony before the authority
event; a conflicting tenant cannot reuse the same signer-held key, including after
the first tenant offboards. Offline-root import accepts
exactly one public certificate PEM (never a private key) and an optional
possession_signature: the root's signature, made on the offline system, over
the import ceremony's challenge. With it the authority records possession as
offline_signature; without it, as not_proven (see
the key ceremony runbook).
It then generates a signer-held
intermediate CSR for the operator to sign outside trstctl, and imports the result
only if it chains to the offline root, matches the reviewed CASpec, and carries the
signer-held public key. Rotation and re-key activations both promote a signer-backed
successor (re-key from a fresh ca-rekey-v2 ceremony), mark the predecessor
superseded, record replaces_id, and keep both issue URLs live while new
certificates chain to the successor; offline-root re-key works the same way but stays
an operator ceremony since the offline key never enters trstctl.
External intermediate CSR signing is also a protected workflow:
the console reviews the public CSR and spec, opens an exact two-custodian
ceremony, refreshes its quorum and signs only after approval. API and CLI
callers supply that ceremony ID with the same CSR and spec; the external
private key stays with its requester. The operator verifies the returned chain
and CSR public key with a stock X.509 client. Every served step
(ca.ceremony.started, ca.ceremony.approved, ca.ceremony.cancelled, ca.root.created,
ca.authority.imported, ca.intermediate_csr.issued, ca.intermediate.created,
ca.authority.rotated, ca.authority.rekeyed, ca.cross_signed,
ca.endentity.issued) is a tenant-scoped event recorded immutably in the
tamper-evident log. Cross-signing uses a purpose-bound
cross-sign:<ca-id>:<sha256-of-target-cert-der> ceremony: the signer-backed route
signs inside the signer, while offline-root routes verify validity, key usage, EKU,
DNS, path length, subject/public key, and both chain directions first. The full
public cross-certificate is recoverable by ceremony ID from the event-projected
GET /api/v1/ca/cross-signs/{id} receipt after a restart. The CA hierarchy
console exposes effect-free review, quorum readback, execution, and that receipt
for both signer-backed cross-signing and public-only offline-root cross-sign
import. The offline import form accepts only public certificate PEM blocks;
two distinct custodians approve the exact issuer, target and cross-certificate
package. Preview and ceremony start verify the offline root's status, the
public signature and the certificate constraints before requesting approvals;
execution repeats the checks before recording the second trust path.
Offline-root succession has a separate, indexed, tenant-scoped receipt at
GET /api/v1/ca/offline-rekeys/{ceremony-id} containing the successor lineage
and both public cross-certificates. It is projected from the same immutable
re-key event as the authority state and can be recovered by ceremony ID after a
lost response or restart. The Imports workspace in the console previews the
exact public package, starts the bound two-custodian ceremony, reads the quorum,
activates the successor, and recovers that receipt. Offline private keys never
enter the browser or control plane. On a fresh browser session, the console
labels a recovered completed receipt as a server result and says that the
original public inputs were not compared in that browser. The full
operator procedure is the
CA key-ceremony runbook.
On upgrade, read-model catch-up checks older version-2 managed-hierarchy
ca.endentity.issued events that were previously recorded only in the CA
responder ledger. The hierarchy payload includes a subject field, even when
empty; same-version responder-only payloads omit it and keep their original
meaning. When the hierarchy event contains public DER and its exact fingerprint,
trstctl verifies the leaf against the named issuing CA and recovers
one stable certificate inventory row. The original event remains the mint; this
does not sign a replacement or claim where the leaf's private key was held.
A same-version responder-only payload keeps its serial meaning in the
projection; if no other event inventoried its leaf, the first lifecycle sweep
verifies that leaf against the managed CA's stored certificate and records a
ca.certificate.inventory_recovered event, which gives it one inventory row
that revokes against the same CA.
Version-1 serial-only events cannot supply a public certificate for this replay.
For those, use a supported certificate import to record an independently obtained
public PEM; an import is an observation and does not by itself prove trstctl
issued or can revoke that leaf. A failed replay leaves startup closed with an
error rather than guessing a certificate or CA.
A CA key retirement checklist also accepts a narrowly reviewed disposition for a revoked, unserved host-held leaf. The release binds the immutable mint, completed local-issuer revocation event, public digest, and two distinct custodian approvals. The signer independently folds both events before accounting for the dependent. Event-only recovery rebuilds the completed revocation authority receipt from that retained event if the independent first-writer table is absent. This is a risk decision about possible external use, not proof that every relying party enforced revocation. See the key-ceremony runbook for the operator path.
For a still-served host certificate, CA succession uses a recorded re-protection
plan and a retained-key CSR from the assigned host agent. This continues the
same deployed subject and key under the direct successor CA. The identity's
selected active certificate profile, or the configured default when the
identity has none, constrains names, key strength, EKU and lifetime. Preview,
registration, queueing, worker dispatch and signing recheck that policy. A
later policy edit that excludes the certificate refuses the cutover;
its allowed_protocols list governs new enrollment, so it does not turn this
administrative continuation into API enrollment. The audit marks the profile
decision ca-reprotect. The host's signed served-state report, independent
TLS readback and updated retirement checklist are needed before treating the
old CA dependent as cleared.
The cutover event updates the identity's selected authority ID and displayed
authority name together, using the tenant's successor CA record. A warm restart
repairs names projected by older builds from the retained cutover events without
changing a later CA selection or recreating an offboarded identity.
Recovering a failed CA-key retirement delivery
The CA Hierarchy lifecycle tab shows the exact frozen retirement command and its
signer-delivery state. If delivery is failed after its attempt budget, review
the failure and enter an audited reason to grant one more attempt. The retry
uses the same tenant-bound command and outbox row, so it cannot silently
alter the dependency evidence or sign a new destruction request. If the signer
already handled the command before its response was lost, its durable journal
returns the same signed refusal or destruction record. Read the checklist again
to see the result; pending means the signer has not yet returned an accepted
result. The CLI command is trstctl-cli ca keys retirement retry <id> -f retirement-retry.json; its body uses command_event_id, expected_attempts,
reason, and confirm_irreversible, with an Idempotency-Key header.
Executing a CA migration in waves (H2)
CA rollover is served as a durable trust-before-leaf workflow, not as a batch loop.
A run lists its cohorts explicitly. To build them from the estate instead of by hand,
ask the read-only cohort planner (P3-L1): GET /api/v1/migrations/cohort-plan selects
the deployed X.509 identities bound to an enabled deployment target by profile, by
issuer_ca_id (the trstctl CA that issued each one's current certificate, the CA a
planned switch revokes it through), by owner_id or by connector, and cuts them
canary first: a canary of canary_size (1 to 10, default 1), then cohorts of
cohort_size (1 to 500, default 25), in name order, each member with its target's
host agent. Identities the selector matched but a run could not take (not deployed,
no enabled target, a current certificate whose issuer cannot be read) are listed as
skipped with the reason. A plan places at most 1,000 members and says when it was
truncated; it records nothing. On the Migration page, Plan cohorts from a selector
on the waves step does the same by profile or issuer, and Use these cohorts fills
the waves for review.
trstctl migrations plan-cohorts --profile web-tls --canary_size 1 --cohort_size 25
Then run the read-only assessment:
trstctl migrations assess -f assessment.json
Then start a reviewed manifest with a fresh Idempotency-Key (the CLI supplies one
unless you override it):
{
"plan_id": "root-rollover-2026",
"new_authority_id": "11111111-1111-4111-8111-111111111111",
"waves": [
{
"id": "canary",
"ordinal": 1,
"members": [
{
"identity_id": "22222222-2222-4222-8222-222222222222",
"agent_id": "33333333-3333-4333-8333-333333333333",
"trust_anchor_path": "/etc/trstctl/roots/next.pem"
}
]
}
]
}
new_authority_id must name an active signer-backed hierarchy authority in the
same tenant. The control plane reads its public certificate and freezes both the
authority ID and anchor fingerprint into every member binding. Private keys never
enter the manifest or control-plane process. When the trust receipt arrives, the
successor is signed by that exact authority; a later CA rotation does not silently
reroute an already-reviewed run.
Each agent_id must be the active host agent saved as required_agent_id on
that identity's enabled deployment target. Another enrolled host in the same
tenant is not interchangeable: Start rejects the mismatch before queuing trust
or successor work. Keep the target assignment and manifest aligned when moving
a workload to a new host, then submit a fresh reviewed run.
A cloud certificate store joins a run with no agent (A166). Where the tenant has
no network relay enrolled, the control plane's own client carries the cloud
stores, so a member whose target is one omits both agent_id and
trust_anchor_path:
{ "identity_id": "44444444-4444-4444-8444-444444444444" }
The binding is frozen when the run starts: a network relay enrolled while the run is under way does not take the member (A166-L5). The control plane still installs the successor, reads the store back for the live gate and executes and reads back a rollback; only a new run must name the relay.
The control plane issues the successor under the new authority with the
predecessor's exact identifiers, creates its fingerprint-named certificate and
points the target's certificate map entry at it, then reads the entry back: the
live gate passes only when the certificate the entry serves is the successor.
Rollback re-points the entry at the predecessor and reads it back the same way.
The store serving a certificate is not yet the load balancer serving it: the
provider propagates a change on its own schedule. When the target's config names
the endpoint its load balancer answers on (verify_address, with
verify_server_name for SNI, and verify_private_egress_cidrs when that endpoint
is on a private range), the gate also needs a TLS handshake there to present the
certificate. Until it does, the member waits, checking every 30 seconds without
spending its retries, for up to 30 minutes from when its row was queued; an endpoint
still presenting another certificate after that fails the gate as a wrong answer
would. Without verify_address the store's readback alone closes the gate, and the
member's evidence in the console says so.
A store that cannot be reached is retried. If it stays unreachable until the
control plane stops retrying, the member fails its gate as though the store had
answered with the wrong certificate: the run rolls the member back on its own,
and if the rollback cannot reach the store either, the run halts and its halt
reason names the member. Once the store answers again, roll the run back once
(Review rollback in the console, or POST /api/v1/migrations/runs/{id}/rollback)
to re-bind the predecessor.
A GCP Certificate Manager target qualifies when it names certificate_map_entry
and trstctl delivered the current certificate to it. AWS Certificate Manager
re-imports in place and Azure Key Vault serves its newest version, so neither can
restore a predecessor; Start refuses both with that reason. Once a relay is
enrolled, cloud-store work runs on the relay: assign the target to it and name
the relay as the member's agent_id. In the console, choose Control plane
(cloud certificate store, no agent) as the member's executor.
The read-only assessment counts trust stores reported by that exact assigned agent. A deployment target's display name does not identify a machine. If a trust-store finding predates the receiver-stamped agent ID or its authenticated completed run, run inventory again from the enrolled host; the historical finding remains visible in discovery but cannot make a migration appear ready.
The phase order is fixed: install and read back the public anchor, issue against a
host-generated CSR, deploy, handshake the configured listener, then release the next
wave. Agent receipts are signature- and lease-verified before they can move a gate.
Any failed receipt halts publication of later work and automatically starts the
newest-first inverse. Every effect published for the failed gate is settled, including
work that was already leased and reports after rollback began. pause prevents new
publication while retaining receipts for already-leased work; resume reconstructs
only the unfinished actions. The manual rollback control uses the same inverse:
restore and verify predecessor leaves newest wave first, restore inventory truth from
the same migration event, and remove successor trust only after the leaf inverse
passes. A failed inverse halts with its durable attempt cursor; a later manual retry
uses fresh outbox identities rather than replaying a terminal job.
Starting and resuming require both keys:write and certs:issue, because either
operation can release a successor mint. Pause and rollback require keys:write.
Closed-loop proof (X3). A run started with "require_observed_traffic": true
holds each wave at its live gate until two proofs agree for every member: the signed
live receipt that the endpoint serves the successor (the probe, which also reads the
chain a wire sensor cannot see on TLS 1.3), and an observer agent's report of new
connections at the member's verify address (or its server name on that port) after
the successor went live — presenting the successor on TLS 1.2. Behind NAT or a load
balancer the observer sits beside the backends, so the member's deployment target can
name them (X3-L2): "observed_addresses": ["10.0.3.17:8443"] in the target's
configuration (host:port, at most 16, refused otherwise when the target is saved) is
frozen into the run, and connections seen there, or under the member's server name on
one of their ports, count as the member's. The leader checks every
30 seconds, reading every one of the tenant's observer TLS findings a page at a time
and keeping only those at the endpoints it is waiting on (X3-L4), so no number of
findings elsewhere hides the one that decides. A member still unobserved observed_traffic_window_seconds (default 3600,
at most 604800) after going live pauses the run with the reason; the run never
advances on the probe alone, and a resume releases the gate only once the wire evidence
is there. Missing wire evidence is never recorded as a failed member. When a member has
both proofs, the observer's findings for its predecessor certificate at that endpoint
are marked remediated, naming the live receipt and the observed connections.
Wire evidence for every replacement (X3-L1). A plain renewal's deployment, or any
other replacement, waits for the same wire evidence without a migration run. When a
verification of an endpoint (GET /api/v1/endpoints/verifications, Connectors →
Health, retries, and rollback) finds it serving another certificate than it last
observed there, the endpoint records which certificate was replaced and when, and its
traffic reads awaiting. On the same 30-second leader pass, an observer agent's new
connections at the endpoint's address, or at one of its target's observed addresses
(X3-L2), after the replacement — presenting the
replacement on TLS 1.2; on TLS 1.3 any new connection, since the verification already
proved the certificate — move it to observed, naming the observer finding
(traffic_finding_id), and mark the observer's findings for the replaced certificate at
that address remediated, naming the endpoint, the replacement and the observed
connections. Nothing closes on the probe alone. An endpoint still unobserved an hour
after its replacement reads unobserved from then (traffic_unobserved_since); later
evidence still closes it. The Connectors page shows each endpoint's wire evidence.
Planned CA switch (P3). A run started with "revoke_predecessors": true (on the
Migration page, "Revoke each predecessor once its successor is verified live") moves its
members to the new issuer and retires what they served before. Each cohort runs the
same gates — the new root distributed and read back first, then the successor issued
and deployed, then the signed live receipt that the endpoint serves the new chain
(and the observed-traffic proof when the run requires it) — and only then is each
member's previous certificate revoked, reason superseded, through the trstctl CA
that issued it, which publishes its CRL before the cohort counts as done. A switch away
from a vendor CA (P3-L2) revokes a previous certificate the external CA registry issued
at that vendor, through its integration (by serial, or from the certificate for an ACME
authority): the revocation is recorded, and the member counts as done, only once the
vendor accepted it. A vendor that holds the request for an approver leaves the member
waiting and the request is sent again until it is accepted; a vendor whose integration
cannot revoke pauses the run with the reason. The assessment names the vendor that
would revoke each predecessor (external_ca_id, ca_name), and the cohort planner's
issuer_ca_id selects by an external CA too. A switch can also move to a vendor CA
(P3-L2): start it with new_external_ca_id (an available CA from the external CA
registry) and new_trust_anchor_pem (that vendor's root) instead of new_authority_id,
or choose it under Vendor CAs on the Migration page with its root pasted. Every host
member trusts and reads back the vendor root first; then the host generates its key and
the vendor signs its certificate request, the endpoint is verified serving the vendor's
chain, and the predecessor is revoked as for any switch. Only host members can move to
a vendor, since the vendor signs a request the host makes. A member whose current
certificate neither a trstctl CA nor a vendor integration can revoke is refused at start
rather than left valid. The first cohort is the canary: a failed trust readback or live check in
any cohort pauses the run, naming the member, with nothing revoked and no later cohort
started. Resuming retries exactly the failed members as the same jobs (the host agent
reuses the key and certificate it kept for that job, so no second successor is
signed); rolling back restores the current cohort, never a cohort whose predecessors
are already revoked. Starting the same switch again with the same Idempotency-Key
returns the existing run and changes nothing.
Before starting, assess the switch as one (P3-L4): POST /api/v1/migrations/assess with
"revoke_predecessors": true lists, member by member, the certificate the switch would
revoke (its id, serial and fingerprint) and the CA that would revoke it, read by the
same rules as the start. A member whose current certificate no trstctl CA can revoke is
listed with the reason and is an unknown (predecessor_not_revocable) that blocks the
start. The Migration page assesses a switch this way and shows Certificates this
switch revokes in the review; an assessment that was not a switch's does not open
Start migration for one. Assessing changes nothing.
trstctl migrations start -f run.json
trstctl migrations list
trstctl migrations show RUN_ID
trstctl migrations pause RUN_ID
trstctl migrations resume RUN_ID
trstctl migrations rollback RUN_ID
The REST equivalents are POST /api/v1/migrations/runs, GET /api/v1/migrations/runs, GET /api/v1/migrations/runs/{id}, and the three action
routes below that run. The executable boundary is X.509 identities deployed by an
enabled host-agent connector with a configured verification listener, by a network
appliance through its relay, or into a cloud certificate store through the control
plane; the successor asserts exactly the predecessor's names and addresses. The new authority may be a
signer-backed CA or an imported offline root; for an offline root the successors
are issued by its one active signer-backed intermediate and the root certificate is
the anchor distributed to hosts. Every unsupported membership is rejected, with
its reason, before any trust job is published.
The run pins each identity's active certificate-profile revision and effective
validity into the immutable member and successor job. A member whose profile requires a
distinct issuance approval makes the start itself the approved operation: the
first start answers 202 with an approval request, reviewers approve it by intent
digest, and the same start posted again records the run with that single-use
approval consumed. Incident fleet reissuance still refuses an approval-required
profile before Start. A profile label alone never authorizes signing.
Profiles and the registration-authority split (F53)
A certificate profile is a versioned, tenant-scoped rulebook: allowed key
algorithms and minimum sizes, extended key usages, maximum validity, DNS suffixes,
and protocols. Editing a profile creates a new version; old versions stay
queryable, so you always know which rules a past certificate was issued under. On
every issuance, enforceProfile fetches the active version, validates the request,
and emits an audit event for the allow-or-deny decision.
A leaf's extended key usages come from its profile, never from the CSR alone: a CSR
may ask only for usages the profile allows, a profile that lists none allows
serverAuth and clientAuth, and no leaf ever carries ocspSigning or
anyExtendedKeyUsage. An ocspSigning leaf from an issuing CA would be a delegated
OCSP responder for every certificate that CA issued; trstctl's own responder
certificate is made separately. A CA hierarchy's any constraint is not copied onto
its leaves either. The platform issuing CA also never signs a timeStamping leaf: its
own TSA certificate, minted separately, anchors the audit trail, so a timestamping
leaf comes from a tenant-managed CA.
For an identity lifecycle issuance, preview and execution check the pinned profile's
known protocol, validity, and subject-name rules before recording identity.issued.
The signing worker still checks the actual CSR, including key strength and requested
usages, before signing. A ready lifecycle preview therefore cannot turn an
ACME-only profile into a queued API issuance.
Each AD CS issuer's health is on Posture → AD CS issuers (and GET /api/v1/adcs/issuers/health): healthy, degraded, failing or idle over the last
seven days, from the certificates it issued and the refusals trstctl diagnosed,
with the latest refusal's cause (for example the template's permissions not
allowing the requester to enroll) and what to do about it.
A profile can also choose who signs. With upstream_external_ca set to a
configured external CA's registry ID, ACME, EST, SCEP and CMP enrollments under
that profile pass the profile's checks in trstctl and are then signed by that
CA instead of trstctl's own: for example ACME in front of an AD CS template, with
one adcs external CA entry per template. The client receives the upstream's
leaf and chain, and the certificate is recorded in inventory like any other.
Saving a profile that names an external CA the tenant does not have is refused
(422). An ACME revoke of an upstream-issued certificate is refused, because
trstctl's CRL covers only its own CA; revoke it through trstctl's revocation
(the Revocation center or POST /api/v1/certificates/bulk-revoke), which goes
to the upstream CA.
Revoking at AD CS. Web Enrollment has no revoke call, so an adcs issuer
revokes through a domain-joined network relay named in its revocation_relay.
A revocation of a certificate that issuer issued is handed to that relay, which
runs certutil -revoke with the reason, publishes a new CRL with certutil -CRL
and reads the CA database row back; trstctl marks the certificate revoked only
when the relay reports the row revoked and the CRL published. A relay that has
not answered yet leaves the revocation waiting (it does not count against its
retries); a failed attempt is retried on the revocation's own backoff. AD CS
takes the reasons unspecified, keyCompromise, cACompromise, affiliationChanged,
superseded and cessationOfOperation. With dual control on (require_approval), a
revocation waits for its approvals before the relay is asked, like every other
revocation.
Approving and denying held AD CS requests (N1). A template that requires CA
manager approval holds a request as pending. Through the same relay, trstctl
approves or denies it at the CA — behind its own approval, never on one person's
word. POST /api/v1/external-cas/{id}/adcs-requests/{request_id}/approval-requests
asks for an exact decision (approve or deny, with a reason); reviewers who may
issue approve it under /api/v1/approval-requests (the requester cannot approve their
own); then the requester executes exactly what was approved with POST .../decisions, which spends the approval on one relay job. The relay reads the
request's row first and acts only while the CA still holds it as pending — so an
approval can never re-issue a request a manager already denied — then runs
certutil -resubmit (approve) or certutil -deny and reads the row back.
GET /api/v1/external-cas/{id}/adcs-requests lists the requests the issuer's CA holds,
newest first, from the relay's latest read of its database (adcs.ca_database, N1-L2):
each one's request id, template, requested name and requester, and whether a trstctl
issuance is held on it. truncated says the CA held more than the newest 500, and
read_at says when it was read; a later read replaces the list. The CA Hierarchy page
shows the same list under each AD CS authority. CLI: external-cas adcs-requests.
GET /api/v1/external-cas/{id}/adcs-decisions lists each decision as queued,
decided with the CA's disposition (and, for an approval, the issued serial), or
failed with the relay's reason. Status: served.
Resuming an issuance the CA held (N1-L1). An issuance through an AD CS template
that requires CA manager approval waits for the request for 60 polls (two minutes at
the default poll_interval), then gives up with the request still held. trstctl
records the CA's request id with the issuance. GET /api/v1/external-cas/{id}/held-issues lists each one as held, issued or denied,
newest first. Once a CA manager has approved the request, POST /api/v1/external-cas/{id}/held-issues/{request_id}/resume reads it back from the CA by
that id. trstctl then records the certificate under the held issuance's own idempotency
key. The issuance's caller gets the certificate by replaying its request, and the
issuance's delivery finishes without contacting the CA again. While the CA still holds
the request, resuming answers 409 and records nothing; a denied request resumes as
denied. Reading a request back cannot issue a second certificate, so resuming again is
safe and gives the same answer. CLI: external-cas held-issues and external-cas held-issues resume. When the issuer's relay approves a request a trstctl issuance is
held on (N1), recording that decision queues the resume in the same transaction, so the
issuance finishes on its own. A manager's approval made at the CA itself is resumed by an
operator. The CA Hierarchy page lists each AD CS authority's held issuances (request id,
names, when they stopped waiting, and whether they were issued or denied) with Resume.
Status: served.
The registration-authority (RA) model is a role split that prevents the classic
PKI abuse of one person approving and fulfilling their own request. The built-in
ra-officer role can read/write profiles and request certificates but doesn't hold
certs:issue — only an operator/admin can issue. The split is enforced by
RBAC, not convention, and a test asserts it. Authoring
profiles is covered in the
certificate-profile guide.
The self-service requester path is served end to end for X.509 requests. /request
lists active profiles and tenant-visible owners, makes the requester choose the
accountable owner by name, and sends the exact proposed body to the effect-free
POST /api/v1/issuance-requests/preview route. The same server admission rule used
by submission checks the owner, validates an optional public CSR, and resolves an
active profile name to its exact immutable version. The answer names blockers,
key-custody posture, independent approval authority, and the writes that a later
submission would perform. Preview itself appends no event, writes no projection,
creates no idempotency record, and contacts no CA. The console fails closed: it
enables submission only while a green preview still matches every reviewed field.
Operators and automation can ask the same question with
trstctl issuance-requests preview --body request.json.
Once the exact approved issuance has a recorded certificate, the signing worker
marks the request fulfilled without requiring the requester to keep a browser
open. The completion API can also reconcile a historical result after deployment,
revocation, or retirement. This changes only the request receipt: the credential's
current state stays intact. issued_by comes from the original issuance event
(and remains empty when that historical event has no actor); issued_at is the
certificate's retained recording time. Missing or ambiguous signing evidence,
a different owner or profile, and an unrelated issuance key still refuse completion.
Submission sends the normalized body to POST /api/v1/issuance-requests. The
server re-runs that shared admission rule before appending
issuance.request.opened; missing or malformed values return 400, while absent and
other-tenant owner UUIDs return the same 422 answer. The console links directly to
Profiles, Owners, and CA hierarchy when a prerequisite needs configuration. This is
important: oidc|dev-1 names an authenticated caller, while an owner UUID names a
tenant database row. They are different identifiers and are never substituted for
one another.
The requester's /request history and the /approvals inbox read the same
first-class issuance-request projection. A submission therefore remains
requested; it does not create an identity or mint a certificate. A distinct
principal can approve or deny the exact request, and approval is still not issuance:
the request becomes issued only after the authorized issuance outcome is linked.
If issuance is interrupted after approval, the approved row remains visible and the
console offers Retry safely. Preparation reuses the request's deterministic
identity and stable issuance key, so retry continues the same operation instead of
minting a second certificate.
The built-in ra-officer can read owners, author/read profiles, and request
certificates, but still cannot write identities or hold certs:issue.
Profile recovery is served as an append-only operation. POST /api/v1/profiles/{name}/versions/{version}/restore/preview returns an effect-free
receipt for copying a known-good historical spec into the next version. The receipt
pins the current active version, semantic spec digest, reason, request fingerprint,
issuance risks, and proof steps; it emits no event and performs no external call.
POST /api/v1/profiles/{name}/versions/{version}/restore rechecks that active-version
fence under the projection lock, creates one new event-sourced active version, and
leaves every historical row untouched. Reusing the idempotency key returns the same
result. A newer concurrent version fails closed with 409, and profiles governed by
requires_approval retain non-requester dual control. The console exposes the same
flow after a version diff; the CLI commands are profiles restore-preview and
profiles restore. See the profile-authoring guide
for a complete example.
A profile create or edit that falls under requires_approval answers 202
with an approval_id and state: awaiting_approval instead of a new version.
A distinct reviewer holding profiles:write finds the parked request with
GET /api/v1/profiles/approvals (or GET /api/v1/profiles/approvals/{id}) and
reviews the exact proposed spec, its spec_sha256 digest, requester, expiry,
and any restore source/reason in the console's Requests waiting for approval
page or in the API/CLI. The console counts these requests alongside other
pending approvals. A reviewer with only profiles:read and profiles:write
can use this inbox without access to certificate, secret, or managed-key
decisions; the console disables the decision for the requester. Approval takes
effect immediately: the reviewer confirms it with
POST /api/v1/profiles/approvals/{id}/approvals; quorum (one
non-requester approval) applies the queued spec as the new active version and the
record reads state: issued with the created profile_id. The requester's own
approval is refused with 403 (dual control). Identical retries of the
approval with the same Idempotency-Key replay the one decision. The parked
request is event-sourced (profile.edit_approval.requested / .approved,
projected into profile_edit_approvals), so it survives a control-plane
restart and is visible to every replica; it expires 24 hours after the 202
(state: expired, approval answers 409) and the requester resubmits the
change to open a new request.
The digest is SHA-256 of the compact JSON spec bytes returned by the review
record, so an API or CLI reviewer can verify what was displayed before deciding.
Telling clients when to renew: ARI (F46)
If thousands of clients renew at the same fixed "30 days before expiry," they stampede — and if a certificate must be replaced early (a mass revocation), there's no way to tell them. ACME Renewal Information (ARI, RFC 9773) fixes both: the CA publishes a suggested renewal window per certificate, and clients renew within it.
trstctl's ACME server computes the window as the last third of the certificate's life. An ACME client can spread its renewals across that window. If trstctl's CA flags a certificate for early renewal, its published window opens immediately; clients that poll ARI can act on that signal.
The local window is served by trstctl's ACME server at
GET /acme/renewal-info/{certid} and used by its lifecycle scheduler for deployed
X.509 identities. Certificates can renew when that local window opens, even before
the fixed renew_before fallback. For certificates issued by an external ACME CA
through trstctl, the control plane polls that CA's renewalInfo and the scheduler
uses the retained window. A leaf that arrived through discovery has no authenticated
issuer; set ari_issuer_key_ids on the letsencrypt-type authority to the Subject
Key Identifiers of its issuing intermediates and a discovered leaf whose Authority
Key Identifier matches is polled the same way, with its posture labeled as matched
by issuer key id rather than by an issuance fact.
Operators inspect the same chain through the read-only
GET /api/v1/acme/ari/posture route, the trstctl-cli acme ari posture command,
or the ARI posture panel on Protocols. The authenticated route requires
lifecycle:read and PostgreSQL RLS limits every certificate and rotation-run row to
the caller's tenant. It reports whether ARI publication is served for that tenant,
the locally computed or published suggested window for each affected certificate, and whether the lifecycle
scheduler is pending, running, succeeded, or failed for that window. It never returns
certificate bytes, fingerprints, tenant IDs, account/order data, or private-key
material.
The public ACME route and the operator route answer different questions:
/acme/renewal-info/{certid} tells an ACME client when it should renew;
/api/v1/acme/ari/posture tells an authenticated operator what is being published
and whether trstctl's scheduler consumed it. If ACME is not mounted for the tenant,
the posture says not_served rather than pretending the certificate is published.
An empty items array honestly means that the tenant has no affected deployed
certificate rows.
Revocation: OCSP and CRLs (F47)
When a certificate must stop being trusted before it expires, you revoke it and
publish that fact two ways. A CRL is a signed list of revoked
serials, regenerated periodically; OCSP answers "is this one
revoked?" live, one certificate at a time. For its own hierarchy trstctl does both:
Revoke(serial, reason) emits ca.certificate.revoked to the tamper-evident log, and
GenerateCRL bumps the CRL number, signs a fresh list through the isolated
cryptography path, and emits a v3 ca.crl.published event (CRL DER, artifact kind,
shard metadata, delta base, validity window) so CRL state rebuilds from the log.
Small estates use the plain /crl/{tenant} full CRL; at scale the same publication
also serves /crl/{tenant}/manifest.json, /crl/{tenant}/shards/{index}, and
/crl/{tenant}/delta/{base}, so relying parties fetch bounded or RFC 5280 delta CRLs
instead of a 10-100M-row monolith. The OCSP responder signs with a delegated
responder certificate (OCSPSigning EKU + ocsp-nocheck) rather than the CA
certificate; rotations emit ca.ocsp_responder.rotated, and the responder runs in
its own bounded lane so a flood can't starve the API.
RFCs 6960 (OCSP), 5280 (CRL).
Revocation is typed and batchable: requests use an RFC 5280 named revocation reason
such as keyCompromise, cessationOfOperation, or privilegeWithdrawn (unknown raw
integers are rejected), and bulk revoke at /api/v1/certificates/bulk-revoke /
/api/v1/identities/bulk-revoke returns matched, revoked, skipped, and failed counts
so a wide incident response is explicit about partial success. OCSP responses echo a
valid OCSP nonce when the request carries one and sign with the delegated responder.
Each response is valid for ten minutes; a nonce-free response is reused only in the
first half of that lifetime, then signed again, so no relying party is handed one about
to expire. If the certificate route's durable event-ID index is catching up, it returns
503 with Retry-After: 1; repeat the exact request with the same
Idempotency-Key rather than creating a second revocation command.
CRL serving returns weak ETag validators and honors If-None-Match with 304 Not Modified so relying parties don't refetch an unchanged CRL. GET /api/v1/revocation/crls / trstctl-cli revocation crls and the Certificates console
expose the same distribution state (full CRL, shards, delta base, freshness window).
Bulk identity revocation requires both identities:write and the privileged
certs:issue authority. Every selected identity must pass the lifecycle policy and
attribute-based restrictions before any selected identity is changed. A denial is
not partial success: no revocation intent is queued for the selected batch.
When dual control is required, the bulk request cannot supply the exact one-use
approval for each identity and is refused. Use the individual reviewed revocation
workflow below; a standing approval does not authorize a bulk action.
To select actual inventory certificates rather than lifecycle identities, send
certificate_ids with 1–100 UUIDs and a named reason. An empty array is invalid.
Do not combine this field with ids, identity_ids, owner/issuer filters, kind,
or status. The same authorization, policy and approval restrictions apply, even
when no certificate matches. removeFromCRL is an unhold operation, not a valid
reason for this exact-certificate revocation command.
The exact path verifies the public certificate's signature against the served
issuing CA and requires an existing tenant-scoped issuer record. It does not guess
from an issuer name or serial, revoke same-owner siblings, or substitute the
internal CA for an external issuer. For external leaves, immutable issuance
evidence binds the exact public certificate to its configured CA adapter. The
request queues that authority's revocation command through the outbox. Its item
status is queued, counted in total_queued; it is not counted in total_revoked.
Missing verified authority or an unavailable adapter returns an explicit per-item
unsupported result. An adapter that cannot revoke through trstctl also fails that
exact item before an upstream outbox command is queued. HTTP 200 can contain queued
or failed items, so always inspect the returned counts and items.
One retained certificate.revocation.batch.applied command records the exact
selection and outcome. Its projection updates certificate inventory, the CA's
revocation record and a revocation.crl.publish outbox intent in the same database
transaction. If inventory and issuer disagree, it reconciles them while preserving
the issuer's first revocation time and reason. revoked therefore includes a
reconciled record; it is not a count of newly revoked serials at the CA. skipped
means inventory and the issuer already agree that the certificate is revoked.
External items use event schema version 2 and a revocation.publish outbox intent.
They change certificate status only after their recorded issuer accepts revocation;
their serials never enter the platform CA ledger or CRL. Issuer errors retain the
command for retry and leave revocation unconfirmed. The console automatically
checks the selected certificate; GET /api/v1/certificates/{id} provides the same
current result. An idempotent replay still returns the original queued receipt.
Reuse the same Idempotency-Key for the same authenticated route and request.
Retries recover the original result from retained events even after the HTTP
result cache expires or an append succeeds before the database transaction fails.
Changing the selection or reason under that key returns 409. CRL publication is
asynchronous: a committed result is not proof that publication or a relying-party
check succeeded. Follow the delivery outcome, verify signed CRL/OCSP data, and test
a client configured to enforce revocation before declaring containment complete.
The local CA denies a Good OCSP answer as soon as an exact revocation command
is accepted, even if its event append or SQL projection must be recovered after
a crash. CRL readers likewise refuse an older signed list while that command
is pending or while the current issuer ledger contains more revoked serials
than the latest full signed CRL. The outbox worker publishes the replacement;
an unavailable CRL during this window is an explicit distribution failure to
monitor, not evidence that a relying party has rejected the leaf. A revoked or
retired identity also stops queued local signing, deployment, and host renewal
work from receiving fresh authority while its terminal event is reconciled.
Certificate list and detail responses expose revocation_pending; identity
list and detail responses expose terminal_pending and the accepted
terminal_target_state. The console labels these pending states so an operator
can distinguish accepted authority removal from completed projection.
For one managed certificate, Certificates → Revocation & CT → Revocation center is the
safe operator path. It is a three-step journey: choose the X.509 identity and factual
RFC 5280 reason; fetch an effect-free server preview; then type the exact credential
name to execute. The preview is bound to the identity's current lifecycle version and
request fingerprint. It explains the event, same-transaction outbox intent,
asynchronous revocation.publish work, graph-derived affected systems, and verification
steps while writing no event and contacting no external system. Execution echoes that
reviewed version, so a certificate that changes after review fails with 409 and must
be reviewed again. After success, the center links directly to the immutable
identity.revoked audit evidence and the affected-systems graph. The lifecycle event
and outbox remain the only mutation authority; the console does not invent a second
browser-only revocation path.
For reason keyCompromise, a serving managed X.509 identity takes the compound
response in deployment connectors.
The preview binds the issuing CA and served leaf before one confirmation queues
separate CA revocation and host containment. The console reads both statuses by
the original request key; revoked is only the lifecycle projection and a
queued host receipt does not mean the listener stopped. Check each certificate
serial through signed CRL/OCSP and verify the live host with a fresh stock TLS
client. The ordinary single-effect transition is refused for a serving X.509
keyCompromise even if target metadata is missing: its CA-only success would
leave endpoint containment unproven. It remains available for other factual
revocation reasons and for an X.509 identity that has not entered a serving
state. If no exact managed host target exists, revoke the exact certificate
with its issuing CA and stop the host through an independently verified
procedure.
A key is recorded as compromised in the tenant from the moment the compromise is
reported, not from the moment the revocation publishes. Revoking an identity with
reason keyCompromise (the ordinary transition or the compound response) records,
in the same transaction, the keys of the certificates that revocation will revoke,
one certificate.subject_key.compromised event per certificate, before
revocation.publish runs. An exact certificate revocation for keyCompromise that
is queued at its external CA records the key when it is queued, whatever the CA
answers later, and a certificate revoked for keyCompromise directly records its
key as the revocation is recorded. From then on no certifying path in that tenant
will certify the key again (CA/B BR 6.1.1.3, condition 4). The ACME, EST, SCEP,
CMP, REST, Vault-compatible PKI, SPIFFE, agent enrollment, CA authority, break-glass
and external CA paths refuse a request for that key with the subject key is recorded as compromised in this tenant, so the replacement must use a new key. The
check stays inside the tenant; another tenant's records are never consulted. Each
record is the certificate's SubjectPublicKeyInfo SHA-256, and each issuance makes
one indexed lookup by tenant and hash. There is no cap, so a mass keyCompromise
revocation never blocks the replacement issuance it calls for. At startup the server
records the keys of revocations an earlier build projected. If the server stops
between an identity's revocation and its key records, the keys are recorded when
the revocation publishes instead. A certificate with no retained DER has no key to
record, and once a key is recorded as compromised it stays recorded, even if the
CA's revocation later fails.
For an X.509 identity, the preview checks the exact unrevoked certificates tied to
its issuance or verified deployment, including superseded leaves that could be
restored. If their recorded issuing CA cannot revoke through this integration,
ready is false and the warning names the certificate ID and directs the
operator to that CA. An identity transition can still stop local automation, but
an identity.revoked state does not prove that the external CA revoked any leaf.
Confirm the exact serial upstream and verify signed revocation evidence before
calling the incident contained. The preview checks one bounded page of
certificates; a selection above 100 is marked not ready for complete verification.
External distribution health is monitored separately from trstctl's own
publication state. An hourly leader derives distinct CDP and AIA OCSP URLs from
the certificate inventory and queues bounded revocation.probe work for a
network relay, so private PKI endpoints are checked from the network that uses
them. An operator with certs:write can also probe now (POST /api/v1/revocation/health/probe, trstctl-cli revocation health probe, or
Probe endpoints now in the console), for example right after fixing a relay's
trust bundle; requests inside one minute share one probe. The relay verifies CRL/OCSP signatures and exact certificate context, then
returns a signed lease-bound report. GET /api/v1/revocation/health and
Certificates → Revocation & CT expose fresh, expiring, stale, unreachable, and invalid
answers with latency, nextUpdate, relay identity, and evidence digest; non-fresh
answers create notifications. With agent_channel.probe_scanned_certificates on, a
certificate known only from a relay's network scan or a cloud listing is also checked for
its own serial, against the issuer the scan saw served with it or the provider returned
with it (else a known issuer that signed it); its certificate detail shows those
answers (whether the CRL lists the serial, the OCSP status) with the relay and evidence,
and the endpoint list stays one row per endpoint. An answer is expiring when its nextUpdate falls in the
last quarter of its own signed lifetime, capped at 24 hours: a seven-day CRL warns in
its last day and a ten-minute OCSP response in its last two and a half minutes, so a
responder that keeps signing short-lived answers reads fresh. A missing observation is
shown as unknown, never healthy.
This proves the relay's observation, not every relying party's fail-closed or
soft-fail configuration. Controlled CRL/OCSP responders cover repository CI; a
real Windows AD CS lab run remains external infrastructure evidence.
Dark-segment serving is a separate local-cache path. A network-role agent can load a
bounded --revocation-cache-config with multiple issuers and explicit local CRL and
OCSP routes. It verifies CRL issuer signature, number, and signed time window before
caching. For OCSP it also binds the request to the configured issuer and validates
the responder signature, requested serial, nonce, thisUpdate, and nextUpdate.
Only nonce-free answers are reused, and stale or invalid objects produce 503 with no
protocol bytes. Each mTLS heartbeat signs metadata per segment/issuer/protocol;
GET /api/v1/revocation/caches and Protocols → Revocation cache by segment expose
fresh, stale, empty, and error without exporting upstream URLs or cached objects.
An unreported relay is unknown, not healthy.
CT submission is served at POST /api/v1/revocation/ct-submissions /
trstctl-cli revocation ct-submit: the outbox queues a precertificate and final
certificate to configured RFC 6962 CT logs, recording ct.submission.queued then
ct.submission.delivered after add-pre-chain/add-chain. Name each log's public key
in log_public_keys and every SCT that log returns is verified against it and kept
(ct.sct.recorded: verified, or invalid with the reason); the leader then proves the
entry is in the log by checking an audit path against a tree head the same key signed
(ct.inclusion.recorded: verified with index, tree size and root; pending until the log
merges it; not included once a tree head signed 24 hours after the SCT still lacks it;
or a failed proof). A log without a key keeps its SCT unverified, and a log that serves
only the static CT API is not checked for inclusion. The certificate detail
(GET /api/v1/certificates/{id}, transparency) shows each log's answer.
Rogue and non-compliant certificate posture is served at
GET /api/v1/revocation/rogue-certificates / trstctl-cli revocation rogue-certificates and the Certificates console: the tenant's own
certificate_posture rules from its active policy module (see the policy engine
in policy and governance), unexpected CT findings from
monitored logs combined with policy violations such as weak keys, expired active
certificates, over-long public-TLS lifetimes, and missing owners/issuer metadata —
metadata and projection references only, never certificate PEM or private-key
material.
Revoke a rogue certificate at its issuer. On the Certificates page, choose a
revocation reason and use Revoke at issuer on a finding. A certificate in
inventory goes through the exact revocation, which reaches its recorded issuer.
A Certificate Transparency finding — a certificate trstctl never asked for, such
as one issued in your own vendor account outside trstctl — is revoked by serial
at the external CA you name; CAs whose revocation takes a serial are offered
(DigiCert, Sectigo, AWS Private CA, Google CAS, step-ca, EJBCA, Vault PKI), and so
is an ACME CA (N10). Each authority in GET /api/v1/external-cas says which way it
can revoke a certificate trstctl found (revokes_by_serial, revokes_by_certificate;
both false while it is unavailable), and the console offers authorities from those. An ACME CA identifies the certificate by its bytes, which
trstctl never held: trstctl reads them back from the Certificate Transparency log
that recorded the finding (through that CT source's own egress settings), checks
them against the finding's fingerprint and serial, gets its own ACME account
authorized for every DNS name in the certificate through the issuer's DNS-01
provider (an order it never finalizes), then asks the CA to revoke, as RFC 8555
§7.6 allows an account authorized for all of a certificate's names. A finding at
a precertificate entry revokes its issued certificate (N10-L1): trstctl looks for
another CT finding of the tenant's under the same serial whose log entry is the
issued certificate from the same issuer, reads that entry back from its own log
the same way, and sends it. Until a watched log has recorded the issued certificate,
the revocation waits with that reason. An entry that is not the finding's
certificate, a log the source no longer names, or a certificate with an IP
identifier is refused without
contacting the CA. The finding then shows the revocation queued, and revoked once
the CA confirms; a CA that has no certificate with that serial, or that refuses
the account, leaves it unconfirmed with the reason. Automation uses POST /api/v1/revocation/rogue-certificates/revoke
with discovery_id, authority_id and reason (identities:write plus
certs:issue), or trstctl-cli revocation rogue-certificates revoke -f req.json.
Where the private key lives: HSM/KMS (F26)
A CA's private key is the system's single most valuable secret — anyone who has it can
forge any certificate, so trstctl keeps it in hardware or a cloud key service that
signs without revealing it. An HSM/KMS backend implements one
interface (Backend → GenerateKey → a Signer that signs via the device); trstctl
supports PKCS#11 HSMs, TPM 2.0, YubiHSM 2, AWS KMS, Azure Key Vault, and GCP Cloud KMS.
Adding one is a single change because all cryptography goes through one isolated
path: key material never leaves the device — it lives in a separate isolated signing
service, wipeable in memory, and only signatures/public keys cross the wire. Every
backend must pass a conformance harness (ConformBackend) that signs a probe,
verifies it, and confirms a wrong message and a tampered signature both fail.
The release includes a dedicated cgo HSM signer artifact: its PKCS#11 adapter opens
the configured native module using stable token CKA_ID values across restarts, TPM
2.0 uses google/go-tpm persistent handles, and YubiHSM 2 uses Yubico's
yubihsm_pkcs11 ABI. The launched-binary gate proves SoftHSM/swtpm lifecycle behavior
with independent command-line readback; the default control-plane artifact stays
static and never loads a native module, keeping provider credentials and private-key
operations inside the separate signer.
CAP-KEY-05 — Multiple algorithms (RSA / ECDSA / Ed25519) + PQC — is served:
the profile path is POST /api/v1/profiles and
trstctl-cli profiles create -f profile.json. The profile API validates
allowed_key_algorithms through internal/crypto, then stores the accepted policy as
profile.created evidence. It accepts classical RSA, ECDSA, and Ed25519 labels
and, with PQC in the core since 2026-09-20, the PQC signature labels
Hybrid-ML-DSA-44-ECDSA-P256, ML-DSA-65, and SLH-DSA-SHA2-128s. Unknown labels fail closed, and ML-KEM stays out of
certificate-signing profiles because it's a key-encapsulation mechanism, not a signing
algorithm. internal/server/crypto_agility_served_test.go's
TestServedCryptoAgilityProfilesValidateBoundaryAlgorithms proves the served profile
create/list round trip; the PQC issuance proofs live under internal/pqc and
internal/pqcmigration.
The managed-key API spine is configuration- and license-gated for AWS KMS, Azure Key
Vault/Managed HSM, GCP Cloud KMS, PKCS#11, TPM 2.0, and YubiHSM 2 custody: once
managed_keys.enabled is true and managed_keys.provider selects aws,
azure-key-vault, gcp-kms, pkcs11, tpm2, or yubihsm2 (or
managed_keys.providers lists several to run at once, with optional per-tenant pins),
the control plane exposes:
GET /api/v1/managed-keys/custody— return a secret-free startup plan for all six providers, the default and every running provider, the caller tenant's pin, signer attachment, and exact blockers;POST /api/v1/managed-keys/preview— validate provider and algorithm without writing state or calling the signer/provider, then name the later execution effects and verification evidence;POST /api/v1/managed-keys— create a non-extractable KMS/HSM-resident signing key (extractable: false; no private material returned);GET /api/v1/managed-keys— read the tenant's event-projected key inventory with bounded cursor pagination, including keys from providers used before a deployment configuration change. A cloud KMS key (AWS KMS, Azure Key Vault/Managed HSM, GCP Cloud KMS) carries the provider's own dates (A168-L3):provider_created_at, when the provider created the key, andprovider_deletion_at, when it will destroy the key's material once deletion is scheduled. The isolated signer reads them from the provider after each lifecycle action (DescribeKey; the key version'screateTimeanddestroyTime; the key'screatedattribute and the deleted key'sscheduledPurgeDate) and records them with the result — a provider that cannot be read leaves the action to be retried — so trstctl's record holds the provider's dates rather than the moment its own operation completed (created_at). A custody proof records the creation date of a key created before this release; a result that reports no dates never erases recorded ones. The console's managed-key inventory shows both;GET /api/v1/managed-keys/{provider}/{key_id}— read the current state and public material of one exact provider/handle pair after a lost response or restart;GET /api/v1/managed-keys/operations/{operation_id}— read a tenant-scoped provider command's durablequeued,completed, orfailedstate. A mutation still running after its short synchronous wait returns202with this opaque ID and status URL, also when a read inside that wait runs out of its time; replaying the same idempotency key returns the same receipt;POST /api/v1/managed-keys/verify-custody— ask the isolated signer to sign a fresh random challenge with the exact owned key, then verify the signature against the saved public key. The response and inventory show the last proof status and time separately from lifecycle state. A historicalactivestate does not prove the provider still has the key; a new key readsnot_checkeduntil this challenge succeeds. The console's Refresh status updates both the list and the selected key. When provider retries are exhausted, the key readsunavailablewhile its lifecycle state remains historical; new failure audit events include the exact provider and action, plus the key ID when one already exists. A failed generation has no key ID to name. Older v1 failure events remain searchable by their operation ID;POST /api/v1/managed-keys/approvals— record a distinct custodian's approval for an opaque key handle androtate/revoke/zeroize;POST /api/v1/managed-keys/rotate— mint a successor key;POST /api/v1/managed-keys/revoke— disable the current key at the provider;POST /api/v1/managed-keys/zeroize— destroy a local device key, or request provider-side deletion for a cloud key. Cloud KMS responses and inventory readdeletion_pendingafter the provider accepts the request. That state means signing is blocked; it does not claim the provider has erased retained key material. Read the provider's deletion deadline with its stock client. Local PKCS#11, TPM, and YubiHSM keys readzeroizedafter confirmed device removal. Older cloud events that reportedzeroizedremain immutable; the API and console present their historical projection asdeletion_pendingrather than making an unsupported physical-destruction claim.
The Certificate authorities → Key custody console turns those first two routes
into a three-step ELI5 journey: choose a provider and algorithm, review proof that
nothing changed, then generate and manage the key. It lists configuration variable
and file-reference names but never accepts provider credentials, file contents, or
private-key bytes in the browser. A provider mismatch, disabled lifecycle, missing
signer attachment, non-effect-free preview, or any server blocker keeps generation
locked. The inventory remains visible after a browser refresh; selecting a key
performs a fresh tenant-scoped read. Keys from a provider no longer attached to
this deployment remain visible for audit, with actions disabled until that
provider is restored.
For a 202 receipt, the console keeps the selected key visible and shows the
operation ID. It checks for a terminal result while the tab is visible and
refreshes on return; Refresh status checks immediately. Copy the ID to
continue with the CLI after leaving the page.
Rotate, revoke, and zeroize require two distinct custodians to approve one exact
request before the requester retries the action. The console retains that
request's Idempotency-Key in first-party browser storage, scoped to the signed-in
tenant, requester, key, and action, through navigation and browser reloads. Return
to the same browser profile after approval and click the action again. The console
clears the saved request only after a terminal provider result. If site storage is
cleared or the requester switches browser profiles, a fresh action key opens a
new approval request and supersedes the old one; the custodians must approve the
new request. API and CLI callers must likewise retain and reuse their original
Idempotency-Key across the approval wait. A different key is a new intent.
The CLI mirrors those verbs under trstctl managed-keys, including list,
get <provider> <key_id>, and approve.
operation <operation_id> reads a pending command without issuing it again.
Approval requires keys:approve; lifecycle mutation requires keys:write; the
requester never counts as an approver; and every request is tenant-scoped, idempotent,
and recorded as a key-material-free event before its PostgreSQL outbox command reaches
the signer. A required gate launches the shipped control plane and cgo signer,
exercises all six providers end to end against faithful cloud emulators, SoftHSM, or
swtpm, and stops the signer mid-rotation to independently verify the resulting state
(see Pitfalls & limits for what that proves and doesn't).
Rotate, revoke, and zeroize first open an exact, immutable approval request when
dual control is required. A pending request returns HTTP 403 with
code=managed_key_approval_pending, approval_request_id, and intent_digest in
the problem body. The key is unchanged. The Key custody console shows this as a
waiting state and links to Requests waiting for approval; an independent
custodian approves there, then the requester retries the action. A denial,
expired request, missing approval authority, or ordinary permission failure is a
refusal and does not use the pending code. The requester cannot approve their own
request.
The same posture includes the served CAP-KEY-03 FIPS path: GET /api/v1/editions and
the Platform page expose the live FIPS POST booleans, make fips-build build target,
fips-capable build (GOFIPS140) CI gate, and internal/crypto boundary, keeping the
NIST CMVP product certificate as the external lab-certification residual.
Use it
Issue and govern through the served API and CLI:
# create a versioned profile (RA officer or admin)
trstctl-cli profiles create -f tls-server-90d.json
# list active profiles
trstctl-cli profiles list
A profile spec looks like this — note the explicit, enforced constraints:
{
"name": "tls-server-90d",
"spec": {
"allowed_key_algorithms": ["ECDSA"],
"min_ecdsa_bits": 256,
"allowed_ekus": ["serverAuth"],
"max_validity": "2160h"
}
}
A hybrid transition profile allows the hybrid key label and binds it to the protocols allowed to request it:
{
"name": "hybrid-web-30d",
"spec": {
"allowed_key_algorithms": ["Hybrid-ML-DSA-44-ECDSA-P256"],
"allowed_protocols": ["acme", "est", "scep", "cmp"],
"allowed_ekus": ["serverAuth"],
"max_validity": "720h"
}
}
Issuance happens through the enrollment protocols (ACME,
EST/SCEP/CMP), the private-CA hierarchy API, and the
external CA registry API, each of which calls the one issuance path with an
Idempotency-Key. Revoke from the incident flow in Incident response.
Pitfalls & limits
- Private-key custody is a deployment boundary. All six managed-key backends are census-served through the separate HSM signer artifact, but the operator must still provision an Enterprise BYOK license, the providers it runs (one, or several with per-tenant pins), credential files, IAM, network egress, and device/module trust — see configuration for the startup contract.
- Emulator proof is not deployment certification. The cloud gate uses faithful vendor-protocol emulators, PKCS#11/YubiHSM use a SoftHSM-backed ABI target, and TPM uses swtpm — stronger than an author-injected registry or unit test, but not a live cloud account, a physical customer HSM, or the device's FIPS certificate. The native bindings ship in the cgo HSM signer artifact, not the default static control-plane artifact.
- ARI scheduling covers trstctl-issued deployed X.509 identities. Certificates discovered from another CA can still be inventoried and risk-scored, but renewing them needs a configured issuer path that can replace that outside certificate.
- External CA registration is operator configuration. Tenants can list and use configured upstream CAs, but provider credentials aren't created through the tenant REST API.
- Revocation covers trstctl's own hierarchy. Third-party CA certificates are revoked through those CAs.
Reference
- CLI groups:
profiles,issuers,external-cas,certificates, andacme ari posture. - Served routes:
POST|GET /api/v1/profiles,GET /api/v1/profiles/{name}/versions/{version}, the effect-freePOST /api/v1/profiles/{name}/versions/{version}/restore/preview, append-onlyPOST /api/v1/profiles/{name}/versions/{version}/restore,POST /api/v1/certificates,GET /api/v1/external-cas,POST /api/v1/external-cas/{id}/issue,GET /api/v1/acme/ari/posture(lifecycle:read),POST /api/v1/ca/authorities/{id}/rotate,POST /api/v1/ca/authorities/{id}/rekey,POST /api/v1/certificates/bulk-revoke,POST /api/v1/identities/bulk-revoke. - Upstream CA adapters: AD CS, AWS Private CA, Azure Key Vault, DigiCert, EJBCA, Entrust, GlobalSign, Google CAS, Let's Encrypt/ACME, Sectigo, shell CA, Smallstep, Vault PKI, and Venafi TPP/TLS Protect.
- Key ceremony:
StartCeremony→ ≥threshold×Approve→CreateRoot/ImportExisting/CreateIntermediate. See the runbook. - Events:
ca.issue,issuance.profile_evaluated,ca.root.created,ca.authority.imported,ca.intermediate.created,ca.authority.rotated,ca.authority.rekeyed,ca.cross_signed,ca.certificate.revoked,ca.crl.published. - RFCs: 5280 (X.509/CRL), 6960 (OCSP), 9773 (ARI).
See also
ACME & DNS · Enrollment protocols · Certificate-profile guide · CA key-ceremony runbook · Signing-service design · glossary: CA, CSR, OCSP, CRL, HSM/KMS
Covers: F4, F48, F53, F46, F47, F26