Key custody: where every private key is generated and where it lives

Whose process created the private key, and whose disk holds it, is the question an auditor asks first and the one a whitepaper is worst at answering. This page answers it per credential kind, in one table, and a CI guard (docs/custody_test.go) keeps the table matched to the code.

Read it as a map of the issuance boundary. A requester-supplied CSR or public key does not send its private key to the control plane. That observation cannot prove where the key was originally generated or whether a previous operation handled it. Generated by the control plane means this issuance did handle it, and says so.

Mechanism

A private key can be created in exactly three places in a trstctl deployment:

• The requester — your host, your workload, your device. A PKCS#10 request travels up, a certificate travels down, and the control plane signs something it cannot read a key from. This is the target shape for every credential kind.
• The isolated signer — CA keys only. They are created inside the signer process (AN-4), never leave it, and are never subject keys.
• The control plane — the remaining legacy paths, each named below with what replaces it.

Per certificate and per kind

The table below answers this per credential KIND, which is the right level for a design review and the wrong level for an audit. An auditor is not asking about ACME in general — they are asking about the certificate in front of them, and a kind-level table cannot say whether that one took the modern path or the deprecated server-keygen path retained beside it.

So every certificate now carries its own custody record (epic B5), written from what the issuing code actually did: origin, storage class, exportability, and the actor where one is known. The certificate detail page shows it, and the served API carries it on every certificate response.

For host-agent renewal, issuance initially records the facts known at CSR signing: the key originated on the authenticated host agent and which agent generated it. The control plane does not guess the final storage location before installation. After the connector installs the certificate, the agent's v2 terminal receipt signs the certificate fingerprint, origin, storage class, exportability, and generator with the same key behind its mTLS certificate. Missing, partial, connector-inconsistent, wrong-attempt, or altered custody is refused; the job is not completed. The verified receipt appends certificate.custody.attested, and that event projects the final four fields so replay reconstructs the same inventory.

Where a host agent's private-key inventory reports a key file whose public key matches a certificate, the certificate detail lists it as Matching key seen on hosts (key_observations: the agent, the path, the file mode, whether the file is encrypted, its format, and when it was last reported). That is an observation of where a matching key sits, taken from the agent's metadata-only report and named by the agent's own certificate; it is never a custody record, so it does not change the custody fields or their summary. Only a certificate whose bytes trstctl holds (issued, adopted from a host, or imported) has a public key to compare.

Empty means unrecorded, and that is a third answer rather than a default. A certificate a network scan found has an origin nobody observed, and the console says so in words — "unrecorded is not the same as safe" — rather than letting a blank field read as reassurance. Custody written at issuance is never overwritten by a later discovery upsert, so a scan re-finding a certificate cannot quietly turn an audit fact back into an unknown.

Framework evidence packs aggregate those tenant-scoped rows. Their signed manifest contains counts by the closed custody vocabulary and explicitly lists every incomplete certificate plus the fields it lacks. The Policy console renders the same gap list. A legacy row with only key_origin is incomplete evidence, not silently counted as a complete custody record.

The table

ACME, EST simple enrollment, SCEP, CMP, direct attested issuance, and a certificate ordered through an external CA (POST /api/v1/external-cas/{id}/issue takes only the caller's CSR; A121) record key_origin=requester through certificate.recorded. They leave storage, exportability, and the named key generator unrecorded: receiving a CSR or valid workload proof does not establish those facts. The detail describes this issuance; it does not attest the key's entire history. EST server key generation uses a separate interface and does not acquire requester provenance from this rule. A certificate issued before its issuing path recorded custody gets the origin its own retained issuance event proves, and nothing else. The projection derives it from the event alone (so a rebuild reproduces it) and marks it derived: the certificate carries key_origin_derivation (the rule) and key_origin_proof_event_id (the event an auditor can read), the summary says the origin was derived and that the key's storage was not recorded, and the certificate detail names the rule. Each rule is one issuing path whose retained event shape, by itself, proves the origin, and it applies only to events older than the moment that path began recording custody itself:

Everything else stays unrecorded: PKI-secret leaves, managed-hierarchy leaves, edge delegations, identity issuance from the window when a CSR could be honored but custody was not yet recorded, and the oldest rows that carry no issuance key. A recorded origin always replaces a derived one; a derived one never replaces a recorded one. On upgrade, rows projected by an older binary are derived once by an atomic rebuild. A derived control-plane origin lets an identity renew through the control-plane path while the tenant allows it; a derived requester origin still renews only through its requester's protocol, because no retained CSR links it to the renewal.

The four retained control-plane generators, and what replaces each

There are five Yes rows because the identity generator is reused by its automated-renewal fallback. There are four distinct control-plane subject-key generators: the three deprecated paths below and the reissue step of a PQC migration run that predates host-csr migration. The tenant's switch governs all four. Kubernetes TLS auto-issuance is not one of them: the agent running in the cluster generates its keys (below).

Identity leaf without a CSR, including its renewal fallback. Supplying subject_csr_pem is the primary direct-identity path: trstctl signs the request you built and generates nothing. On renewal, the recorded CSR is honored again, so the requester keeps the same key and the control plane still generates nothing.

If the deployment target says executor=agent, first issuance and renewal take a different branch before any mint. The control plane queues endpoint.renew; the host agent generates the key, sends only a CSR upward, receives the certificate, installs the pair, verifies the live endpoint, and signs the custody receipt. This is the shipped B2 path, not future work.

Before requesting a signature, the host agent seals the CSR and private key in its tenant-bound, 0600 host rollback directory. If the job is claimed again after a lost reply or agent restart, it reuses that exact CSR and key; the signer's slot-zero, job-scoped idempotency key returns the same signed leaf even when the claim attempt changes. A different CSR for that durable job is refused, including when an older issuer recorded an attempt-scoped key. The agent removes the pending copy only after the control plane accepts its successful terminal report. If the acceptance reply is lost, the agent first replays its encrypted original report after restart, clears only that job's pending key slot, and then acknowledges the local report before claiming another job. Keep --host-rollback-dir on durable local storage: losing or corrupting that state makes the agent refuse recovery rather than silently generating another subject key for the same job.

The retained fallback applies only when there is no CSR and the target is not agent-executed. It calls the same control-plane key generator for first issuance and renewal, and records issuance.server_side_keygen. The key is held in locked memory. Before signing, a tenant-bound encrypted recovery record retains the key and CSR so a retry uses the same subject. Inventory records key_origin=control_plane, key_storage=sealed_store, and key_exportable=exportable. These describe the encrypted recovery material; they do not assert hardware protection. Older records with missing exportability remain unrecorded because no new issuance evidence establishes their history. If a control-plane connector is configured, the bytes are also sealed into its outbox deploy intent; otherwise no deploy material is emitted. Working copies are wiped after the dispatcher finishes. This compatibility path is deprecated, and the event identifies the flows that must move to subject_csr_pem or an agent-executed target.

Per-tenant switch. Each tenant decides whether this fallback may run at all: GET/PUT /api/v1/tenant/key-generation (access:read/access:write), trstctl tenant key-generation show|set, or Platform → Configuration → Control-plane key generation. A tenant registered after the switch starts with it off; the evaluation preset starts with it on and records why; a tenant that predates the switch is decided once, on the leader's first start of a version that has it, from its own use: it stays on if any of its certificates' custody (recorded, or derived from the retained issuance event) says the control plane generated the key, if any certificate from a path that could have generated one still has an unrecorded origin, if it has a retained issuance.server_side_keygen record or an X.509-SVID from the compatibility socket, or if that socket serves it; otherwise it is turned off. The decision keeps its evidence, shown beside it, and never replaces an operator's own decision; until it is made, the tenant keeps the behavior it had. Every change names who made it and why in a tenant.control_plane_keygen.set event. With it off, an identity issuance that would use the fallback is refused before anything is recorded (409, control_plane_keygen_off, naming subject_csr_pem, an agent-executed target, or turning the switch on), and the console's issue review says so and takes a pasted CSR. Work queued before the change is refused when it reaches the mint, before a key exists; it completes if the tenant turns key generation back on. The renewal of an identity whose key the control plane generated fails its rotation run with the same reason. The switch also governs PKI-as-a-secret by common name and the Vault-compatible pki/issue/{role}, the compatibility Workload API socket, and the reissue step of an older PQC migration run (all below).

Lifecycle issuance also retains the exact public certificate template before the CA signature. The isolated signer journals that operation and returns the same signature on retry. An unfinished outbox command keeps its template and encrypted subject preparation beyond the normal idempotency retention window. Once delivery completes, normal retention can reclaim these preparation records. Retained deployment intents have their own lifetime. Historical commands that signed before this mechanism cannot reconstruct a lost certificate or key from a request ID.

If first issuance records the certificate but cannot queue its connector deployment, redelivery resumes deployment with the original encrypted key and recorded public chain. It does not sign another certificate. Recovery refuses missing key preparation, changed issuer or subject binding, mismatched material, and an expired or revoked certificate. A deployment already queued is not queued again by this issuance retry. These checks do not replace verification of the actual endpoint.

Control-plane compatibility Workload API. The production host agent now serves the SPIFFE Workload API on the workload's machine. Its FetchX509SVID generates a locked key before calling upward, sends only the public DER plus locally attested selectors to the control plane, and returns the private half only over that host's Unix-domain socket.

The control plane still composes its older Workload API socket during the deprecation window. A workload that dials that compatibility socket receives a key generated in the control-plane process, so that row remains Yes. Where serve_local_workloads keeps that socket issuing, the tenant's control-plane key generation switch still decides before every X.509-SVID key: with it off, FetchX509SVID is refused with PermissionDenied, naming the host agent's --workload-api-socket, and the refusal is recorded like the socket's other uses. JWT-SVIDs carry no key and are still issued. The host-agent Workload API is its replacement; the two rows stay separate until the compatibility socket is removed.

PKI-as-a-secret with only a common name. /api/v1/secrets/pki now requires exactly one custody mode. Supply csr_pem and trstctl validates the self-signed PKCS#10 request through the crypto boundary, signs it, and returns only the certificate; the matching key never entered the control plane. Supplying common_name keeps the legacy brain-local convenience: trstctl first commits an issuance.server_side_keygen deprecation receipt, then generates and returns a keypair, while the tenant allows control-plane key generation. If that receipt cannot become durable, issuance fails before key generation. With the tenant's switch off, the request is refused before anything is recorded or generated (409, control_plane_keygen_off, naming csr_pem), and pki/issue/{role} is refused the same way, naming pki/sign/{role}. The Secrets console defaults to CSR mode and links the legacy choice to its filtered Audit evidence. Vault/OpenBao clients use pki/sign/{role} for CSR custody; pki/issue/{role} is the same deprecated server-keygen choice.

Kubernetes TLS auto-issuance. A Kubernetes TLS certificate is delivered by the host-role agent running in the cluster: a k8s_ingress_gateway source names it in cluster_agent_id, and each TLS block becomes a k8s_secret destination with executor=agent. The agent generates the key inside the cluster, sends only a CSR, writes the TLS Secret, reads it back and returns a signed receipt, so the key never leaves the cluster and the control plane never generates it. The certificate records key_origin=host_agent and key_storage=kubernetes_secret. A k8s_secret destination saved without executor=agent is refused, because it would route through control-plane key generation. A source with no agent bound records each Ingress or Gateway TLS resource it finds and reports it not_delivered with the reason; the run completes as partial, and no key, certificate or issuance.server_side_keygen event is made, whatever the tenant's switch says. Certificates that earlier versions minted for these resources (with a control-plane key that was wiped, never delivered to the cluster) stay in the inventory with key_origin=control_plane. cert-manager against trstctl's ACME directory remains another way to serve Kubernetes TLS with a cluster-generated key.

PQC migration reissue from an older run. A PQC migration started today binds each certificate asset to a host target and runs as host-csr migration: the host agent generates the post-quantum key. A run started before host-csr migration existed may still hold reissue work whose step generates an ECDSA P-256 and ML-DSA-44 key in the control plane, builds the hybrid CSR, and destroys the key after issuance. That step now records key_origin=control_plane, records an issuance.server_side_keygen event, and is refused before any key exists while the tenant's switch is off, as a retryable no-effect; the replacement is to start the migration again with host-csr.

Device-bound claims a TPM signed

The custody record above is the control plane's own account of what it did: evidence, not an attestation. A certificate can also carry a claim the hardware holding its key signed, and that anyone can re-check without trusting trstctl's account.

Today the claim comes from ACME device-attest-01, which a certificate profile turns on explicitly. The device's TPM attests the key it will put in its CSR (TPM2_Certify by the TPM's attestation key, in the WebAuthn TPM format), over a challenge that binds the tenant, account, order, nonce, identifier and the key itself. When the attributes the TPM certified for that key say the TPM generated it (sensitiveDataOrigin) and will release it neither to another TPM (fixedTPM) nor under another parent (fixedParent), the issued certificate records custody device / device_bound / non_exportable, generated by the TPM's attestation key (tpm-ak:sha256:<digest of its certificate>), and keeps the attestation and its binding exactly as they were verified (certificate.custody.device_attested). An attestation of a key without those attributes still proves the device and validates the challenge, but the certificate keeps the requester origin: the key could be duplicated to another TPM.

Re-check a claim offline, against TPM roots you choose:

trstctl-cli certificates key-attestation <certificate-id> > evidence.json
trstctl-cli certificates verify-key-attestation --evidence evidence.json --roots tpm-roots.pem

The verifier checks the attestation under your roots at the time it was first verified, over SHA-256 of the retained binding; that both the binding and the attested key are the certificate's key; and the key's attributes. It needs no connection to trstctl once you hold the evidence, which includes the certificate.

What the claim does not say: it is what the TPM certified when the certificate was issued, not the device's state since, and it is only as strong as the TPM roots you trust: a root that certifies attestation keys carelessly makes it worthless. The claim exists for certificates issued through device-attest-01. A host agent that generates endpoint keys in its own TPM, so certificates for agent-executed targets carry the same claim, is what comes next.

What this page does not claim

Custody is about the subject key. It says nothing about whether a certificate reached its endpoint, which is a separate question with a separate answer, and nothing about how a requester protects the key once it has it — a key generated on your host and left world-readable is your risk, not a custody boundary trstctl can enforce.