Key custody: where every private key is generated and where it lives
Whose process created the private key, and whose disk holds it, is the question
an auditor asks first and the one a whitepaper is worst at answering. This page
answers it per credential kind, in one table, and a CI guard
(docs/custody_test.go) keeps the table matched to the code.
Read it as a map of the issuance boundary. A requester-supplied CSR or public key does not send its private key to the control plane. That observation cannot prove where the key was originally generated or whether a previous operation handled it. Generated by the control plane means this issuance did handle it, and says so.
Mechanism
A private key can be created in exactly three places in a trstctl deployment:
- The requester — your host, your workload, your device. A PKCS#10 request travels up, a certificate travels down, and the control plane signs something it cannot read a key from. This is the target shape for every credential kind.
- The isolated signer — CA keys only. They are created inside the signer process (AN-4), never leave it, and are never subject keys.
- The control plane — the remaining legacy paths, each named below with what replaces it.
Per certificate and per kind
The table below answers this per credential KIND, which is the right level for a design review and the wrong level for an audit. An auditor is not asking about ACME in general — they are asking about the certificate in front of them, and a kind-level table cannot say whether that one took the modern path or the deprecated server-keygen path retained beside it.
So every certificate now carries its own custody record (epic B5), written from what the issuing code actually did: origin, storage class, exportability, and the actor where one is known. The certificate detail page shows it, and the served API carries it on every certificate response.
For host-agent renewal, issuance initially records the facts known at CSR signing:
the key originated on the authenticated host agent and which agent generated it.
The control plane does not guess the final storage location before installation.
After the connector installs the certificate, the agent's v2 terminal receipt
signs the certificate fingerprint, origin, storage class, exportability, and
generator with the same key behind its mTLS certificate. Missing, partial,
connector-inconsistent, wrong-attempt, or altered custody is refused; the job is
not completed. The verified receipt appends certificate.custody.attested, and
that event projects the final four fields so replay reconstructs the same inventory.
Empty means unrecorded, and that is a third answer rather than a default. A certificate a network scan found has an origin nobody observed, and the console says so in words — "unrecorded is not the same as safe" — rather than letting a blank field read as reassurance. Custody written at issuance is never overwritten by a later discovery upsert, so a scan re-finding a certificate cannot quietly turn an audit fact back into an unknown.
Framework evidence packs aggregate those tenant-scoped rows. Their signed
manifest contains counts by the closed custody vocabulary and explicitly lists
every incomplete certificate plus the fields it lacks. The Policy console renders
the same gap list. A legacy row with only key_origin is incomplete evidence, not
silently counted as a complete custody record.
The table
| Credential kind | Key source for this issuance | Key stored where | Does this issuance handle the private key in the control plane? |
|---|---|---|---|
| ACME leaf (RFC 8555) | The ACME client | Wherever the client put it | No |
| EST leaf (RFC 7030) | The EST client | Wherever the client put it | No |
| SCEP leaf (RFC 8894) | The SCEP client / MDM | The device | No |
| CMP leaf (RFC 4210) | The CMP client | Wherever the client put it | No |
Identity leaf, subject_csr_pem supplied |
The requester | Wherever the requester put it | No |
| Identity leaf, no CSR, agent-executed target | The host agent | Installed into the target's connector-specific host store; transient copies stay in locked host memory | No — only the CSR travels up |
| Identity leaf, no CSR, retained control-plane target | The control plane | Tenant-bound encrypted recovery record; sealed deploy intent when a control-plane connector is configured | Yes — deprecated, see below |
| CA root / intermediate | The isolated signer | Inside the signer's key store or an HSM | No — it never leaves the signer |
| Agent enrollment identity | The agent | The agent's own key store | No |
| SSH host / user certificate | The requesting host or user | Wherever the requester put it | No |
| SPIFFE X.509-SVID, host-agent Workload API | The host agent | Transient locked memory in the agent, then returned to the workload over that host's UDS | No — only the public key travels up |
| SPIFFE X.509-SVID, control-plane compatibility Workload API | The control plane | Transient locked memory, then returned over the control-plane-local UDS | Yes — deprecated compatibility path, see below |
PKI-as-a-secret, csr_pem supplied |
The requester | Wherever the requester generated it | No |
PKI-as-a-secret, common_name supplied |
The control plane | Returned once in the response, not persisted | Yes — deprecated, see below |
| Automated renewal successor, agent-executed target | The host agent | Installed into the target's connector-specific host store; transient copies stay in locked host memory | No — only the CSR travels up |
Automated renewal successor, recorded subject_csr_pem |
The original requester | Wherever the requester kept that matching key; renewal creates no new subject key | No |
| Automated renewal successor, no CSR and retained control-plane target | The control plane | Tenant-bound encrypted recovery record; sealed deploy intent when a control-plane connector is configured | Yes — deprecated identity fallback, see below |
| Ephemeral workload credential (attested) | The attested workload | Wherever the workload put it | No — it presents its own public key |
| Direct attested X.509-SVID | The requesting workload | Not established by the attestation request | No — only the public key travels up |
ACME, EST simple enrollment, SCEP, CMP, and direct attested issuance record
key_origin=requester through certificate.recorded. They leave storage,
exportability, and the named key generator unrecorded: receiving a CSR or valid
workload proof does not establish those facts. The detail describes this issuance;
it does not attest the key's entire history. EST server key generation uses a
separate interface and does not acquire requester provenance from this rule.
Older rows without custody evidence stay unknown; the console does not invent
an explanation for missing evidence or silently backfill a security claim.
The three retained control-plane generators, and what replaces each
There are four Yes rows because the identity generator is reused by its
automated-renewal fallback. There are only three distinct control-plane subject-key
generators to remove.
Identity leaf without a CSR, including its renewal fallback. Supplying
subject_csr_pem is the primary direct-identity path: trstctl signs the request
you built and generates nothing. On renewal, the recorded CSR is honoured again,
so the requester keeps the same key and the control plane still generates nothing.
If the deployment target says executor=agent, first issuance and renewal take a
different branch before any mint. The control plane queues endpoint.renew; the
host agent generates the key, sends only a CSR upward, receives the certificate,
installs the pair, verifies the live endpoint, and signs the custody receipt. This
is the shipped B2 path, not future work.
The retained fallback applies only when there is no CSR and the target is not
agent-executed. It calls the same control-plane key generator for first issuance
and renewal, and records issuance.server_side_keygen. The key is held in locked
memory. Before signing, a tenant-bound encrypted recovery record retains the key
and CSR so a retry uses the same subject. Inventory records
key_origin=control_plane, key_storage=sealed_store, and
key_exportable=exportable. These describe the encrypted recovery material;
they do not assert hardware protection. Older records with missing exportability
remain unrecorded because no new issuance evidence establishes their history. If a
control-plane connector is configured, the bytes are also sealed into its outbox
deploy intent; otherwise no deploy material is emitted. Working copies are wiped
after the dispatcher finishes. This compatibility path is deprecated, and
the event identifies the flows that must move to subject_csr_pem or an
agent-executed target.
Lifecycle issuance also retains the exact public certificate template before the CA signature. The isolated signer journals that operation and returns the same signature on retry. An unfinished outbox command keeps its template and encrypted subject preparation beyond the normal idempotency retention window. Once delivery completes, normal retention can reclaim these preparation records. Retained deployment intents have their own lifetime. Historical commands that signed before this mechanism cannot reconstruct a lost certificate or key from a request ID.
If first issuance records the certificate but cannot queue its connector deployment, redelivery resumes deployment with the original encrypted key and recorded public chain. It does not sign another certificate. Recovery refuses missing key preparation, changed issuer or subject binding, mismatched material, and an expired or revoked certificate. A deployment already queued is not queued again by this issuance retry. These checks do not replace verification of the actual endpoint.
Control-plane compatibility Workload API. The production host agent now serves
the SPIFFE Workload API on the workload's machine. Its FetchX509SVID generates a
locked key before calling upward, sends only the public DER plus locally attested
selectors to the control plane, and returns the private half only over that host's
Unix-domain socket.
The control plane still composes its older Workload API socket during the
deprecation window. A workload that dials that compatibility socket receives a key
generated in the control-plane process, so that row remains Yes. The host-agent
Workload API is its replacement; the two rows stay separate until the compatibility
socket is removed.
PKI-as-a-secret with only a common name. /api/v1/secrets/pki now requires
exactly one custody mode.
Supply csr_pem and trstctl validates the self-signed PKCS#10 request through the
crypto boundary, signs it, and returns only the certificate; the matching key never
entered the control plane. Supplying common_name keeps the legacy brain-local convenience:
trstctl first commits an issuance.server_side_keygen deprecation receipt, then
generates and returns a keypair. If that receipt cannot become durable, issuance
fails before key generation. The Secrets console defaults to CSR mode and links the
legacy choice to its filtered Audit evidence. Vault/OpenBao clients use
pki/sign/{role} for CSR custody; pki/issue/{role} is the same deprecated
server-keygen choice.
What this page does not claim
Custody is about the subject key. It says nothing about whether a certificate reached its endpoint, which is a separate question with a separate answer, and nothing about how a requester protects the key once it has it — a key generated on your host and left world-readable is your risk, not a custody boundary trstctl can enforce.