Secrets — store, issue, rotate, and encrypt the credentials machines use

What it is

A secret is any sensitive value software needs but shouldn't expose: a database password, an API token, an encryption key. trstctl is a full secrets platform alongside its certificate work: it stores secrets encrypted, hands out short-lived ones on demand, rotates them safely, encrypts data on behalf of apps, syncs secrets to other platforms, and governs who can read or change them.

The mental model: think of a bank. The vault stores valuables encrypted (the secret store). The safe-deposit clerk issues a temporary key that self-destructs after an hour (dynamic secrets). The armored-car service moves valuables to other branches (secret sync). The teller window encrypts your deposit without you ever seeing the master key (encryption-as-a-service). And every action needs ID and is logged (auth + approvals + audit).

One honest note up front. Most of the secrets domain is now served on the running control plane: the secret store (CRUD + rotation), dynamic secret leases, one-time secret sharing, the dynamic PKI secret, machine login, secret-sync, secret scanning, workload injection, ephemeral API keys, and the Vault/OpenBao common compatibility paths are all mounted. The /api/v1/secrets/* surface is off by default (secrets.enable_api) and fail-closed when off; ephemeral API keys live under /api/v1/ephemeral/api-keys since they mint tenant credentials, not stored secret values. Transit encryption-as-a-service is served separately at /api/v1/transit/* (transit CLI group, keys:* RBAC scopes). KMIP is served as an opt-in mTLS listener (protocols.kmip.*) for AES-256 SymmetricKey Create/Get/Locate/Revoke/Destroy and for RSA and ECDSA key pairs that are activated, sign and verify on the server, with client attributes; remaining gaps are in Current limitations.

Why it exists

Leaked secrets are one of the most common breach causes: the traditional approach — long-lived secrets copied into config files, environment variables, images, and CI — spreads them everywhere and never expires them. trstctl attacks the problem from every angle: encrypt at rest, prefer short-lived/dynamic secrets that can't be hoarded, rotate long-lived ones automatically, never let a secret value touch a log or disk it shouldn't, and wrap access in approvals and a tamper-evident audit trail. Secret material lives in wipeable []byte buffers zeroed after use, never a Go string — Go copies strings freely, so a value placed in one can linger in memory beyond your control.

How it works

How every secret is encrypted at rest

trstctl uses envelope encryption through the single isolated cryptography path. Each secret gets a fresh per-secret data key (DEK, AES-256-GCM), itself encrypted under a master key-encryption key (KEK), bound to the secret's tenant and path so a sealed blob can't be moved elsewhere. The KEK loads at startup from TRSTCTL_SECRETS_KEK_FILE (0600), stays transient, and is zeroized. Rotating protection means re-wrapping small DEKs, not all your data.

A tenant can also move off the deployment KEK into its own tenant key domain: a domain KEK kept under an operator wrapper (a local wrapper file, or an HSM key the isolated signer holds), so sealing one tenant never touches another. The console's tenant custody panel and trstctl-cli platform tenant-key-domain serve its lifecycle: migrate (create the domain KEK and re-wrap the tenant's data), seal and unseal (a reversible revoke), and rotate (a new generation of the domain KEK, under the same or another wrapper, with every stored DEK and the tenant's event history re-wrapped and both generations readable until the switch). Rotation re-wraps DEKs, not payloads, so backups, archives, and exports taken before it still open with the previous generation through its wrapper. See configuration.

The deployment KEK itself rotates as well. The operator tenant (secrets.kek_operator_tenant_id) moves it from the local key file to a key under an operator wrapper file or an HSM key the signer holds, from the console's deployment key panel or with trstctl-cli platform deployment-kek rotate. New data is wrapped under the new generation at once; a background pass re-wraps every stored data key, the history of tenants that have not moved to their own domain, and the deployment's sealed files, then reads them all back before the previous generation is retired. Credentials stay readable throughout, and afterwards only through the wrapper. See configuration.

The native secret store (F63)

A served, tenant-isolated key-value store for application secrets: POST /api/v1/secrets/store/preview checks an exact create request without changing anything, POST /api/v1/secrets/store creates version 1, PUT /api/v1/secrets/store/{name} writes the next version, and GET /api/v1/secrets/store/{name} reveals only the latest value to a secrets:read caller. Metadata responses list names, versions, and timestamps only, never values.

The console's create flow has two clear steps: enter the name, owner, and value; then review the server's plan. Preview applies the same tenant-owner and duplicate name checks as create, but writes no row, event, audit record, idempotency record, outbox intent, or external effect. It never returns the value. Instead it returns a server-keyed fingerprint that changes if the tenant, caller, name, owner, or value changes. Editing any input throws away the reviewed plan. Only a current, ready plan reveals Create reviewed secret; creation remains a separate idempotent mutation.

Headless operators use the same oracle with trstctl-cli secrets store preview -f secret-create.json, then pass that exact JSON body to secrets store put. Keep the body in a permission-restricted file or pipe it with -f -; do not place plaintext on a shared command line or in QA evidence. A ready preview is evidence about the request, not a reservation: the create call rechecks authority and current state.

Every create, rotation, and recovery stores a row in secret_store_versions under PostgreSQL RLS and emits secret.version.written without plaintext. GET /api/v1/secrets/store/history/{name}?version=N reads a prior version, and POST /api/v1/secrets/store/recover/{name} with body {"at":"2026-06-25T12:00:00Z"} recovers whichever version was current then as the next monotonic version, keeping rollbacks auditable. DELETE /api/v1/secrets/store/{name} purges the current row and sealed history for that tenant; another tenant gets a 404 — both tables are RLS-scoped.

Values may contain ${secret.path} placeholders: a normal read returns the literal value, while GET /api/v1/secrets/store/{name}?resolve=true expands references within tenant/permission scope instead. Cycles like a -> b -> a return a structured 409 with the cycle path; missing references return a normal 404.

POST /api/v1/secrets/store/import accepts {"prefix":"optional/path","values": {"name":"value"}} for 1–100 new secrets. Its one event holds tenant/name-bound sealed values, not plaintext. Per-name fences serialize concurrent single-secret and batch mutations; projection commits every version-1 row and one metadata-only batch receipt in one PostgreSQL transaction. An exact Idempotency-Key retry returns the same receipt. Any existing or reserved name returns 409 before publishing a partial batch. The CLI command is trstctl-cli secrets store import -f import.json; the console reviews names before execution and never displays values. Keep the input file private and remove it after use.

The served store seals through a versioned binary container, its KEK loaded into locked, zeroizable memory at startup, never a raw byte slice on the heap. An older store core, kept for legacy event replay and compatibility, holds its KEK the same way and still replays both the binary container and earlier JSON-envelope history; new writes go through the path above.

Vault/OpenBao-compatible common API

Teams migrating from Vault or OpenBao can point a stock vault CLI at trstctl — a compatibility shim over the served secret store and dynamic PKI secret. Enable the same surface (secrets.enable_api) and use a tenant API token (secrets:read/secrets:write) as X-Vault-Token, which is what the Vault CLI sends:

export VAULT_ADDR=https://trstctl.example.com
export VAULT_TOKEN=trst_...

vault login -no-store "$VAULT_TOKEN"
vault kv put secret/payments/db username=payments password='correct horse battery staple'
vault kv get -format=json secret/payments/db
vault write -format=json pki/sign/default csr=@payments.csr ttl=1h
vault read -format=json database/creds/readonly
vault lease revoke -sync database/creds/readonly/lease-...

Supported paths are intentionally small:

The served compatibility contract also includes tenant-authored KV v2, PKI, and transit mounts, Vault ACL policy management, common transit operations, the cubbyhole, response wrapping, and the native dynamic leases at the Vault database, AWS and Kubernetes engine paths. The engines' configuration paths are not served (providers are operator-configured); the native trstctl API remains the full surface. The machine-readable contract is pinned separately as docs/contracts/vault-openbao-compat.openapi.json, since /v1 paths preserve Vault/OpenBao wire shapes. Mutating calls accept Idempotency-Key. For the stock CLI's KV put and PKI issue/sign commands, trstctl derives a token-keyed digest of method, path, and body when the header is absent, so an exact retry cannot mint a duplicate certificate or secret version. A dynamic-credential read without the header opens a new lease each time, as Vault does; sys/leases renew targets an absolute expiry and revoke is a state transition, so a retry cannot extend or revoke twice. For transit writes, pass the CLI's -header option explicitly. Use a fresh key for each distinct mutation and reuse that key when retrying the same request:

idem="$(openssl rand -hex 16)"
vault write -header="Idempotency-Key=$idem" transit/keys/payments type=aes256-gcm96

The same header option applies to transit encrypt, decrypt, rewrap, HMAC, sign, and verify. The stock CLI sends these vault write calls with HTTP PUT; the compatibility API serves both POST and PUT for the listed transit paths. Vault transit verify/{name} accepts a signature only when its public key belongs to a retained version of that named key in the current tenant. Rotation keeps older signatures verifiable under the same name, while another name or tenant returns valid: false. The stock CLI's vault secrets enable/disable and vault policy write/delete commands also work without a custom header: the fallback is bound to the last event for that exact mount or policy, so a retry does not append twice and a later delete/recreate cycle is a fresh operation. Disabling a tenant-authored KV mount closes that namespace. Re-enabling the same path starts an empty namespace, so previously written values cannot reappear through the Vault-compatible route. The sealed predecessor remains subject to the native secret store's retention and erasure policy; mount disable is not a physical-erasure receipt. An identical request from the same token replays its original result; force a fresh issue with a different explicit Idempotency-Key, or use the native /api/v1/secrets/pki route. Transit mutations require an explicit Idempotency-Key so intentional repeated rotations stay distinguishable.

The developer secrets experience (F64)

The Use secrets in apps workspace at /secrets/developer turns one secret name into an exact, value-free access plan. Choose the secret, environment variable, and whether references should resolve; then select Review access plan. The served POST /api/v1/secrets/access/preview route proves the caller can open the current tenant-scoped version, immediately wipes those bytes, and returns only metadata: the required secrets:read permission, current version, CLI arguments, HTTP path, TypeScript example, recovery steps, verification steps, and a server-keyed plan fingerprint. It writes no event or audit row and makes no external call.

Only a ready, unchanged plan enables Run reviewed access test. That test performs the real served read, checks that the returned name and version still match the review, and renders a value-free receipt. A denied read can be retried with the same review. Changed inputs or a changed secret version invalidate the plan and require a new review. Missing references, cycles, wrong-tenant names, and invalid environment variables fail closed with a specific recovery step. Copyable CLI, HTTP, and TypeScript examples come from the server response, so the portal cannot drift from the API contract or invent a secret value.

The CLI exposes the same plan before the process boundary:

printf '%s\n' '{"name":"db/password","env_var":"DB_PASSWORD","resolve":false}' > access-plan.json
trstctl-cli secrets access preview -f access-plan.json

After review, trstctl-cli run fetches named secrets from the served store and runs your program with them in the child environment, never writing the value to disk, via the normal GET /api/v1/secrets/store/{name} RBAC path:

trstctl-cli run --secret DB_PASSWORD=db/password -- env
trstctl-cli run --resolve --secret DATABASE_URL=app/db/dsn -- ./payments-api

The --resolve flag maps to ?resolve=true; without it, a value like ${secret.app/db/password} passes through literally instead of expanding further. After the child exits, trstctl wipes the byte-backed copies fetched — the OS environment remains an edge string API, so use run for trusted processes and skip debug commands printing the full environment.

Developers can read with trstctl-cli secrets store get NAME --resolve=true for the same opt-in resolve/cycle-detect behavior, scoped to the caller's tenant and RBAC. For multiple new names, trstctl-cli secrets store import -f import.json sends one atomic request and prints the metadata receipt. The file contains secret values; use a private file, or -f - for standard input, and do not log its content.

Dynamic secrets (F65) and PKI-as-a-secrets-engine (F67)

Instead of a long-lived secret to steal, dynamic secrets are minted on demand, scoped, and time-limited by a lease; on expiry trstctl revokes the credential automatically where the provider supports native revocation, even across a restart, since the revocation intent is journaled first to a durable outbox and delivered at-least-once. Eight backend families ship behind one interface, each backed by real infrastructure, not a stub: PostgreSQL, MySQL, MongoDB, AWS STS, GCP IAM, Azure Entra, Kubernetes ServiceAccount tokens, and Redis ACL users.

The control plane mounts the lease lifecycle when secrets.enable_api is on and secret_integrations.dynamic_providers supplies tenant-bound endpoints, allowed roles, maximum TTLs, egress policy, and file:/secret:// credential references (buildRunDeps wires the registry). A postgresql, mysql or aws-iam provider with allow_static_rotation: true also serves the static-rotation worker (F37): it mints credentials with no native expiry for the roles in allowed_roles, and trstctl retires them explicitly after their successor is verified. Issuance commits a pending lease and sealed outbox command first, so only the outbox worker calls the provider. Providers with a native lookup or replay identity recover the same identity after a crash; AWS STS instead fences an uncertain AssumeRole call and never blindly retries it. Cloud/Kubernetes endpoints require HTTPS; allow_insecure_loopback is a same-host-emulator exception limited to localhost, 127.0.0.0/8, or ::1.

The MySQL backend requires an explicit account_host for every generated account. Wildcard host patterns need the separate allow_wildcard_account_host operator opt-in; omitting the host never creates a global user@'%' login. The provider catalog shows both settings before issue.

• GET /api/v1/secrets/leases/providers returns nine safe setup recipes (including the aws-iam compatibility name for aws-sts) plus this tenant's real configured provider IDs, allowed roles, maximum TTLs, and runtime configuration revisions. It never returns endpoints, native bindings, credential references, or provider credential values. Configuration remains a startup-admin operation: edit the server configuration or mounted secret reference and restart the control plane; the console does not turn infrastructure credentials into browser form fields.
• POST /api/v1/secrets/leases/preview validates the exact configured ID, role, TTL, caller, tenant, and running configuration revision. It opens no credential reference, calls no provider, writes no event/audit/idempotency/outbox state, and returns the exact execution effects, recovery steps, verification steps, CLI command, and a server-keyed request_fingerprint.
• POST /api/v1/secrets/leases issues one credential copy for a provider, role, and TTL, guarded by secrets:write plus Idempotency-Key. The console always supplies the current preview fingerprint; changed inputs or a restarted/reconfigured provider return 409 before a provider call. API clients may omit the fingerprint for backward compatibility.
• GET /api/v1/secrets/leases/{lease_id} returns lease metadata only, never replaying the credential after first issue. expires_at is the lease deadline: the expiry worker queues revocation on its next pass (normally every 30 seconds), and provider access can continue during delivery or retries. hard_expires_at is the original renewal ceiling, not proof of native credential expiry. revocation_status is none, pending, completed, or failed; only completed with revocation_completed_at confirms the provider removal operation, or for AWS STS, that its native session expiration has passed. revoked_at records when the control plane queued the request. Older immutable operation receipts may omit these additional fields; GET returns current durable evidence. If provider creation finishes after expires_at, issued_at records the real late completion time, the API withholds the expired credential, and the expiry worker queues native revocation. A retry that starts after expires_at fails before provider contact. Check revocation_status=completed before assuming the backend identity is gone. A one-line PostgreSQL file: DSN may end in CR/LF; the PostgreSQL adapter removes only that terminal line ending before parsing it.
• AWS dynamic leases use STS AssumeRole, with an explicit role ARN binding and a 15-minute to 12-hour maximum TTL. The aws-iam provider name remains a compatibility spelling for this issuer; it does not create IAM users or long-lived access keys. Configure the caller's signing key through a mode-0600 file: or tenant-scoped secret:// reference. A short-lived caller also needs session_token_ref. The issued credential contains an access key ID, secret access key, session token, and expiration; native_expires_at on metadata is the AWS session expiry. STS has no per-session delete call: early revoke closes the trstctl lease and leaves revocation_status=pending until native expiry. AWS can still accept the old session during that window. Set the shortest workable TTL and replace dependent workload credentials promptly. A saved AssumeRole result can be recovered with the original idempotency key. If AWS may have accepted the request but its response was lost, trstctl fences that operation as indeterminate and does not submit it again; reconcile it with CloudTrail before starting a new lease.
• POST /api/v1/secrets/leases/{lease_id}/renew extends a lease without returning the credential again for renewable backends. Kubernetes TokenRequest tokens are immutable: renewal issues a new bound token and lease, confirms provider-side revocation of the predecessor, and returns the replacement credential once with a new lease ID. Store that token only in the workload's protected credential location, update the lease ID used for later renew/revoke calls, and verify the previous token is rejected with kubectl. A Kubernetes replacement must have at least 600 seconds remaining and fit within the predecessor's renewal limit. AWS STS renewal likewise issues a replacement lease and reveals one new session; the predecessor remains usable in AWS until its own native_expires_at. A replacement needs at least 900 seconds of requested validity and must fit within the original renewal limit.
• POST /api/v1/secrets/leases/{lease_id}/revoke closes the lease and queues backend revocation through the outbox worker. If GET later reports revocation_status=failed, correct the provider outage and POST the same revoke path with a new Idempotency-Key. That creates a separate audited outbox attempt, preserves the failed predecessor, and returns pending; use GET until completed, then test the old credential with the native client. Replaying an earlier key returns that earlier immutable receipt and never queues another provider call. The Workloads console can load an existing lease ID after a restart, show failed removal, and retry it. A revoked lease is unusable in trstctl while provider cleanup is pending, but native access can persist.

The console presents the same contract as three small steps: Choose connection → Review exact plan → Use and retire. It selects only a real configured provider ID and an allowlisted role; the old free-text provider/role form is gone. The first step shows the selected backend's safe prerequisites and a disclosure containing all eight built-in recipes. Review must prove zero writes and zero calls before Create reviewed credential appears. The issued credential lives only in the reveal-once panel; dismissing it, navigating away, reaching the lease deadline, losing the metadata read, or revoking the lease removes the browser copy, while durable metadata remains available for renewal, revocation, and verification. The visible receipt refreshes every five seconds and on return to the tab. A revoke response acknowledges queuing; its idempotent replay retains that original result even after the provider finishes. Read current metadata and test the same login against the provider to confirm that access has ended.

PKI-as-a-secrets-engine plugs certificate issuance into the same operator workspace. The recommended csr_pem mode accepts one self-signed PKCS#10 request, signs it through the isolated signer, and returns only the short-lived certificate. The mutually exclusive common_name compatibility mode generates and returns the leaf key from wipeable memory. That mode is deprecated: its issuance.server_side_keygen receipt must be durable before key generation begins, and its idempotent replay returns the original sealed response without generating a second key. The console defaults to CSR mode; Vault/OpenBao uses pki/sign for the same custody model and retains pki/issue only as the legacy equivalent. Both key-returning routes run only while the tenant allows control-plane key generation (PUT /api/v1/tenant/key-generation, off for tenants registered since that switch); with it off they are refused before anything is recorded, naming the CSR route. Native issuance returns certificate_id alongside the serial and public PEM. The event projects that exact leaf into tenant certificate inventory, records its subject-key origin, and queues its initial signed CRL publication. Use POST /api/v1/certificates/bulk-revoke with certificate_ids: [certificate_id] and a named RFC 5280 reason; then check the certificate row and signed CRL or OCSP response. That mutation requires certs:issue and identities:write in addition to secrets:write used for issuance. The console keeps a link to the exact revocation selection after its one-time certificate reveal is dismissed.

The console does not jump from those inputs straight to signing. Its three-step journey is Configure certificate → Review exact plan → Verify result. The review calls POST /api/v1/secrets/pki/preview, which uses the same profile validator as execution but does not sign, append events, write audit, create an idempotency row, enqueue work, or contact anything. It explains, in plain language:

• whether the secrets API, issuing CA, selected custody input, revocation tracking, and—only for the deprecated mode—durable deprecation evidence are ready;
• the exact common name and every DNS SAN, the effective lifetime after the secrets-api profile cap, public-key algorithm and size, issuing-CA and CSR SHA-256 fingerprints, and matching Vault/OpenBao path;
• exactly what issuance writes and calls, how to revoke/recover, how to verify, and the equivalent trstctl-cli secrets pki invocation.

The response includes a server-keyed request_fingerprint bound to the tenant, authenticated principal, CA, custody input, profile, and requested/effective TTL. The console supplies it as preview_fingerprint when issuing; changed input or CA state therefore fails closed with 409 and requires a fresh review. API clients may omit it for backward compatibility, but automation can use trstctl-cli secrets pki preview -f REQUEST.json before the idempotent mutation. Neither preview nor its evidence contains CSR bytes, certificate bytes, or a private key. Richer live OCSP/CRL state and revoke diagnostics remain a documented roadmap residual; the existing serial/revocation pipeline is not represented as richer UI than it is.

Secret rotation (F37)

Provider rotation runs through durable commands: the request commits the command and returns queued, non-secret evidence; a bounded worker performs every provider effect and a restart resumes from the last persisted phase. Before execution, POST /api/v1/secrets/rotations/preview returns an exact, effect-free F37 plan. It reads only tenant-scoped secret metadata and configuration: it generates no successor value, decrypts no secret, writes no event or outbox row, and contacts no connector. The plan reports current/next versions, the resolved connector destination, required permission, a tenant-bound request fingerprint, execution effects, recovery steps, and explicit blockers. The console and trstctl-cli secrets rotations preview use this plan without an Idempotency-Key; changing console inputs invalidates the review. Only a ready, still-current review reveals the separate execution action.

POST /api/v1/secrets/rotations accepts three provider modes. connector:<target> commits one application-secret event plus its sealed outbox intent before returning queued, non-secret evidence. Any other provider names a configured dynamic-secret provider (secret_integrations.dynamic_providers, type postgresql, mysql or aws-iam) with allow_static_rotation: true whose allowed_roles lists the secret name: the request commits one durable static-rotation command (an immutable secret.static_rotation.queued event, its projected row and one outbox intent) and returns queued:true with a command_id. The secret-integrations outbox worker then runs the phases, each recorded as an immutable secret.static_rotation.advanced event: stage mints a long-lived credential for role key at the provider, pinned to the command id so a retry after a crash replaces the same principal instead of leaking one; cutover publishes it as the next secret version through the same fenced secret.rotated mutation every write uses and keeps the predecessor value sealed; verify opens a session with the staged credential (PostgreSQL, MySQL; AWS IAM has no side-effect-free proof and records verify_mode: not_available); retire revokes old_ref at the provider, or skips when old_ref is none. The staged credential and the predecessor value stay sealed under the tenant key domain between phases (AN-8). A verification that still fails on its third worker attempt rolls back: the predecessor is restored as a new version and the successor revoked (rolled_back); a restore that fails ends in rollback_failed; a retirement that exhausts the outbox budget leaves the successor live as retire_pending. GET /api/v1/secrets/rotations/{id} and GET /api/v1/secrets/rotations?key=<name> (CLI secrets rotations get / list) follow a command to its terminal phase; no response carries the credential.

dynamic-lease:<provider> rotates the lease an application secret publishes, on a configured postgresql, mysql or redis provider. The secret's name is the provider role, old_ref is the lease id the secret publishes now (or none), and ttl_seconds (required, at most the provider's max_ttl) is the new lease's lifetime. It runs as the same durable command: stage issues the new lease through the durable lease lifecycle under an idempotency key derived from the command, so a crash resumes the same lease; cutover publishes its credential as the next secret version; verify opens a database session, or authenticates to Redis as the lease user and runs PING; retire revokes the old_ref lease and completes only once the provider confirms the removal (until then the outbox retries, and a removal that never completes ends as retire_pending). A failed verification restores the predecessor and revokes the new lease. The command view reports ttl_seconds and the new lease's new_expires_at, and the console's rotation form takes the lease TTL. Rotating a MongoDB, cloud IAM or Kubernetes lease is what comes next (503 today); issue, renew and revoke those leases through the lease endpoints. Native-store value rotation remains a separate event-sourced operation.

The scheduled path records connector cadences at POST /api/v1/secrets/rotation-schedules, lists them with GET /api/v1/secrets/rotation-schedules, and runs due ones with POST /api/v1/secrets/rotation-schedules/run-due. A schedule may name a connector target or a static provider that allows static rotation for the secret; dynamic-lease schedules fail closed with 503. A static due edge queues one durable command, chains its predecessor from the credential the last terminal command left live (the successor after completed, the preserved predecessor after rolled_back or failed), defers with command_in_flight while a command is still running, and fails the edge after an unresolved rollback_failed until an operator resolves that command. A schedule whose provider no longer allows static rotation, or a historical dynamic-lease schedule, is terminalized as unsupported, disabled, and makes zero provider calls. A queued connector run advances old_ref; a connector whose local version committed before terminal delivery failure also advances to that committed version so later cadences cannot stale-loop.

An operator can read one schedule at GET /api/v1/secrets/rotation-schedules/{id} and use POST .../{id}/pause, .../{id}/resume, or .../{id}/retire with a fresh Idempotency-Key and secrets:write. The console exposes these actions in the schedule table; the CLI has matching secrets rotation-schedules commands. Pause stops future due-run claims but retains the exact next due time, predecessor reference, and run history. Resume checks that the connector still exists and the predecessor matches the current native secret version, then starts from that retained due time; if it is past due, the next scheduler tick can run it. Retirement is permanent, keeps the schedule and its receipts visible, and reserves its name. The immutable secret.rotation_schedule.paused, .resumed, and .retired events each carry a matching closed action and operator-request reason code, so audit filters for one action cannot silently include the others. Their current events also retain the authenticated request binding and exact first public response, including a successful no-op. If the HTTP result is lost, retry the same key, caller, route, and body: the original response is returned without applying the old decision over a newer operator action. A changed request under that key is rejected. If a control event is durable but its SQL projection has not caught up, a different-key control or a brand-new due child fails closed. The due tick records a terminal 503 with the affected schedule ID and creates no child command or provider effect; after projection recovery, run a new tick with a fresh key. The tenant-scoped projection changes only future eligibility. A run already claimed when an operator pauses or retires can finish and record its exact successor and delivery receipt. The console shows a separate retired state and reviews this effect before retirement.

The terminal error field is a closed, status-specific evidence class, never a provider response or wrapped error string. Empty-success statuses carry no error; delivery and rollback failures use their exact fixed classes; unavailable and other terminal statuses accept only their documented finite classes. Startup also sanitizes retained schema-v1 events written by older binaries. That rewrite is fleet-gated by TRSTCTL_SECRET_ROTATION_HISTORY_FLEET_READY, signed, sequence preserving, and limited to the one JSON error token. Until it is safe to stop old writers, projection catch-up, audit/search, retention, and backup export fail closed without returning any legacy payload bytes.

The console revalidates that same closed vocabulary before rendering a successful or partial tick. An otherwise well-formed response with an unknown run, rollback, deferred, or system error is treated as a malformed scheduler failure; neither its error nor its problem detail is displayed. This keeps a stale cache or damaged proxy from turning rejected provider text back into operator-visible copy.

The manual connector form is review-first. A blocked preview explains why a provider or reference cannot run; a ready preview states what the mutation will write, which external connector it will contact, how secret material is handled, and how to retry or recover. Preview and execute remain separate server calls so viewing a plan can never become an accidental rotation.

Each run-due idempotency key owns one durable PostgreSQL tick. The tick freezes due_through from the outer idempotency row's database created_at, the UUID-ring cursor where scanning began, the current row snapshot, the ordered non-secret receipt, and the exact 50-run/500-scan budgets. One tenant has one live tick owner: every progress or terminal compare-and-swap checks its fresh lease token, generation, and PostgreSQL-clock lease. A simultaneous same-key caller receives a non-cached 503 in-progress error; a different live key receives a busy 503. After expiry, the same key resumes the same cutoff, cursor, receipt, and budgets. A different key first freezes the abandoned tick as an exact indeterminate 503, without moving its cursor, then begins from that cursor; replaying the abandoned key returns those retained bytes.

Before a scheduled connector effect can start, the tick saves the row it is about to run and PostgreSQL records one immutable child command for the exact (tenant, schedule, due_at) edge. Run, command, and terminal-event IDs are deterministic, while each live claimant receives a fresh lease token. Overlap, runner replacement, restart, and projection rebuild therefore resume or reconcile the same command instead of minting another secret version. The version-3 terminal event carries the exact due time, provider, key, old reference, command key, and request binding. Cadence replay advances from due_at, not a producer's wall clock; an unmatched schedule compare-and-swap rolls back terminalization instead of manufacturing a completed command. Tick progress, receipt, cursor movement, and an optional deferred child-lease release commit in one transaction, so a crash exposes either all of a row's progress or none of it.

There are exactly four nonterminal deferral reasons: approval_pending, command_in_flight, command_claimed, and config_revision_unanchored. The last one means a schedule created before immutable configuration evidence existed must be saved again; that scan creates no child command and calls no provider. Every deferral keeps the same due edge and is returned with schedule ID, reason, due time, and optional error; later due rows are still scanned. The stable UUID ring wraps at most once per tick. One tick executes at most 50 rotations and scans at most 500 rows. Landing exactly on either bound is reported conservatively as run_limit_reached or scan_limit_reached, with complete:false, because only an under-budget empty or short ring proves that no due row remains. The console names the consumed bound and tells the operator to run the next tick from the durable cursor.

If shared store, event, custody, or integrity state fails after earlier rows finish, the endpoint returns a truthful 503 envelope with complete:false, partial:true, the failed schedule ID, and the system error. The outer idempotency record caches that whole envelope: replaying the same key is byte-identical and executes no child command; a new key reconciles any retained terminal event and continues later rows. A tick's composite foreign key keeps its bound outer idempotency row out of normal garbage collection. The indeterminate takeover receipt stays until its original key returns and completes the outer row; deleting that completed outer row then cascades to the tick. Terminal child-command receivers are garbage collected only after their exact immutable version-3 terminal event is still retained, the schedule has advanced past that due edge, and a newer command exists. The newest command remains as a finite one-row-per-schedule lineage fence; claimed or ambiguous commands are never purged.

Scheduled runs execute two provider classes:

• connector:<target> atomically commits a native-store version and one sealed secret-sync outbox command. The response sets queued:true and completed:false; only the bounded outbox worker calls the connector. Delivery failure stays on the durable job for retry and does not perform a compensating direct write.
• A static provider queues one durable static-rotation command per due edge, as described above.

Scheduled dynamic-lease:<provider> rotation is what comes next: creating such a schedule returns 503, and a historical one is terminalized as unsupported. Run a dynamic-lease rotation on request through POST /api/v1/secrets/rotations.

The rotation package's library rotators (stage/cutover/verify/retire-and-rollback for PostgreSQL, MySQL and AWS IAM) are not what the served commands run: those run the durable phases above and use the package's PostgreSQL and MySQL credential checks.

Ephemeral API keys (F38)

For high-churn automation, trstctl issues short-lived API keys through the served control plane:

• POST /api/v1/ephemeral/api-keys/preview validates and normalizes the exact tenant/caller/subject/scopes/TTL plan without creating a bearer, event, idempotency row, projection, or external effect. It returns a server-keyed request_fingerprint and complete recovery/verification instructions.
• POST /api/v1/ephemeral/api-keys mints a tenant API token with subject, scopes, ttl_seconds, and an optional reviewed preview_fingerprint. A stale fingerprint returns 409 before issuance.
• trstctl-cli ephemeral api-keys preview -f body.json and ephemeral api-keys issue -f reviewed.json drive the same two-step contract.
• Both review and issue require access:write; both also attenuate authority by refusing any requested scope the caller does not already possess.
• Execution is guarded by Idempotency-Key, so a byte-identical retry with the same key returns the original response instead of minting twice. Reusing that key with changed input returns 409.
• The response returns the raw trst_... token once; the event log stores only the token hash in api_token.created — the raw token is never persisted or emitted.
• The served leaseworker sweeps expired keys and emits api_token.revoked, so the read model shows revoked_at evidence and authentication rejects the key after TTL.
• The console makes the whole lifecycle executable: configure, effect-free review, reviewed issue, metadata-ledger observation, same-key recovery after uncertainty, bearer-only verification, immediate revocation, and post-revoke cleanup. The raw bearer lives only in React memory and is removed on dismiss or revoke.

{
  "subject": "ci-preview-deploy",
  "scopes": ["access:read"],
  "ttl_seconds": 900,
  "preview_fingerprint": "sha256:<value returned by preview>"
}

Use it for short CI jobs, deploy previews, partner imports, and similar workflows needing a narrow bearer credential for minutes, not a reusable key.

Encryption-as-a-service & KMIP (F66)

Transit encrypts, decrypts, HMACs, signs, verifies, and rewraps data using tenant-scoped named keys the application never sees, mounted at /api/v1/transit/* with a one-for-one CLI: GET /api/v1/transit/keys / transit keys list reads only tenant-scoped name, purpose, and current-version metadata; GET /api/v1/transit/keys/{name}/versions / transit keys versions <name> reads the complete retained version history without key material; GET /api/v1/transit/status / transit status reads effect-free sealed-restore and KMIP runtime posture; POST /api/v1/transit/keys / transit keys create mints a key, .../keys/rotate / transit keys rotate rotates it, and the same POST /api/v1/transit/<op> / trstctl-cli transit <op> pairing covers encrypt, decrypt, rewrap, hmac, sign, and verify.

Ciphertexts are versioned (trv:<version>:...) so rotation can rewrap old data to the newest version. Requests are tenant-bound and idempotent, auth-gated by keys:write for creation/rotation/encrypt/decrypt/rewrap/HMAC/sign and keys:read for verify. Plaintext and associated data live in wipeable []byte buffers zeroized after the response is written. With the deployment KEK configured, each tenant's keyring is one sealed row in PostgreSQL that every replica shares: every create/rotate is saved before it is reported, a replica that lacks a key or version reloads the row, and a key created or rotated on one replica works on all of them and after a restart. A keyring an earlier version checkpointed to transit.keyring_dir is imported once at startup; a row or file the original KEK cannot open fails the operation or the start instead of silently replacing the keys. There is deliberately no browser upload/import path for key material. Events — transit.key.created, transit.key.rotated, transit.encrypt, transit.rewrap, transit.hmac, transit.sign — give audit evidence without logging key bytes or plaintext.

The console's Encryption and signing task at /secrets/engines reads back safe key metadata, creates AEAD/HMAC/signing keys, selects only a compatible key for each operation, rotates the selected key of each type, and operates encrypt, decrypt, rewrap, HMAC, sign, and verify. It also shows the full retained version history, Transit-only immutable audit receipts, automatic sealed-restore state, an explicit proof-first failed-operation recovery journey, and the real tenant-bound KMIP listener/profile status. Transit is a separate encryption service, so those controls remain available when the optional native secret store is disabled. No key bytes enter the browser. The raw plaintext returned by decrypt is shown in a reveal-once panel and removed from the page when dismissed.

cat > transit-key.json <<'JSON'
{"name":"payments","kind":"aead"}
JSON
trstctl-cli --idempotency-key transit-payments-create transit keys create -f transit-key.json
trstctl-cli transit keys list
trstctl-cli transit keys versions payments
trstctl-cli transit status

cat > transit-encrypt.json <<'JSON'
{"key":"payments","plaintext":"Y2FyZC10b2tlbi0xMjM=","aad":"dGVuYW50PXBheW1lbnRz"}
JSON
trstctl-cli --idempotency-key transit-payments-encrypt transit encrypt -f transit-encrypt.json

If a create, rotate, encrypt, rewrap, HMAC, or sign response is uncertain, do not invent a replacement key or changed request. Keep the original inputs and Idempotency-Key, read transit status, the key's version history, and the filtered transit.* audit receipts, then replay the byte-identical request with the same key. For a restore failure, repair the original KEK, tenant key domain or database and restart; never delete or overwrite the sealed keyring row or file just to make the process start.

For legacy gear, the binary can also mount an opt-in KMIP listener:

• TRSTCTL_PROTOCOLS_KMIP_ENABLED=true
• TRSTCTL_PROTOCOLS_KMIP_TENANT_ID=<tenant-uuid>
• TRSTCTL_PROTOCOLS_KMIP_ADDR=:5696
• TRSTCTL_PROTOCOLS_KMIP_CERT_FILE=/path/server.crt
• TRSTCTL_PROTOCOLS_KMIP_KEY_FILE=/path/server.key
• TRSTCTL_PROTOCOLS_KMIP_CLIENT_CA_FILE=/path/client-ca.crt
• TRSTCTL_PROTOCOLS_KMIP_CLIENT_REGISTRATIONS=true (optional: every tenant's own clients)

The listener is raw KMIP over mutual TLS 1.3; the TLS layer verifies the client cert chain before the KMIP handler sees a frame. The service stores objects under the configured tenant, emits immutable kmip.object.created, kmip.object.revoke, and kmip.object.destroyed audit events, and zeroizes in-memory key material on destroy, rekey, and shutdown. The served OASIS 1.4 profile is stock-client-tested: Query, DiscoverVersions, Create/Register an AES-256 SymmetricKey, Get it plain or AES-GCM-wrapped (and register the wrapped value back), Locate, Revoke, and Destroy over TTLV. Unsupported operations get a KMIP failure response, not an unframed TCP close.

The listener also serves key pairs. Create Key Pair generates an RSA (2048, 3072 or 4096 bits) or ECDSA (P-256 or P-384) pair inside the server; each half links to the other and starts Pre-Active, so it neither signs nor verifies until Activate. Sign runs with the private key (RSA PKCS#1 v1.5 or PSS, or ECDSA, over SHA-256, -384 or -512, single-part Data) and Signature Verify with the public key, which Get returns as an X.509 SubjectPublicKeyInfo. The private key is sealed like every key the server holds and Get refuses it (Not Extractable) unless the client set Extractable when it created the pair, so by default it never leaves the server; a key a client makes unextractable stays unextractable. Get Attributes, Get Attribute List, Add, Modify and Delete Attribute serve names, contact information and x- custom attributes with text values; the server's own attributes (state, dates, links, usage mask) are read-only. A revocation for key or CA compromise leaves the object Compromised, and it signs and verifies nothing after that. Locate narrows by object type, algorithm and name. A stock client drives all of this in an opt-in proof that builds PyKMIP 0.10: TRSTCTL_PYKMIP_PROOF=1 go test ./internal/kmip -run PyKMIP -v.

With TRSTCTL_PROTOCOLS_KMIP_CLIENT_REGISTRATIONS=true one listener serves every tenant's own clients. A tenant registers a client on the Transit page, with trstctl-cli kmip clients register, or at POST /api/v1/kmip/clients (keys:write), and gets a one-time enrollment: a username (the registration's ID) and a password shown once, valid for 24 hours, of which the server keeps only the SHA-256. Configure the client with the listener's server CA, its own TLS certificate and that username and password; its first KMIP request binds the certificate it presents to the tenant, and from then on the certificate alone selects the tenant. The handshake proves the client holds the certificate's key and the secret proves the tenant issued the enrollment, so a tenant cannot claim another tenant's client by registering a certificate it has only seen, and a certificate binds to one registration at a time. Each tenant sees only its own objects. When the client moves to a new certificate, issue a new enrollment (trstctl-cli kmip clients enroll <id>); the old certificate keeps working until the new one enrolls. Removing a registration (trstctl-cli kmip clients remove <id>) refuses its certificate from its next request. TRSTCTL_PROTOCOLS_KMIP_TENANT_ID and TRSTCTL_PROTOCOLS_KMIP_CLIENT_CA_FILE become optional; set together, they keep serving the clients the deployment's CA issued for that tenant. The Transit status endpoint and console report not_configured, starting, or listening from the assembled tenant binding and live listener state. That status is not a conformance claim: use a stock KMIP client to prove the complete mTLS wire path.

Secret sync (F68)

trstctl pushes secrets outward via the durable outbox (journaled first, at-least-once, no half-writes). Operators first call POST /api/v1/secrets/syncs/preview. Preview checks only the tenant-scoped secret metadata and configured target catalog. It does not open the secret value, emit an event, write an outbox row, or contact the target. For a ready plan it returns the exact secret version, destination, effects, recovery steps, verification steps, and a caller-bound request fingerprint.

POST /api/v1/secrets/syncs reads the stored secret only after the operator submits that reviewed fingerprint, writes a sealed outbox row in the same tenant-scoped transaction, delivers through the configured pusher, and returns metadata only (job_id, name, target, remote_key, enqueued/delivered flags). GET /api/v1/secrets/syncs/{id} reads the exact job's tenant-scoped pending, delivered, or failed projection; delivered records the worker's target receipt, so independently read the target to verify its value. GET /api/v1/secrets/syncs/targets lists the catalog and marks which targets are configured. Shipped concrete pushers: AWS Secrets Manager, AWS Systems Manager Parameter Store, GCP Secret Manager, Azure Key Vault, GitHub Actions, GitLab CI, Vercel, Netlify, Kubernetes, Terraform Cloud/OpenTofu, HashiCorp Vault/OpenBao KV v2, a signed webhook (templated JSON with an HMAC-SHA256 signature over a timestamp and the body; see configuration), and a generic CI/JSON endpoint with no payload signature. The console's Check a previous delivery field accepts a job ID from any receipt or CLI run. It reads the same durable status and keeps the ID in the page URL, including when the target is no longer configured. A pending status does not prove the external value changed; read the destination independently.

Every redelivery reuses the same sync-operation ID — AWS Secrets Manager sends it as both ClientRequestToken and Idempotency-Key; GCP Secret Manager and Azure Key Vault compare the current version before creating another, forwarding the ID too — so a crash between commit and acknowledgment reconciles to the existing value instead of duplicating, rejecting any changed replay outright.

Commands are FIFO across the whole tenant+target, not merely one spelling of a remote key: receivers commonly normalize /TOKEN/ and TOKEN to the same object. Each new command copies its positive immutable event sequence into the projected job and outbox row; a retry/backoff therefore blocks later same-target commands while a different target can progress. Terminal delivery evidence is itself a deterministic event receipt, and startup compares retained history with SQL before workers run. Immediately before receiver I/O, the worker records a monotonic command-global start token under the same tenant+job terminal-choice lock. A failed event is safe only when durable authority proves no generation could have changed the receiver; configured HTTP targets now place that token after a successful TCP dial but before any request bytes. A refused dial therefore leaves zero possible receiver starts, so a later successful retry can release the next command in target order. A TLS failure after connect, a request timeout, or a lost response still counts as a possible effect; the worker cannot infer that the receiver made no change. An ordinary timeout or expired worker remains effect_possible, keeps its FIFO barrier, and is retried with the same receiver idempotency key until delivery is proved. The one exception is a typed local no-network result owned by the first and still-only start token. Its closed error and event attempt count are frozen before append so crash recovery reproduces the same canonical evidence bytes.

A lost result is proved by reading the receiver back, never by sending the value again. Targets whose receiver exposes its state (Vault KV v2, Terraform Cloud and OpenTofu through the operation marker, GCP Secret Manager, Azure Key Vault and Kubernetes; the catalog lists receiver-reconcile) are read back under the same operation ID right after an ambiguous error, again when the retry budget is spent, and once more after any delivery that needed more than one receiver start. A match delivers, or settles the delivered command with one retained secret.sync.receiver_settled receipt that releases the lane; a provable absence fails the command with reconciled authority so the next version proceeds. Targets with no readable state (GitHub, GitLab, Vercel, Netlify, the signed webhook and the generic CI JSON receiver), and Parameter Store, whose readback is what comes next, keep the lane closed after a multi-start delivery until an operator reconciles the job: POST /api/v1/secrets/syncs/{id}/reconcile, trstctl-cli secrets syncs reconcile, or the Secrets console, with operator_verified and what they saw at the destination (value_present or value_absent). The attestation is recorded with the operator's name; a blind call on such a target is refused with 409 unknown, and GET /api/v1/secrets/syncs/{id} names the blocking predecessor (blocked_by_job_id), the receiver authority (receiver_effect_state, receiver_io_starts) and whether the target is reconcilable. The command ID also includes the tenant's secret lifecycle epoch, so offboarding and re-registering the same tenant UUID cannot reactivate old ciphertext or outcomes.

Worker capacity and fairness use the tenant plus that effective target lane. Two tenants may both call a target ci, but one tenant's slow ci receiver does not consume the other tenant's ci lane or circuit allowance. Within either tenant, the whole target remains serial and FIFO, including remote-key aliases. The bounded secret-sync family worker pool and queue still cap aggregate work across tenants and reject promptly when that global provider-plane budget is full.

Targets are configured under secret_integrations.sync_targets, one tenant/credential reference each, resolved for a single outbox attempt before the locked buffer is destroyed. GitHub values use its X25519/XSalsa20-Poly1305 sealed-box format — sealed locally with the repo's public key, decryptable only by the matching private key's holder, never sent as plaintext. Sync endpoints follow the dynamic-provider transport rule: HTTPS by default, allow_private_endpoint is only an address grant, plaintext needs the explicit loopback-only switch.

AWS, GCP, and Azure workload identity for secret sync

Each cloud is configured independently on its matching sync target: aws_workload_identity, gcp_workload_identity, or azure_workload_identity. Every switch is false by default. Enabling one forbids that target's static credential fields, so a missing workload-identity source fails closed instead of falling back to a long-lived token or access key. The three provider exchanges are thin, hand-written REST encoders over one shared internal/cloudauth cache and refresh-before-expiry seam; there is no cloud vendor SDK or parallel authentication stack.

AWS Secrets Manager targets can replace long-lived access keys with an explicitly configured workload identity. Set aws_workload_identity: true on that target and leave access_key_id, secret_access_key_ref, and session_token_ref empty. workload_identity_endpoint is optional and defaults to the AWS STS endpoint; it is primarily useful for an approved private endpoint or test substrate. The default is still off, so existing static-credential targets do not change and a fresh install makes no token-exchange egress.

An operator then creates a tenant source at /api/v1/secrets/syncs/workload-identity-sources (or with trstctl-cli secrets syncs workload-identities create --body-file ...). The source binds exactly one configured AWS target to an IAM role ARN, expected OIDC issuer trust source, audience, subject, allowed remote-key prefixes, and a file: or secret:// proof reference. The API stores only this configuration and honest ready, active, disabled, offline_disabled, or exchange_failed status; it never accepts or stores an inline proof or AWS token. The console under Secrets serves create, list, edit, delete, expiry, and failure-state workflows. The target ID is an operator-chosen name, not a provider type. The target catalog reports its configured protocol separately and offers it for this workflow only when that target's workload-identity switch is enabled. Source creation refuses an unconfigured target, a static-credential target, or a target for another cloud. The cloud-manager coverage and unvaulted-secret posture also count configured targets by their provider type. A custom target ID appears as the actual sync target in the coverage response; the built-in ID remains the example when that provider has no configured target.

Only the bounded secret-sync outbox worker resolves the proof, validates its signature and exact issuer/audience/subject/expiry against the tenant's JWT/JWKS trust source, and performs AssumeRoleWithWebIdentity. The shared internal/cloudauth minter caches the short-lived result until its refresh window, keeps secret bytes locked and wipeable, and passes them into the existing hand-written AWS SigV4 pusher. No vendor SDK or parallel AWS integration is involved. When air-gap policy is enabled, the worker records offline_disabled before opening a connection, marks that delivery failed once with a stable reason, and does not retry forever.

GCP Secret Manager targets use the same boundary with gcp_workload_identity: true and no token_ref. workload_identity_endpoint defaults to the RFC 8693 endpoint at sts.googleapis.com. A GCP source leaves role_arn empty and may set service_account: when it is blank, the worker passes the short-lived STS bearer to the existing GCP pusher; when it is present, the worker performs the optional IAM Credentials generateAccessToken step at the configured workload_identity_impersonation_endpoint. Both paths resolve and validate the OIDC proof, exchange it, cache it, and refresh it through the same internal/cloudauth minter used by AWS. The API, CLI, generated clients, and console expose the same tenant-scoped source and honest runtime status.

Azure Key Vault targets use that same boundary with azure_workload_identity: true and no token_ref. workload_identity_endpoint is optional; when absent, the source's azure_tenant_id selects https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token. The Azure source sets provider: "azure", the Entra application client_id, and a Key Vault target_scope, normally https://vault.azure.net/.default, in addition to the same trust source, exact audience/subject, target, allowed remote-key prefixes, and proof reference used by the other clouds.

After the outbox worker validates the referenced OIDC proof, the thin Entra exchange sends grant_type=client_credentials, the application client ID, target scope, and that existing proof unchanged in the JWT-bearer client_assertion field. This is the Entra federated-credential flow: trstctl does not mint a second JWT, sign an assertion with a certificate, or hold an Entra certificate/private key. The returned short-lived bearer goes through the existing hand-written internal/secretsync Azure Key Vault pusher. It replaces this sync target's static token_ref; the similarly named TRSTCTL_MANAGED_KEYS_AZURE_BEARER_TOKEN(_FILE) settings and internal/kms/azurekv are a separate managed-key child-signer path.

Like the other two providers, Azure exchanges only during a bounded secret-sync outbox delivery. The request handler merely journals the work. Air-gapped mode records offline_disabled before any Entra or Key Vault network request, fails the queued delivery once with a stable reason, and does not retry forever. The served Azure proofs also verify tenant isolation and that neither the OIDC proof nor minted bearer reaches the API response, event log, outbox payload, job error, or source status. See Secrets configuration for the complete target and source fields.

GET /api/v1/secrets/cloud-secret-managers / trstctl-cli secrets cloud-secret-managers: read-only cloud_secret discovery for AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and HashiCorp Vault KV, plus sealed-outbox sync coverage for AWS/GCP/Azure — counts, operations, evidence, residuals only, never values, credentials, or tokens.

GET /api/v1/secrets/kubernetes-operator / trstctl-cli secrets kubernetes-operator: the TrstctlSecretSync CRD declares the target Kubernetes Secret, the trstctl references to resolve, and which Deployment/StatefulSet/DaemonSet workloads reload via a pod-template hash annotation; the operator writes Secret.data and records status.phase, status.contentHash, and status.reloadedWorkloads — metadata only.

GET /api/v1/secrets/workload-injection / trstctl-cli secrets workload-injection: the TrstctlSecretInjection CRD consumes a namespace-local Kubernetes Secret and patches Deployment/StatefulSet/DaemonSet pod templates with a memory-backed shared volume, app-container mounts, optional valueFrom.secretKeyRef env entries, and the trstctl-agent --secret-inject sidecar; the operator checks source Secret key names before patching, reads its metadata/content hash, and discards the Kubernetes response bytes. The sidecar copies volume files as byte slices and wipes buffers. A missing requested key sets the CR to Error before a workload rollout. Ready reports a successful template patch; check workload pod readiness separately for sidecar runtime failures.

GET /api/v1/secrets/unvaulted / trstctl-cli secrets unvaulted: repository and third-party scanning sources, redacted leaked_secret finding counts, cloud-secret discovery across AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and HashiCorp Vault KV, and configured AWS/GCP/Azure sync targets — read-only and metadata-only.

The auth-method framework (F58)

Before reading a secret, a workload must authenticate to trstctl via the auth-method framework: it presents a credential (a token, an OIDC JWT, a Kubernetes SA token, cloud IAM, etc.), trstctl verifies it through the single isolated cryptography path (timing-safe), and issues a scoped, time-bounded session bearer. Credential bytes are never logged (wipeable memory, never a copyable string). The raw bearer is shown once; only its one-way hash enters the event-sourced API-token projection. Successful sessions and operator revocations become tenant-scoped evidence in the durable session ledger, and revocation atomically disables the linked bearer. Authentication failures return only a generic denial.

POST /api/v1/secrets/login serves seven machine methods — token, kubernetes, aws-iam, gcp, azure, oidc, jwt. JWT-family methods verify against an operator-supplied JWKS, check issuer/audience/expiry, and bind a tenant claim or pin the method to one tenant. Each configured method must grant non-empty static scopes, or OIDC/generic JWT may name one signed scopes_claim; setting both or neither fails startup. A token's ordinary scope/scopes claim never overrides static operator policy. AWS IAM uses the Vault-style signed sts:GetCallerIdentity request, verifying the caller via STS without ever receiving the AWS secret access key.

secrets:
  enable_api: true
  # Optional builtin HMAC token method. Its deployment-wide verifier is usable
  # only for this tenant and grants exactly these scopes.
  auth_secret_file: /etc/trstctl/machine-auth.bin
  auth_token_tenant_id: 11111111-1111-1111-1111-111111111111
  auth_token_scopes: ["secrets:read"]
  machine_auth:
    - name: kubernetes
      tenant_claim: trstctl.io/tenant
      issuer: https://kubernetes.default.svc
      audience: trstctl
      jwks_file: /etc/trstctl/k8s-sa-jwks.json
      allowed_namespaces: ["payments"]
      allowed_service_accounts: ["payments/api"]
      scopes: ["secrets:read"]
    - name: aws-iam
      tenant_id: 11111111-1111-1111-1111-111111111111
      allowed_accounts: ["123456789012"]
      scopes: ["secrets:read"]

The /secrets/access console turns that backend framework into one complete, review-first journey:

1. Choose method. Select one method from the server's secret-free method inventory. The console shows its real type, source, issuer/audience rules, tenant restrictions, and enabled/disabled overlay. Method definitions remain deployment-owned configuration; the browser cannot invent or silently edit an authentication authority.

2. Review login. POST /api/v1/secrets/login/preview receives only the method name. It reads configuration and returns the exact tenant binding, credential shape, session lifetime, prerequisites, blockers, writes, outside calls, recovery steps, verification steps, and CLI command. It does not receive a credential, call the verifier, create a session/event/idempotency row, or contact AWS STS.

3. Test reviewed login. Only a ready plan reveals the password-style credential field. POST /api/v1/secrets/login consumes the credential once and requires an Idempotency-Key; the request is bound with a server-keyed digest of the tenant, method, credential, and review fingerprint. An exact retry returns the original response—including the same one-time bearer—instead of issuing a second session. Reusing the key for different input returns 409. The presented credential and raw session bearer never enter an event or read model.

4. Verify and recover. The result shows session id, principal, method, scopes, expiry, and a one-time reveal panel for the trst_… bearer. Copy it directly into the workload, then dismiss it; trstctl cannot recover it from the stored hash. The page refreshes Issued sessions so the durable row can be checked and revoked. Revocation immediately makes the linked bearer fail authentication. The browser clears the presented credential after both success and failure. A normal authentication failure keeps the reviewed plan for a corrected retry; a 409 configuration/fingerprint change discards it and requires a fresh review. A disabled method shows blockers and never accepts a credential.

   Disabling a method commits a tenant-and-method denial before publishing its event, so a projection delay cannot admit a new login. A login already verifying a credential must pass the same final admission lock before its session event is appended; disable and that final check have one clear order across replicas. Pending enable stays denied until its event projects. The exact pending command and denial state survive backup and restore; retry the same operation after recovery, then confirm the method status in the console.

   For a machine session with an issued bearer, revocation commits a tenant- and session-bound denial before publishing its revocation event. If event delivery or projection is interrupted, the bearer remains refused while the exact command is recovered. Retry with the same idempotency key and read the issued session ledger to confirm completion. Older ledger-only sessions have no linked bearer; they remain revocable as historical records.

Headless users get the same contract. Keep credential JSON in a permission-restricted file or provide it through a protected pipe; do not put credentials in shell history:

trstctl-cli secrets login preview -f machine-login.json
trstctl-cli --idempotency-key workload-login-20260902 secrets login -f machine-login.json
trstctl-cli secrets sessions list
trstctl-cli --idempotency-key revoke-synthetic-session-1 secrets sessions revoke SESSION_ID

Preview requires secrets:read because method configuration is operator metadata. The login exchange itself is intentionally public: the presented machine credential authenticates the workload, while X-Tenant-ID remains only a lookup hint and never grants access. The session list/revoke and method enable/disable operations remain RBAC-protected. This preserves CA/provider agnosticism: the framework validates the credential against the configured authority but does not require trstctl to have issued it.

Secret scanning bridge (F39) and sharing & approvals (F60)

Decoy API credentials

The console's Secrets → Automatic secret sources → Plant a decoy credential workflow, trstctl-cli secrets honeytokens, and /api/v1/secrets/honeytokens serve the same tenant-scoped decoy lifecycle. A caller with secrets:write plants a name and intended placement. The create response reveals one ordinary-looking trst_ bearer; an identical idempotency replay to the same authenticated caller returns that original response. Subsequent list/get responses contain only metadata. Put that value in a controlled location where a real trstctl API bearer would not belong. Do not put a production credential there. The decoy is stored in honey_tokens, never api_tokens, and has no permission or authentication path. Even a public API route refuses it.

The first bearer use appends honeytoken.triggered, changes the decoy to triggered, and queues one notification.honeytoken critical alert in the same tenant transaction. The event and detail view retain the matched route pattern and method, not the raw URL, query, or bearer. Configure an operator notification channel and routing policy before planting so the alert has a receiver; the notification inbox and delivery receipts show retry or failure. Refresh the decoy list to investigate, then retire it. A retired value remains unusable. To replace exposed bait, retire the old record and plant a new one.

AWS IAM decoy keys

Secrets → Automatic secret sources → Plant a decoy credential also has an AWS IAM decoy workflow. An operator first attaches one AWS account to one tenant with secret_integrations.honey_aws_accounts. The browser lists only the account ID, configured Regions, polling interval, and key lifetime limit. IAM writer and CloudTrail reader authority are separate file: or tenant secret:// references; neither credential is sent to the browser. Each referenced file must be mode 0600 and contain JSON fields access_key_id, secret_access_key, session_token, and expires_at. These are short-lived operator caller sessions, refreshed in place before expiry. The bait key itself is an IAM access key and remains stable until its lease expires or the operator retires it; an STS session would expire before it could serve as durable bait.

Example attachment (substitute owned values and private file paths):

{
  "secret_integrations": {
    "honey_aws_accounts": [{
      "tenant_id": "11111111-1111-1111-1111-111111111111",
      "id": "security-decoys",
      "account_id": "123456789012",
      "regions": ["us-east-1"],
      "iam_credentials_ref": "file:/var/lib/trstctl/private/iam-decoy-writer.json",
      "cloudtrail_credentials_ref": "file:/var/lib/trstctl/private/cloudtrail-decoy-reader.json",
      "max_ttl": "720h",
      "poll_interval": "5m"
    }]
  }
}

The IAM writer needs narrowly scoped create/get/tag/policy/key/delete permissions under /trstctl/honeytokens/; the reader needs cloudtrail:LookupEvents in each listed Region. Grant those through independently assumed roles. trstctl checks the account, IAM path, and ownership tag; installs an explicit deny-all inline user policy; reads that policy back; then creates the bait access key. The create response reveals the secret access key once, with an idempotent same-caller recovery path. The normal dynamic-secret issue endpoint cannot issue these keys. For an owned local IAM/CloudTrail fixture, set explicit loopback endpoints and allow_private_endpoint, allow_insecure_loopback, and a narrow private_egress_cidrs entry. Private overrides are operator-only; tenant webhooks retain their public HTTPS rule.

trstctl-cli secrets honeytokens aws accounts lists attachments. Run trstctl-cli secrets honeytokens aws preview -f request.json to review the exact account, Regions, TTL, IAM sequence, detection limit, recovery, and verification without an IAM or CloudTrail call. It validates local account configuration; remote IAM permission is checked only by the creation attempt. Plant with trstctl-cli secrets honeytokens aws plant -f request.json where the JSON has account_id (the configured attachment ID), name, placement, and ttl_seconds. Supplying the returned preview_fingerprint binds the plant request to the reviewed caller, account attachment, and fields; a changed request is rejected before IAM effects. The console requires this review step. list, get <id>, rearm <id>, and retire <id> match the console and the /api/v1/secrets/honeytokens/aws API. Retirement queues IAM key/user removal; lease.revocation_status=completed means a subsequent IAM GetUser confirmed the user absent. A queued retiring state does not claim deletion. Monitoring continues for 72 hours after confirmed removal to catch delayed CloudTrail delivery.

The monitor calls CloudTrail LookupEvents through a durable outbox. Each page becomes an immutable event before the next page is queued; account/Region calls share a PostgreSQL rate slot across replicas. Duplicate event IDs yield one incident. The first observed use enters the existing critical honeytoken alert route; the investigation view shows Region watermarks, gaps, AWS event IDs, operation names, source addresses, and detection times. Coverage is limited to management events in the configured account and Regions. CloudTrail delivery can lag; after triage, rearm makes the same live bait alert on its next distinct observed use while preserving earlier incident records. An already seen CloudTrail event does not trigger another alert. Data events, other accounts/Regions, and gaps older than CloudTrail's 90-day lookup window are not covered. A failed/dead-lettered scan is a monitoring outage, not evidence that the bait was untouched. No real AWS account was available for the local QA campaign, so the AWS adapter has local contract evidence only.

The scanning bridge runs the pinned Gitleaks scanner from the served control plane, recording redacted findings into discovery, the credential graph, and the risk view. TRSTCTL_SECRETS_GITLEAKS_BIN points at the Gitleaks v8.27.2 binary from tools/gitleaks/install.sh, which checksums the release tarball before installing. secrets.scan_roots (or TRSTCTL_SECRETS_SCAN_ROOTS, a comma-separated list) is the closed set of mounted filesystem trees the served scanner may read; an empty setting confines scans to the control-plane working directory. This prevents a caller with secrets:write from turning the scanner into a general host-file reader. POST /api/v1/secrets/scans/preview first validates the exact confined path, scan mode, installed scanner, and additive rules without starting Git or Gitleaks, creating a file, appending an event, or recording idempotency. Its keyed fingerprint is bound to the tenant, caller, normalized target, mode, rules path, and capabilities. POST /api/v1/secrets/scans may require that fingerprint and fails with 409 before the scanner starts if the reviewed command changed. Execution scans a repo or workspace with the pinned 213-rule default set (above the 140-rule floor); the response and stored finding carry only rule id, file, line, scanner version, and fingerprint — Gitleaks redacts the value, never reaching the API, event log, graph, or audit output.

cat > secret-scan.json <<'JSON'
{"path":".","mode":"git_history","custom_rules_path":"./gitleaks-custom-rules.toml"}
JSON
trstctl-cli secrets scans preview -f secret-scan.json
# Copy request_fingerprint from the review into preview_fingerprint, then run.
trstctl-cli --idempotency-key ci-secret-scan-1 secrets scans run -f reviewed-secret-scan.json

A bare {"path":"."} scans the working tree by default; mode/custom_rules_path above switch on full Git-history scanning (--log-opts --all) plus an additive [[rules]] TOML fragment wrapped with [extend] useDefault = true — never an allowlist or disabled-rule override. The response's mode, custom_rules, and capabilities fields prove full-history, entropy, pattern, 100+ default-rule, and custom-rule coverage without storing a secret value; run_id can be replayed against GET /api/v1/discovery/findings?run_id=... or the graph view. TruffleHog JSON ingestion still exists for offline import/contract tests, but Gitleaks is the served engine. If a scanner or network failure interrupts execution, retry the identical reviewed body with the same Idempotency-Key: a pre-recording failure can try again, while a completed request returns its original run instead of scanning twice. The console preserves that recovery key only in memory, keeps the original failure visible, and labels the action Retry reviewed scan.

Locally, the same runner works without a server: secrets scans staged-diff and secrets scans pre-commit install scan only staged Git blobs, or an explicit base/head CI diff, into a temporary tree, dropping raw values from stdout, stderr, and JSON output. Findings block commits and pipeline steps by default; --advisory keeps the report but exits zero for non-blocking rollout:

trstctl-cli secrets scans staged-diff --repo . --base origin/main --head HEAD --advisory

For realtime repo ingress, GET /api/v1/secrets/scans/repositories reports provider posture for GitHub, GitLab, and Bitbucket; POST /api/v1/secrets/scans/repositories/{provider}/webhook takes a normalized event — repository, checkout_path, ref, commit_sha, event, credential_ref — and upserts a tenant-scoped secret_repo discovery source plus a discovery.run outbox row. The outbox worker scans checkout_path directly, or clones a public/local clone_url; credential-bearing URLs are rejected, and private credentials must stay secret references, never payload values. Native GitHub/GitLab/Bitbucket signature verification and private credential_ref clone resolution remain shortfalls, not served.

Third-party artifacts use the same shape: GET /api/v1/secrets/scans/third-party and POST /api/v1/secrets/scans/third-party/{provider}/ingest cover cicd_log, container_registry, slack, and jira — a source plus operator-owned artifact_path queues a secret_third_party discovery run against the same Gitleaks runner. Raw CI logs, registry exports, chat transcripts, and issue exports stay outside trstctl storage; events record only provider, artifact kind, rule id, file, line, scanner version, and credential-ref. When the provider signs its export, add signature_path (a detached RSA or ECDSA signature over the artifact bytes, raw or base64) and signer_public_key_pem: trstctl verifies the signature before the scanner opens the artifact, fails the run without scanning when it does not verify, and stamps every finding with signature_verified and the signer key's SHA-256. Set require_signature on sources that must never scan an unsigned export; the ingest is refused with 400 when the signature inputs are missing. Add delete_artifact_after_scan when the export must not outlive its scan: trstctl removes that one file once the run is terminal and the run message records the removal. The redacted findings themselves age out through the privacy retention worker after TRSTCTL_PRIVACY_RETENTION_THIRD_PARTY_SCANS (default 90 days); the discovery run record stays as evidence that the scan happened. Native Slack/Jira/container-registry API polling and provider-native annotations remain shortfalls, not served.

cat > third-party-scan.json <<'JSON'
{"source":"acme/slack","artifact_path":"/var/lib/trstctl/exports/slack.jsonl","event":"message_export"}
JSON
trstctl-cli --idempotency-key third-party-scan-1 \
  secrets scans third-party ingest slack -f third-party-scan.json

In the console, open Find leaked secrets in code, expand Scanner setup and exact evidence, and queue the same provider, source reference, artifact path, and optional event. The receipt includes a discovery run ID. The page opens that run automatically; after a restart, paste its ID into Third-party scan result or open /secrets/scanning?third_party_run=<run-id>. Refresh until the outbox run is terminal. The page shows the run status and the first 50 redacted findings, then links to Discovery filtered to that exact run for the full finding list and triage. Neither the artifact contents nor scanner matches appear in the console. The artifact path must be readable by the running control plane, inside an operator-configured scan root, and the pinned Gitleaks binary must be installed.

Secret sharing is a review-first, one-time delivery journey. POST /api/v1/secrets/shares/preview accepts only ttl_seconds: the value never enters the preview request. It returns the normalized lifetime, permission, exact execution writes, recovery and verification steps, empty preview/external-effect lists, and a tenant-and-principal-bound fingerprint. The preview creates no share, event, audit row, idempotency record, signer request, outbox work, or external call. Zero selects the 24-hour default; explicit lifetimes must be between 60 seconds and 7 days.

POST /api/v1/secrets/shares accepts the value only at execution and may bind it to the reviewed lifetime with preview_fingerprint. A changed lifetime then fails with 409 before any state change. PostgreSQL stores only SHA-256(token) plus the envelope-encrypted value in secret_shares; the raw bearer token is returned once. A valid share survives a restart, while a stolen database or backup holds neither the token nor plaintext. If the response is interrupted, retry the identical request with the same Idempotency-Key: trstctl returns the exact original token instead of creating a duplicate. Reusing that key for changed input fails closed.

POST /api/v1/secrets/shares/redeem requires the share token plus an authenticated principal with secrets:read in the same tenant. It atomically deletes the row and returns the value for one redemption; another new request, expired token, or wrong tenant gets a normal 404. A retry of the identical authenticated request with its original Idempotency-Key recovers the original response. The console retains that key only in memory after an ambiguous network/server error and offers Retry same redemption. Changing the token or leaving the page discards this recovery key. This is authenticated token delivery, not an anonymous share URL. Individual share cancellation before consumption or expiry is not currently served. The /secrets/sharing console exposes the same configure → review → create → reveal → redeem/verify path, keeps the value out of the preview, and preserves each recovery key only in memory while an ambiguous request is retried. Neither value nor token is written to browser storage or product evidence.

Change approvals appear beside sharing and reuse the same dual-control approval store as privileged issuance: with ca.policy.require_approval enabled, rotate/recover/ delete mutations open a tenant-scoped approval request and fail with 403 until enough distinct approvers approve it, and a requester can't approve their own change. Approvers call POST /api/v1/secrets/store/approvals/{name} with {"action":"rotate"}, {"action":"recover"}, or {"action":"delete"}; the response holds only resource, action, approver, and the approval count.

In the console

The console renders the native store at /secrets and separate task-focused workspaces for machine access, one-time sharing, engines, scanning, and delivery. Transit lives in the Encryption and signing task at /secrets/engines; the exact console/API boundary is listed above. Bulk secret import reviews the selected JSON file's names, executes one batch, and displays its metadata receipt. See The web console.

Use it

Served workflows run through the API/CLI; embedders can use the same lower-level Go interfaces directly, shown here in byte-oriented shape:

store.Put(ctx, "db/password", []byte("s3cr3t"), "idem-key") // -> version 1
val, _ := store.Get(ctx, "db/password")                     // latest live version
lease, _ := dyn.Issue(ctx, "postgresql", "readonly", time.Hour, "req-1") // auto-revoked
ct, _ := keyring.Encrypt(ctx, "app-key", []byte("hello"), nil) // -> "trv:1:..."

The secretstore.APIServer exposes the store over HTTP (PUT/GET /secrets/<path>, with Idempotency-Key and tenant headers) once mounted.

cat > secret-sync.json <<'JSON'
{"name":"sync/source","target":"github-actions","remote_key":"DB_PASSWORD"}
JSON

# Effect-free: reads metadata only and contacts no target.
trstctl-cli secrets syncs preview -f secret-sync.json

# Copy request_fingerprint from preview into preview_fingerprint, without adding a
# secret value. Execution rejects a stale fingerprint if the secret version,
# destination, caller, or remote key changed after review.
cat > reviewed-secret-sync.json <<'JSON'
{"name":"sync/source","target":"github-actions","remote_key":"DB_PASSWORD","preview_fingerprint":"sha256:<copy-from-preview>"}
JSON
trstctl-cli --idempotency-key sync-db-password-1 secrets syncs run -f reviewed-secret-sync.json

# Use the job_id returned by run; repeat this read after reconnecting to see the
# durable worker outcome. A delivered receipt still needs target readback.
trstctl-cli secrets syncs status <job_id>

curl -fsS -H "Authorization: Bearer $TRSTCTL_TOKEN" \
  "$TRSTCTL_URL/api/v1/secrets/syncs/targets"

Pitfalls & limits

• Serving status: rotation serves worker-queued connector-backed manual and scheduled variants and durable static-provider commands for PostgreSQL, MySQL and AWS IAM providers that opt in with allow_static_rotation (AWS IAM verification is recorded as not available). Dynamic-lease rotation still fails closed before provider effects. An unconfigured secret-sync target fails closed with 503 instead of dropping the write; a missing Gitleaks binary fails scanning closed with 503 too.
• Machine login tenant binding: token credentials MAC-bind the tenant, the machine-login audience, principal, and expiry. X-Tenant-ID is a lookup hint on the public login route; a token for tenant A is rejected if presented with tenant B. The builtin HMAC verifier itself must also name exactly one auth_token_tenant_id and one or more auth_token_scopes; startup rejects an unpinned or unscoped authority.
• Protect the KEK. Everything at rest is only as safe as TRSTCTL_SECRETS_KEK_FILE; in production back it with an HSM/KMS.
• Dynamic beats static. Prefer dynamic/ephemeral secrets over long-lived ones; if you must store one, put it on a rotation schedule.
• Transit/KMIP boundaries: appliance-specific templates are not shipped; tenants manage their own KMIP clients with client registrations — see Current limitations.
• Sync is push + drift-detect, not a two-way merge — trstctl is the source of truth.

Reference

• At rest: envelope encryption (AES-256-GCM DEK wrapped by KEK); config TRSTCTL_SECRETS_KEK_FILE.
• Store: Put/Get/GetVersion/Versions/Rollback/Delete/Purge; APIServer (PUT/GET /secrets/<path>, Idempotency-Key).
• Developer run wrapper: trstctl-cli run --secret ENV=secret/path -- <cmd> fetches via /api/v1/secrets/store/{name} and injects only into the child env.
• Dynamic backends: postgresql, mysql, mongodb, aws-sts (aws-iam alias), gcp-iam, azure-entra, kubernetes, redis, plus pki.
• Transit: /api/v1/transit/{keys,encrypt,decrypt,rewrap,hmac,sign,verify}, trstctl-cli transit ..., versioned trv:<n>: ciphertext.
• KMIP: opt-in mTLS listener (TRSTCTL_PROTOCOLS_KMIP_ENABLED=true) for AES-256 SymmetricKey Create/Register/Get/Locate/Revoke/Destroy and RSA/ECDSA Create Key Pair, Activate, Sign, Signature Verify and the attribute operations, default address :5696, tenant bound by TRSTCTL_PROTOCOLS_KMIP_TENANT_ID.
• Sync targets: AWS Secrets Manager, AWS Systems Manager Parameter Store, GCP Secret Manager, Azure Key Vault, GitHub Actions, GitLab CI, Vercel, Netlify, Kubernetes, Terraform Cloud/OpenTofu, HashiCorp Vault/OpenBao KV v2, signed webhook, generic CI/JSON.
• Scanning: GET /api/v1/secrets/scans/repositories, POST /api/v1/secrets/scans/repositories/{provider}/webhook, GET /api/v1/secrets/scans/third-party, POST /api/v1/secrets/scans/third-party/{provider}/ingest, POST /api/v1/secrets/scans; trstctl-cli secrets scans subcommands repositories, repositories webhook, third-party, third-party ingest, run, staged-diff, pre-commit install; Gitleaks v8.27.2, 213 default rules active, workspace and full-Git-history modes, additive custom [[rules]] fragments, CI-log/container-registry/Slack/Jira artifact ingress, redacted findings only.
• Events: secret.version.written, rotation.*, rotation.rollback_failed, secret.rotation.connector_cutover, secret.rotation.connector_rolled_back, secret.rotation_schedule.upserted, secret.rotation_schedule.ran, auth.session.issued, discovery.finding.recorded, discovery.run.completed.

See also

Workload identity (attestation behind ephemeral secrets) · Issuance & certificate authorities (HSM-backed KEK; PKI engine) · Incident response & JIT (compromise + approvals) · Discovery & inventory (finding existing secrets) · Current limitations · glossary: secret, envelope encryption, KEK/DEK, dynamic secret, lease, transit, KMIP

Covers: F37, F38, F39, F63, F64, F65, F66, F67, F68, F58, F60, F80