Deployment connectors — get the renewed certificate onto the thing that needs it

What it is

Issuing a certificate is only half the job; it has to actually land on the server, load balancer, or appliance that will use it. A deployment connector is a small implementation that knows how to install a credential on one kind of target — write it to nginx and reload, import it into AWS Certificate Manager, push it to an F5 load balancer — and trstctl ships a set of them plus an SDK for writing your own.

The mental model: the CA cuts a new key; a connector is the courier who drives it to the right door and fits it, carrying only a narrow, sealed instruction packet so it can't wander into rooms it has no business in.

Why it exists

The painful, error-prone part of certificate management is the "last mile": copying a new certificate to dozens of systems, each with its own format, API, and reload dance, reliably and without an outage. Connectors make that automatic and safe: deployment is driven by the outbox so it can't be lost, it's idempotent so a retry can't break anything, and each connector is sandboxed so a buggy or hostile one can't reach your database or keys.

How it works

The connector SDK and its sandbox

Every connector implements exactly three methods: Name(), Capabilities(), and Deploy(ctx, sandbox, deployment). The deployment carries the certificate and key as wipeable []byte buffers, zeroed after use, plus a fingerprint; everything else comes from the SDK, so a connector stays tiny and focused.

Sandboxing uses the same capability-grant model as trstctl's WASM plugins: a connector declares the narrow capabilities it needs — fs.write to a path prefix, net.dial to a host, process.exec to run a reload — and the sandbox checks every operation against that grant, denying anything outside it with ErrDenied.

On the control plane the shipped families also run in the WASM sandbox by default (connectors.isolation: wasm). The binary embeds one first-party connector module, built from the same source and pinned by its SHA-256; every deploy, readback, rollback, takedown, preview, TLS-posture or appliance-key operation runs in a fresh wazero instance of it. Inside, the connector has no filesystem, socket, process or database access: each of its sandbox operations crosses to the host, which applies the family's grant to the attempt's operations exactly as for an in-process connector, and the instance's memory is wiped when the operation ends. The connector catalog reports isolation: wasm_sandbox for these families. connectors.isolation: in_process runs them as trusted code in the control plane instead. Enrolled host agents and network relays run the families they execute in the same sandbox by default (trstctl-agent --connector-isolation); an agent does not report its choice, so the catalog makes no isolation claim for agent-executed families.

Delivery is reliable and journaled: the orchestrator writes a connector.deploy message in the same transaction as the state change that requested deployment, so a crash can't drop it. Before sealing the credential-bearing payload it stamps the row with the execution vantage from the closed connector census. The control-plane worker refuses every row stamped host before it opens the payload or looks up a native connector. An old row without a stamp is classified from its connector family and is refused the same way; an unknown stamp also fails closed. This means the one row cannot race between the control plane and a host agent, and a missing or offline host agent leaves it visibly pending instead of silently writing the control-plane filesystem.

An enrolled host-role agent claims that durable row over the served mTLS channel, redeems its sealed credential once for that attempt, and executes the connector on the machine that actually serves the workload. The shipped host census has 14 families: the 13 file/reload connectors plus co-resident Envoy SDS. A generated parity test compares this list with the control-plane vantage census, so a newly classified host family cannot ship without an agent constructor. Cloud-store work remains in the control plane; appliance migration keeps its separately documented network-relay gate.

After choosing the permitted execution path, the worker checks the trusted native ConnectorRegistry, then provenance-verified signed WASM plugins. If neither owns the name it records a failed receipt and leaves the row pending; a queued receipt has attempts=0 and means only that durable work exists, not that deployment happened. Deterministic/readback connectors may retry after a crash; others without an enforced replay contract are claimed before I/O and left operator-indeterminate after an ambiguous failure, avoiding a duplicate mutation. Connectors compute fingerprints and request signing through the signer boundary; none do crypto directly.

Verified relay plugins

A network relay reports what its running plugin loader actually accepted through a signed, metadata-only census on its mTLS heartbeat. Each row contains the stable plugin name, verified module digest, publisher fingerprint, the network_relay_wasm execution context, and its effective grants and normalized constraints. This list comes from the same load operation that checked the module signature and started the WASM instance; scanning a directory would only prove that a file exists, not that the runtime trusted and loaded it.

The relay signs the tenant, its certificate common name, the normalized list, and the issued-at time with its agent identity. The control plane independently takes the tenant and relay identity from the authenticated certificate, rejects tampered, stale, cross-tenant, or host-agent reports, and projects an accepted heartbeat from the immutable event stream. A signed empty census is explicit evidence that the relay currently loaded no partner modules. A missing census means the agent version cannot report this state; the two cases are not collapsed.

GET /api/v1/connectors/catalog returns these per-relay rows in relay_plugins, with cursor pagination. The Verified relay plugins table on Connectors shows the relay identity, signature status, signer certificate fingerprint, plugin provenance, execution context, effective grants and normalized constraints, and report time. This surface contains no module bytes, publisher keys, credential bytes, or secrets. It is evidence of what the relay loaded, not a control-plane mechanism for pushing modules or widening an operator-owned grant.

Retries use capped backoff with jitter, and a tenant/destination circuit breaker opens after repeated failures, skipping new claims until a half-open probe succeeds. A failure that provably happened before any receiver I/O (a local refusal such as a certificate profile's, a missing configuration, a policy gate) says nothing about the receiver, so it never counts toward the circuit; it is bounded by its own message's retry budget. Operators inspect live state with GET /api/v1/connectors/outbox-circuits; Prometheus exposes transitions through trstctl_outbox_circuit_transitions_total{destination,from,to} (circuits stay per tenant and destination; the metric names no tenant because /metrics is unauthenticated).

The initial connector set (F7)

The first cohort covers the most common deployment targets, in two shapes: write a file and reload, or call a cloud/appliance API.

• Web servers — nginx, Apache, Caddy, HAProxy, IIS, Traefik: write the cert/key (or a PKCS#12/PFX), validate the config, and reload or update the dynamic file provider.
• Cloud certificate stores — AWS Certificate Manager, Azure Key Vault, GCP Certificate Manager: import via each provider's native API.
• Other targets — Java KeyStore (deterministic PKCS#12/JKS), Envoy (SDS push), Postfix and Dovecot (mail-server reload), and F5 BIG-IP (iControl REST bind to the SSL profile).

Additional connectors (F27)

The second cohort adds network appliances that speak HTTPS APIs instead of the file-and-reload pattern — Citrix NetScaler/ADC, Cisco ASA/ISE, Fortinet FortiGate/FortiWeb, and Palo Alto PAN-OS — each declaring only net.dial to its appliance host.

Use it

Console journey: Where credentials are installed

The /connectors page is named Where credentials are installed because the operator's first question is not “which plugins exist?” It is “which systems can trstctl update, and what proof says those systems are healthy?” The opening view therefore shows the configured-destination count, the count with listener verification evidence, and how many connector types can execute rollback. A target without verification evidence is never described as healthy. Opening one destination reads its rollback_readiness: whether a rollback would find a predecessor to restore (re-bind on the appliance or the host agent's retained bundle) and, when it would not, the exact reason and what makes it rollable.

The first view has one action, Add destination. Its dialog explains that saving a destination does not deploy a credential. It accepts a non-secret name, one connector type, and the connector's exact JSON configuration; credential fields must be secret:// references. Cancel closes the dialog without an API mutation. Binding, testing, deploying, and rolling back remain separate reviewed actions.

Endpoint enrollment preview checks the DNS name, API enrollment protocol, and effective validity against the exact certificate-profile revision shown in the review. A mismatch is refused before an identity is created or issuance is queued. For a new or replacement identity, choose a matching Certificate profile in the CA step. Open profiles in another tab to create or review one, then refresh the choices without losing the endpoint inputs. A reused identity keeps its existing policy; review its identity profile (profile_name or profile) before trying a different policy. The signer still validates the actual CSR, key strength, and requested key usages when issuance runs; a metadata preview does not replace those checks.

Choose Certificate key algorithm in the same CA step when enrollment or replacement must use a specific subject key. The choice is retained for renewal; leaving it blank preserves a configured algorithm on an existing or replaced identity. ML-DSA needs a compatible CA, certificate profile, endpoint, and clients. It uses a host-executed destination so the host keeps the private key through signing and installation. Configure that host's native TLS probe before deployment and independently test the application's clients. A successful metadata preview cannot establish receiver or client compatibility.

Preview and execution accept subject_key_algorithm: ECDSA-P256, ML-DSA-44, ML-DSA-65, or ML-DSA-87. An explicit selection is checked against the profile and included in preview_fingerprint; changing it requires a new review. The console refuses a preview that drops or changes the explicit choice. A reused requested identity cannot be repurposed from one configured algorithm to another; use the replacement flow for that change. The existing CA selection and approval checks still apply. This endpoint flow is separate from binding a CBOM asset into a migration campaign; issuance alone is not evidence that an asset was migrated.

Three closed sections keep implementation machinery available without making it the default reading path:

• Destinations and safe actions loads identities only when opened, then exposes target evidence, binding, Preview changes (no writes), deploy, Review restore, and the exact activity timeline. Preview shows a real dry_run_planned or dry_run_blocked receipt when the target agent or relay answers. If no target-side executor is enabled, it says Configuration valid — target not contacted and explicitly refuses to treat that local check as deployment readiness. An asynchronous preview is refreshed through its exact idempotency-linked result receipt, so a different target's answer cannot be substituted. The lookup filters by the exact receipt key on the server; it does not depend on which page of tenant history is loaded. API clients can use GET /api/v1/connectors/deliveries?idempotency_key=<URL-encoded-key> with the queued preview's key followed by :result. An empty result means no matching result has been projected yet. This read requires connectors:read and remains scoped to the authenticated tenant.
• Health, retries, and rollback loads delivery receipts, listener verification, key custody, and outbox-circuit evidence only when opened. These are distinct observations; a successful outbox attempt does not silently stand in for a listener check.
• Connector capabilities and plugin evidence retains the complete registry, execution vantage, rollback behavior, device proof, relay-migration disposition, signed publisher identity, and effective plugin grants.

The closed sections reduce first-view noise only. They do not remove APIs, evidence, or state-changing controls, and they do not prefetch the larger expert datasets until an operator asks to inspect them.

Delivery receipts appear by most recent update, with the update time beside each status. A retry can therefore bring an older receipt back to the top. The API uses updated_at descending, then receipt ID descending to break timestamp ties. Continuation cursors retain both values, so updating or removing the last row of a page does not shift its older-page boundary. Refresh the first page to see new activity; paginated history is not a frozen snapshot. Older UUID-only cursors are accepted while their receipt still exists in the caller's tenant. An expired or invalid cursor returns HTTP 400; restart from the first page. Tenant, identity and exact receipt-key filters apply to every page.

Receipt details open at the heading so the receipt identity remains visible on small screens. Tab and Shift+Tab stay within the dialog; Escape closes it and returns focus to the Details button that opened it.

Enrollment, replacement, and recovery share paged identity choices. The console shows how many identities are loaded and offers Load more identities whenever the server returns another page. A failed page can be retried without discarding the existing choices. After an action, the selected identity is reloaded by its exact ID even when it falls outside the first page; its absence from that page does not clear the operator's selection.

Replacement preview refuses a different request when the original already has an active successor, and identifies that successor for recovery. Complete or revoke that successor before starting another replacement. Reviewing or retrying the same unchanged request remains safe and recovers its existing identity; execution still rechecks the original and successor while holding the original's database lock.

One destination has one active certificate writer. Enrollment and replacement review name a competing identity and return HTTP 409 before queuing issuance; execution checks again while locking that destination. A legacy destination with multiple active bindings also holds automatic renewal so different CAs cannot keep overwriting one listener. Retire or move the competing identity, check the live listener independently, then confirm the automation plan has no target blocker. For another replacement of a verified successor, the reviewed replacement chain may retain older ancestors in deployed state while their revocation is resolved. Those ancestors are not independent writers; an active identity outside that exact chain still blocks the new request. Each replacement must name its direct predecessor, and the live listener must be checked before retiring older records. Resolve the conflict before the served certificate expires: a held renewal does not extend its validity. A historical deployed label or verified receipt does not establish what the listener serves now.

Tenant operators create non-secret deployment targets through the served API, CLI, or console. A target names the connector, the route name, and references to credentials or operator-managed endpoint config; it never stores passwords, tokens, private keys, or certificate key bytes.

The control-plane operator first enables the native connector. For a local file/reload target, the operator also enrolls a host-role agent and starts an agent channel. Enable endpoint.renew in agent_channel.claimable_job_kinds for host-generated keys (executor: agent): this kind performs both the first issuance and later renewals. connector.deploy handles delivery of centrally prepared material and does not replace endpoint.renew. Enable connector.test for target tests and connector.rollback for supported restores, endpoint.contain for emergency stop of a compromised leaf on a host listener, and endpoint.purge to remove that leaf's certificate and key files and ledger snapshots from the host. Enabling endpoint.contain also enables endpoint.recovery_check, the read-only check a replacement reads after a containment. Start the agent with --relay-claim, and place the exact roots and executables in the file named by --host-exec-profile. The profile lives on the target host because an allowlist for /usr/sbin/nginx on the control plane says nothing about the binary the target host will actually run. Tenant target rows select a logical profile name but cannot add a root, executable, or argument:

{
  "allowed_roots": ["/etc/nginx/tls"],
  "actions": [{
    "logical_name": "nginx",
    "command": "/usr/sbin/nginx",
    "pass_args": true,
    "timeout_seconds": 15
  }]
}

The containment and combined key-compromise previews return 503 with the exact missing job kind when the channel is absent or endpoint.contain is disabled.

Cancel in-flight work. POST /api/v1/connectors/targets/{id}/cancel (trstctl-cli connector target cancel <id> <body.json>, body {"reason": ..., "identity_id": optional}) drains the target's deploy/rollback lane and answers with three honest dispositions plus a cancelled receipt for each stopped command. A command nobody holds is withdrawn: it stops being claimable and the target was not changed by it. A command an enrolled agent holds is recalled: the agent's next lease extension, credential redemption and rollback authorization are refused and a late signed result is retained as a receipt rather than applied, but the agent may already have changed the target, so verify the listener and roll back if needed. A command the control-plane outbox worker is executing inside a connector call right now is reported as executing and left alone, because interrupting it would leave the appliance half-written; cancel again once it lands, or roll back. A second cancel finds nothing. Cancellation never undoes an effect that already happened. If an operator withdraws that allowlist entry after accepting a command, the existing host job remains queued; restore the entry to let the enrolled host claim it, then verify the signed receipt and listener independently.

For a host that serves ML-DSA certificates, the optional tls_probe_openssl property in that same local profile names an absolute OpenSSL executable, for example "tls_probe_openssl": "/opt/openssl-3.5/bin/openssl". Install a build that supports the selected ML-DSA signature and verify its provenance on the host. The file must be a regular executable, not a symlink; a missing or invalid path refuses the profile at startup. Tenant targets and jobs cannot supply this path.

Stop a host that still serves a compromised leaf

CA revocation changes relying-party status; it does not stop Apache, NGINX, or another workload from serving the old certificate. For a host target, the host operator can bind one exact target UUID to one local TLS listener and one pinned stop action in --host-exec-profile. The action's logical_args: [] is intentional: a missing list permits connector-supplied arguments and is refused for containment. The executable must be a regular non-symlink file; pass_args must be false. A listener and action cannot be shared by two target bindings. Stopping a service may still affect other listeners that service owns, so the host operator must review that service boundary before installing the profile.

{
  "allowed_roots": ["/etc/apache2/tls"],
  "actions": [{
    "logical_name": "stop-incident-apache",
    "logical_args": [],
    "command": "/usr/sbin/apachectl",
    "args": ["-k", "stop"],
    "pass_args": false,
    "timeout_seconds": 15
  }],
  "containments": [{
    "target_id": "11111111-1111-4111-8111-111111111111",
    "address": "127.0.0.1:10443",
    "server_name": "payments.internal.example",
    "action": "stop-incident-apache"
  }]
}

In Where credentials are installed, select the exact host target, choose Review host containment, inspect the leaf the host serves (its last delivered leaf, or the pair a rollback restored), identity, target revision, enrolled agent and outage warning, enter an incident reason, then choose Queue exact host stop. This review is still available when the target is disabled: disabling its trstctl configuration does not stop a live listener. The server rechecks the reviewed binding at submission. The agent checks the live fingerprint immediately before the pinned action. A different leaf or an unobservable listener causes no stop. After the pinned stop action, the agent records up to 40 local TLS probes 250 milliseconds apart so a graceful reload can finish. A successful stop requires the last two consecutive probes to observe no serving leaf; the signed receipt retains earlier observations that still saw the compromised leaf. The same receipt reports queued, stopped, different-leaf, unverified, or failed; only stopped describes an executed action with repeated local TLS refusal. Check from the client segment with a stock TLS client and verify CA revocation separately. The immutable request records the exact host command, outbox ID and receipt ID. If its event append survives a failed database transaction, startup recovery restores that same job and receipt; retrying the same idempotency key cannot silently allocate a different host action. If the host is unreachable, inspect Jobs and queues and restore the exact agent/profile path. After a terminal non-success, review a new attempt; the old idempotency key must not be reused for a changed target or reason. Predecessors from before fingerprint naming. An appliance rollback tries the fingerprint-named predecessor object first and, when none exists, re-binds to the object the earlier deploy installed under the target-derived name; the receipt says so (rolled_back_to_legacy_object) and does not claim the object's contents were re-read. A host agent adopts the pair it finds installed before its first deploy to a target as its retained predecessor.

Cloud stores and rollback. A gcp-certificate-manager target that names a certificate_map_entry deploys each renewal as a new fingerprint-named Certificate resource and points the map entry at it, so the predecessor stays in the project and the served rollback route re-points the entry back to it without uploading a key; the rollback executes on the control plane's outbox worker, where that family's deploys also run, and its receipt reads rolled_back the same way an agent's would. A target without the map entry patches the named certificate in place and its rollback is refused as not configured. aws-acm and azure-keyvault have no executable rollback, and the refusal and the destination's rollback_readiness say why: ACM re-imports in place under one ARN and keeps no previous version, and Key Vault serves the newest version by name with no call that makes an older one current. Objects accumulate one certificate per deploy on a map-entry target and nothing prunes them.

Take down a rogue object on an appliance. For f5, kemp, netscaler and a10, connector.takedown lets a network relay delete the fingerprint-named certificate and key objects a deploy installed, under the same credential redemption and grant as a deploy. It never unbinds a listener: an object the target still serves is reported takedown_bound and left in place, so roll back or deploy a replacement first. The rogue certificate listing names each delivered target with its remediation model, and POST /api/v1/revocation/rogue-certificates/takedown dispatches to this takedown or, for a host target, to the material purge below. Enable connector.takedown in agent_channel.claimable_job_kinds for the relay.

Remove compromised material. Containment stops the listener; the compromised private key is still on disk and in the agent's rollback ledger. GET /api/v1/connectors/targets/{id}/purge/preview and POST /api/v1/connectors/targets/{id}/purge (trstctl-cli connector target purge-preview / purge, the same reviewed request body as containment) queue endpoint.purge for the exact enrolled host agent. The agent reads the installed certificate at the target's configured paths (cert_path/key_path, HAProxy crt_path, the Postfix/Dovecot pair) and removes nothing unless its fingerprint is exactly the reviewed leaf; it then overwrites each file with zeros, removes it, wipes every rollback-ledger snapshot carrying that fingerprint so no rollback can reinstall it, and runs this target's bound stop action from --host-exec-profile when one exists. The receipt reports purge_queued, purged, purge_different_leaf, purge_absent or purge_failed; only purged means files were removed and the ledger wiped, and its signed report names the removed paths and whether the listener was stopped. Families whose material is not an addressable file (java-keystore, iis, envoy) are refused at the preview. After a purge the target has no certificate until a fresh issuance is deployed. CA revocation is separate. When the host refuses the action, the exact receipt moves to containment_failed and the job dead-letters instead of repeating an ambiguous stop. A signed execution can also report containment_failed, containment_unverified, or containment_different_leaf; those results are recorded without claiming a successful stop. Every non-stopped result creates a critical endpoint.containment_failed alert in the normal tenant routing and inbox. The alert identifies the identity, target, certificate fingerprint and receipt. Inspect the signed host result and live listener, correct the profile or service, fetch a fresh preview, and submit a new request with a new Idempotency-Key; the previous receipt remains as incident evidence. A host that has not claimed the job has not reported failure, so its receipt remains containment_queued; inspect the exact agent and queue age before deciding whether to attempt recovery.

The CLI keeps review separate from execution:

trstctl-cli connector target contain-preview "$TARGET_ID" > reviewed-containment.json
# Inspect the target, identity, agent, fingerprint, revision, and outage warnings.
jq -e --arg reason 'confirmed key compromise' \
  'if .ready and .effect_free then
     {target_revision, identity_id, expected_fingerprint, required_agent_id,
      preview_fingerprint, reason: $reason}
   else error("containment preview is not ready") end' \
  reviewed-containment.json > containment-request.json
trstctl-cli --idempotency-key "$INCIDENT_KEY" --force \
  connector target contain "$TARGET_ID" -f containment-request.json

Use a new incident key for a changed preview or reason. Keep the reviewed preview and exact request in the incident record without adding credential material. --force acknowledges that the command can stop a host listener; the server and agent still enforce the reviewed binding and exact live leaf.

The API uses GET /api/v1/connectors/targets/{id}/contain/preview and POST /api/v1/connectors/targets/{id}/contain; the POST needs the preview's revision, identity, leaf, agent and fingerprint plus an Idempotency-Key and reason. Read the returned connector receipt by its exact ID. A queued receipt never means the listener stopped.

Replace a certificate after host containment

Keep the stopped listener closed while creating a replacement identity in Where credentials are installed. Select the exact revoked original, the same destination, an explicit issuing CA and a profile that permits the name. Do not use connector rollback: that would restore a revoked predecessor. The replacement preview is effect-free and binds the original identity, destination revision, issuer, profile and request fingerprint.

For a service whose containment action removed one listener from its local configuration, the host operator prepares a separate, pinned recovery action in --host-exec-profile and names it in the containment binding's recovery_action. For NGINX, the connector calls the logical nginx action with only -t and -s reload. An operator-owned executable can whitelist exactly those arguments and, during -t, re-enable only the contained server block after the connector writes the new certificate and key. Before re-enabling it, check the new certificate's hostname and validity, that its public key matches the private key, and that its fingerprint is not any revoked predecessor. If configuration validation fails, put the include back into its disabled state. The subsequent reload must refuse an invalid or revoked leaf. Keep the executable as a regular, non-symlink file under host ownership; the tenant request cannot replace it. Other listeners should continue to use their normal NGINX test and reload path while this one is contained. The recovery action must be one of the profile's actions and must not be the stop action:

{
  "actions": [
    {"logical_name": "stop-incident-nginx", "logical_args": [], "command": "/usr/local/sbin/nginx-contain",
     "args": ["disable", "payments"], "pass_args": false},
    {"logical_name": "nginx", "command": "/usr/local/sbin/nginx-recover", "args": ["payments"], "pass_args": true}
  ],
  "containments": [{
    "target_id": "22222222-2222-4222-8222-222222222222",
    "address": "127.0.0.1:8443",
    "server_name": "payments.example.test",
    "action": "stop-incident-nginx",
    "recovery_action": "nginx"
  }]
}

After the stop, the replacement preview reads the listener's host recovery. Until a recovery check has run since the stop it reads ready: false with recovery_readiness.state unchecked, and the console shows Check host recovery readiness (API POST /api/v1/connectors/targets/{id}/recovery-check with the contained identity_id; CLI trstctl connector target recovery-check <target-id> -f body.json). The target's enrolled host answers that read-only endpoint.recovery_check job in a signed receipt: whether its profile pins a recovery action for this listener, the SHA-256 of that action's executable and of its pinned command line (never the path), and what one local TLS probe of the listener saw (stopped, serving, serving_revoked or unverified). Build the preview again once it answers. It reads ready only when a check requested after the stop shows a recovery action the target's connector runs while deploying (for NGINX, nginx) and a listener that is not serving the contained leaf; otherwise it names what is missing, and creating the replacement answers 409 with the same reasons and queues nothing. A check that ran before the stop, or before the target or its enrolled host changed, reads stale. GET /api/v1/connectors/targets/{id}/recovery-check (CLI recovery-status) reads the latest check and its job ID.

The host agent's deployment receipt must then say verified for the new fingerprint, and a fresh stock TLS client must see that same leaf and validate its hostname and chain. Read the revoked original and replacement by their distinct identity IDs, then retire the original only after checking its CA revocation result. A failed deploy leaves the listener stopped; an HTTP 201 binding response alone is not recovery proof. The recovery check reports what the host profile pins and what the listener answered, not what the action will do: test that the action re-enables only this listener and refuses every revoked predecessor before relying on it.

One reviewed key-compromise response

When a managed X.509 identity's key is compromised and one registered host is still serving its leaf, open Certificates → Revocation & CT → Revocation center, select that identity, reason keyCompromise, and the exact host target. The ordinary identity transition preview is ready=false for a serving X.509 key compromise, and ordinary execution returns 409 with key_compromise_containment_required before any event or outbox intent; a CA-only lifecycle success cannot claim endpoint containment. The effect-free preview reads the issuing CA authority for every bounded certificate record, the current identity version, target revision, enrolled agent, and last served leaf. A missing authority or exact host binding refuses the command. The route requires identities:write and connectors:write; dual-control policy still requires a distinct approver. The command emits one retained identity.revoked event and queues two independent jobs in the same tenant transaction: revocation.publish and endpoint.contain. Acceptance changes the lifecycle state but proves neither CA publication nor host stop. A host agent outage does not delay the CA dispatcher, and a CA outage does not delay the host agent. Never restore a revoked predecessor during recovery. The publisher attributes every bounded certificate to the authority that signed it from its retained issuance evidence: a leaf from the platform issuing CA publishes through the tenant CRL, a leaf from a private CA hierarchy authority (for example a host-executed identity whose issuing_authority_source is private) is revoked in that authority's own issued-certificate ledger and CRL, and a leaf from an external CA is revoked at that CA. A superseded predecessor whose managed CA key was destroyed by signed retirement is excluded by the preview and skipped by the publisher, since nothing can sign for it any more.

The downloadable trstctl-cli uses the same reviewed API command. Save its effect-free preview, inspect every authority and host binding, then copy only the exact reviewed fields into the execute body. Keep one incident key through retry and status readback:

printf '{"target_id":"%s"}\n' "$TARGET_ID" |
  trstctl-cli identities compromise-preview "$IDENTITY_ID" -f - > reviewed-compromise.json
# Inspect every certificate/authority and the exact target, leaf, agent and version.
jq '{target_id: .target.target_id, target_revision: .target.target_revision,
     target_name: .target.target_name, connector: .target.connector, identity_id,
     expected_fingerprint: .target.expected_fingerprint,
     required_agent_id: .target.required_agent_id, expected_version,
     preview_fingerprint}' reviewed-compromise.json > execute-compromise.json
trstctl-cli --idempotency-key "$INCIDENT_KEY" --force \
  identities compromise "$IDENTITY_ID" -f execute-compromise.json
trstctl-cli identities compromise-status "$IDENTITY_ID" --request_key "$INCIDENT_KEY"

The server binary's direct admin command can instead consume the reviewed preview file without the jq copy step:

trstctl connector target compromise-preview \
  --identity "$IDENTITY_ID" --target "$TARGET_ID" > reviewed-compromise.json
trstctl connector target compromise --preview-file reviewed-compromise.json \
  --idempotency-key "$INCIDENT_KEY"
trstctl connector target compromise-status \
  --identity "$IDENTITY_ID" --request-key "$INCIDENT_KEY"

API clients use POST /api/v1/identities/{id}/compromise/preview, then echo the reviewed bindings to POST /api/v1/identities/{id}/compromise with the stable Idempotency-Key. The preview lists only certificates its issuing CA can still revoke. A superseded predecessor whose managed CA key was destroyed by signed retirement appears under excluded_certificates with a warning instead of making the preview ready for a revocation nothing can sign; relying parties that still trust that retired CA must remove it from their trust stores. If the served leaf's own CA key was destroyed, the preview refuses with 409 before any irreversible command. GET /api/v1/identities/{id}/compromise?request_key=… reads the CA outbox attempt or retained completed-worker receipt and the exact host receipt separately; it requires identities:read and connectors:read. A delivered CA command still needs signed CRL/OCSP and a stock client that enforces revocation. A stopped host receipt still needs an independent fresh-client TLS check. If one job fails, inspect Jobs and queues, its exact receipt, and the live endpoint, then retry that one effect on the same command: POST /api/v1/identities/{id}/compromise/retry with {"request_key": …, "effect": "host"} queues a fresh endpoint.contain job for the exact reviewed leaf, target and agent on the same receipt after a containment_failed or containment_unverified result; "effect": "ca" re-arms a dead-lettered revocation.publish command with its attempt history kept. Each retry is one retained identity.key_compromise_effect_retried event, so repeating the same retry queues nothing more, a queued or delivered effect is refused with 409, and the status read follows the newest retry. The CLI command is identities compromise-retry with a {"request_key","effect"} body; the console's revocation center offers the same retry beside each effect. The command stays readable and retryable after the identity is retired: the compromised leaf, not the identity's later state, is what needs containment. Historical commands whose delivered outbox row expired before completion receipts were introduced report evidence_unavailable for the CA effect while retaining the host result. This means the CA outcome is unknown, even if the identity says revoked. Check every serial with its issuing CA and signed status service before any new reviewed command; retrying the ambiguous old key is refused. Reusing the original key with changed bindings is also refused.

The ordinary probe runs first. If a direct TLS handshake fails, the configured native probe attempts TLS 1.3 within the same deadline and reports the exact presented leaf and chain. This applies to target tests, deployments, host renewals and host rollback. It preserves working classical and STARTTLS paths; it does not replace a failed STARTTLS negotiation with direct TLS. Without the option, a listener Go cannot handshake remains unverified. The native probe sends no application requests and does not establish CA trust or revocation status. Those checks, and testing the application's actual operation, remain separate qualification steps.

Envoy uses its co-resident HTTP SDS endpoint and does not consume filesystem or exec permissions, but it is still host-vantage work: the control plane never falls back to dialing a loopback Envoy endpoint on its own machine. The host executor accepts only localhost or a literal loopback IP, so an Envoy target cannot become arbitrary network egress from the agent host.

A tenant with connectors:write then creates the target with the connector's strict schema and drives its lifecycle. Replace the paths and listener address with the service's actual values; the verification name must match the requested identity's DNS name:

: "${AGENT_ID:?Set this to the enrolled host agent UUID shown by agents list}"
: "${IDENTITY_ID:?Set this to a requested X.509 identity UUID}"
TARGET_ID=$(jq -n --arg agent "$AGENT_ID" '{
  name:"edge/prod/payments", connector:"nginx", enabled:true,
  config:{profile:"nginx-prod",cert_path:"/etc/nginx/tls/payments.crt",
    key_path:"/etc/nginx/tls/payments.key",executor:"agent",
    required_agent_role:"host",required_agent_id:$agent,
    verify_address:"127.0.0.1:443",verify_server_name:"payments.svc"}
}' | trstctl-cli --idempotency-key payments-target-create \
  connector target create -f - | jq -er .id)
jq -n --arg target "$TARGET_ID" '{target_id:$target}' |
  trstctl-cli --idempotency-key payments-target-bind \
    connector target bind "$IDENTITY_ID" -f -
trstctl-cli --idempotency-key payments-target-test \
  connector target test "$TARGET_ID"
jq -n --arg identity "$IDENTITY_ID" '{identity_id:$identity}' |
  trstctl-cli --idempotency-key payments-target-deploy \
    connector target deploy "$TARGET_ID" -f -
# After a later rotation leaves a proven predecessor, rollback uses the same body:
jq -n --arg identity "$IDENTITY_ID" '{identity_id:$identity}' |
  trstctl-cli --idempotency-key payments-target-rollback \
    connector target rollback "$TARGET_ID" -f -

target deploy is not a private-key recovery command. For a requested identity, it binds the target first and starts issuance so the new certificate and its private key move directly into one sealed connector job. If an identity was already issued before that credential-bearing path existed, trstctl does not have a retained private key to copy: the deploy call returns 409, leaves state and queues unchanged, and tells the operator to wait for an already-bound issuance or renew/reissue after binding. A running renewal is also left alone; its successor will deploy through its own bound job. This refusal prevents the console from saying Deployed when no executor received a credential.

When this path generates the key in the control plane, trstctl retains a tenant-bound encrypted copy before contacting the CA. A retry can then finish deployment using the original certificate and key. Temporary plaintext buffers are wiped; successful delivery does not delete the encrypted recovery record. The lifecycle preview discloses this custody before authorization. A destination with executor=agent instead generates its key on the host and submits only a CSR.

The identity's activity timeline includes failed host attempts even when signing stops before any certificate exists. It shows the assigned agent's safe error, attempt count, and exact background job ID. A failed attempt does not confirm delivery or offer a restore reference. Later recovery updates the same delivery record; the event history retains the earlier failure. Inspect the named profile, destination settings and issuance job to resolve the reported cause. A fixed signing-stage message means the host did not receive a usable certificate; the CA may still have issued one, so inspect the retained issuance result before retrying. If any attempt of the job redeemed credentials, arbitrary free-text errors stay withheld from permanent history. Only the exact known host-signing marker is classified into fixed public text. The redemption audit reference and attached evidence remain available. An agent that claims jobs (--relay-claim) keeps each failed connector deploy's step and scrubbed reason in its own failure log on its host, read there with trstctl-agent --show-deploy-failures; a failed host renewal's underlying error stays in the agent's process log.

When an external CA returns a certificate, trstctl saves that response before recording the issuance locally. A recording failure can then retry with the same request key and recover the same certificate without another CA call. Success still requires the issuance record and notification intent to commit. If the CA call failed without returning a certificate, an ACME adapter may permit retry only when that live attempt proves it stopped before submitting the CSR for finalization. An ambiguous result, a process crash after claiming the provider call, or an unverified claim-release failure does not authorize another issuance. See the external issuance recovery contract.

Delivery and endpoint verification appear separately in the activity timeline. Delivery keeps the actual retry count, job ID, and restore reference. Verification shows the observed result and time for that exact delivery; its single probe does not reset the delivery's attempt count. The console matches the recorded operation key, identity, connector, target, and certificate fingerprint before associating those receipts. A missing match stays unverified. These are historical observations, not a promise that the listener still serves the same certificate now.

The Certificates cockpit separates current repair work from history. Its deployment attention count uses the latest loaded receipt for each live certificate's identity, connector, destination, and target. A later verified retry or successor clears an older failed attempt from that count; the failed attempt remains visible in the outcome chart and receipt history. The count is based on the loaded tenant-scoped read, so a fresh listener handshake is still needed to prove what serves now.

The certificate detail drawer limits this history to the selected fingerprint and its retained identity links. A later successor's delivery or renewal is not proof for the selected certificate. If several identities used that certificate, choose one identity at a time. Missing links or a missing fingerprint say that correlation is unavailable. Paginated or failed reads remain incomplete until more records are loaded or a retry succeeds; they do not prove that no lifecycle work occurred.

target test is an effect-free dress rehearsal from the machine or network that would perform the real deployment. Enable connector.test as a claimable job kind in addition to enrolling the required agent role. For all 14 host-vantage families, the host agent validates the exact target fields, confirms every path and logical command fits its operator-owned --host-exec-profile, and, when configured, handshakes verify_address using verify_server_name. With no verify_address, the plan says the deploy can run but live-state verification is not configured; it never upgrades that omission into a claim that the certificate is serving. When a discovery scan has seen a certificate delivered to the target, the destination's Verification address panel (connector target verification-suggestions) suggests that host:port with the findings as evidence, and confirming it (connector target confirm-verification-address) sets verify_address as a new target revision; nothing is set without that confirmation. It does not write a file, run a command, call Deploy, or redeem certificate/private-key material. A host secret needed to open the target, such as a Java keystore password reference, is redeemed only for that one attempt. Network-relay appliance tests instead redeem the appliance-management credential and perform the connector's read-only probe. A ready test returns dry_run_planned with the later mutation list; an unreachable target, missing grant, or other deterministic prerequisite returns terminal dry_run_blocked rather than retrying forever. A signed relay plugin that exports test runs it under a read-only grant (no file writes; HTTP GET and HEAD only, a write refused before it is sent); one that does not fails closed without being invoked.

For a postgresql destination, the agent first sends PostgreSQL's SSLRequest and requires the server to accept TLS before starting the handshake. This applies to preview, post-deployment checks, and verification after a restore. Scheduled network-agent checks also retain the registered target's connector type and verify_server_name, together with the expected certificate fingerprint. A database that refuses TLS fails verification; there is no plaintext fallback. These checks establish the served TLS identity, not database login or query availability. Other connector types and legacy observations without a registered target keep their existing direct-TLS behavior.

The console keeps recovery separate from deployment. Review restore opens a confirmation that names the exact destination, identity, and operator reason; no restore is queued by opening it. Identity choices and the review show the lifecycle state and full identity ID so same-name replacements remain distinguishable. Queue restore uses the served rollback route. The first response may say only rollback_queued: that is waiting state, not success. The console reports success only from a later rolled_back receipt written after the required agent or relay restores the proven predecessor. While queued, the console rereads that exact receipt every three seconds while the tab is visible. It stops polling at an attempt result or read failure; Check restore result remains available after a failure because the same job may retry. It only reads the receipt and never queues another restore. A mismatched or unavailable receipt is not shown as a successful recovery. Repeating a pending restore joins its existing job and preserves any recorded attempt result. Re-arming a finished job keeps its result ID; its next claim uses a new signed agent attempt. A delayed result from the prior execution cannot complete the new request, and replaying older evidence cannot move the displayed receipt backward. On the first upgrade with older rollback receipts, startup rebuilds the read model from retained event history before serving; a failed rebuild leaves the existing model intact and refuses startup. This also applies when restoring an older snapshot. Connector families without an executable restore path show the manual boundary and keep the button disabled; the API also fails closed instead of recording a rollback-shaped success.

Disabling a destination pauses the agent-claimable work already queued for it: deploys and rollbacks stamped with that destination's lane are handed to no agent while it is disabled, nothing is dropped, and the same rows resume unchanged the moment it is enabled again. The Jobs page keeps showing them as pending meanwhile.

The REST surface is /api/v1/connectors/targets for CRUD, /api/v1/identities/{id}/connector-target for identity binding, and /api/v1/connectors/targets/{id}/{test,deploy,rollback} for normal actions, plus the reviewed host containment routes described above; CRUD and binding are immutable events (deployment_target.upserted, deployment_target.deleted, identity.connector_target_bound) projected into the read model, so disaster recovery rebuilds routing from the event log. The effect-free POST /api/v1/lifecycle/endpoint-bindings/preview resolves one exact configured issuer and destination revision and fingerprints the proposed identity, custody, writes, queued effects, recovery, and verification. The preview also compares the single requested DNS name with a host target's verify_server_name. A mismatch is rejected before issuance, file replacement, or reload, because that deployment is guaranteed to fail its own TLS proof. The paired POST /api/v1/lifecycle/endpoint-bindings accepts only that unchanged preview: it creates or reuses the target, creates the X.509 identity for an existing owner, pins the selected built-in, private, or external issuer, binds the route, and queues the normal ca.issue/connector.deploy intents after issuance authorization succeeds. Endpoint preview and enrollment both require certs:issue as well as connectors:write: a caller who cannot start issuance gets a clear 403 instead of a ready: true plan they cannot execute. The destination's Deploy action for a requested identity requires the same permissions. They apply the same policy, attribute-based denial, quota, and independent-approval checks as ordinary identity issuance.

The endpoint review includes the active certificate profile's name, immutable version and digest, requested validity, effective validity, and approval requirement. Preview and execution accept an optional profile_name, the active profile name in the caller's tenant. The console lists active revisions; the server resolves and validates the exact revision. An explicit choice is retained on a new or replacement identity for later issuance and renewal. It cannot override a reused identity's current policy: a different choice returns HTTP 409 before mutation. Omitting the field preserves the existing identity policy or uses the configured default for a new identity. This certificate policy is separate from the target's host execution profile, which limits file paths and reload commands. The implicit request is 30 days, bounded by the profile maximum: a 12-minute profile queues a 720-second certificate request. The profile binding stays with the command when it is handed to a host agent. A changed profile or destination invalidates an unexecuted review; it cannot silently change what was authorized.

When approval is required, the first submission prepares the requested identity and destination binding, then returns HTTP 403 with code=identity_approval_required, identity_id, approval_request_id, and approval_status. This is pending review, not a completed issuance. Independent reviewers approve that exact request. Retry the original body with the original Idempotency-Key; the prepared identity and reviewed version are retained so the retry does not create another identity or invalidate the approvals. The complete identity snapshot is included in approval evidence and checked under its row lock before issuance, because metadata edits do not advance the lifecycle version. Self-approval is refused. Expired, denied, or changed authority must be reviewed again.

The console shows the profile and exact validity before submission and retains the request key while the review stays open. Open approvals in another tab preserves that review; Retry this request resumes it. This retention is limited to the open form, not a browser-reload recovery contract. API clients should persist the exact request body and key. Acceptance means issuance was authorized and queued; inspect the certificate, delivery receipt, and a fresh application connection to prove that it was issued and installed.

Scheduled renewal reuses the leader-only ca.renew path and the same issuer selection, delivering the successor to the same binding and producing a second connector receipt. The worker fails closed when that issuer is unavailable; it never silently substitutes another CA.

The shipped trstctl composition builds the trusted native registry from connectors.enabled (applications never call connector.NewRegistry directly). Remote targets carry only secret://name references, read from the tenant's encrypted secret store into locked memory for one outbox attempt then destroyed — as in an F5 target's {"endpoint":"https://bigip.internal","client_ssl_profile":"payments","username":"svc-trstctl","password_ref":"secret://connectors/f5/password"}. Private endpoints must also be admitted by the operator-owned connectors.allow_private_cidrs list; plain HTTP needs an explicit development-only loopback override that never widens the plaintext exception in production. Issue and renewal flows keep the same discipline: the issuer builds the credential-bearing connector.deploy payload while the key is in memory, wipes the exported buffer once the outbox intent is recorded, and the connector runs in its sandbox to land the certificate — receipts keep only identity, route, fingerprint, and status metadata (metadata-only operator actions still produce a receipt but claim no target mutation).

The target JSON is closed and provider-specific; unknown fields fail the attempt instead of being ignored:

Host-executed keys

Host-executed targets (the file-and-reload families above) also accept the custody and verification keys the enrolled host agent honors: "executor": "agent" makes the agent generate the private key on the host and submit only a CSR; required_agent_role (host) names the enrollment role that may claim the work. Choose Host agent when adding or editing the destination; the console loads enrolled agents, with pagination for larger fleets. The selection is saved as required_agent_id, the UUID shown on the Agents page. An enabled host destination requires an enrolled, same-tenant agent with the host role.

On an appliance or other relay-executed target, "executor": "agent" only names the agent that executes the work (a CA migration requires it on every member, with the network relay pinned in required_agent_id). No host serves such a target, so its key cannot be generated on one: the control plane generates it under the tenant's control-plane key-generation switch, seals it to the deploy job, and the relay installs it. The key-custody readout and the lifecycle preview report these targets as control-plane generated.

A host-executed first issuance does its work on the host before the identity can reach deployed: the agent generates the key, the CA signs its CSR, the files are installed and the listener is probed. The ownership-readiness authority that guards the deployed edge (an attested owner, or a recorded exception) is therefore resolved at the deploy call itself. POST /api/v1/connectors/targets/{id}/deploy, trstctl-cli connector target deploy and the console refuse with 409 and the ownership reason before any key is generated or any job is queued; attest the owner (trstctl-cli owners attest <owner-id>) and deploy again. If a lifecycle refusal still happens after the host has finished (a policy or owner change in between), the job is failed terminally so it never re-executes, and the identity's delivery timeline records a failed lifecycle_transition_refused receipt naming the job and the cause, beside the signed host receipt. The identity stays issued; repair the cause, then deploy or renew again.

To issue a host-executed identity from a private CA hierarchy authority instead of the platform issuer, create the identity with "attributes": {"issuing_authority_source": "private", "issuing_authority_id": "<CA authority id>"}. The selection is never substituted: a superseded or destroyed authority refuses signing on the host job rather than issuing from another CA, and the failed attempt is recorded on the identity's delivery timeline with that reason.

Preview, deployment, and renewal keep this exact assignment through retries and control-plane restart. An offline host leaves its work pending; another host never takes over. Changing the destination affects newly reviewed work, not jobs already queued against an older revision. Existing unassigned host jobs remain held; review the destination assignment and recover those jobs explicitly before relying on unattended renewal. Rollback stays with the host holding the verified predecessor.

If the external CA is still processing a host request, the agent keeps its locked key and repeats the identical CSR on the same job attempt. It extends that claim while waiting and confirms it again before installation. A lost claim or operator cancellation stops the attempt; an explicit signing refusal is not retried. Temporary transport failures can retry the same request while the claim remains valid. No new key is generated for a pending response.

The agent executes one claimed job at a time while continuing to heartbeat. Host renewal is bounded to ten minutes, and a job may use at most half the agent certificate's remaining life. The agent finishes or cancels that job before rotating its own channel identity, so its result receipt uses the same identity as the authenticated connection. Individual signing calls wait at most 45 seconds; pending responses request a one-second retry. Both the control plane and agent must support this pending-response contract. A process restart cannot recover a subject key that existed only in locked memory; check the certificate inventory and revoke any unused issued leaf before recovering an abandoned attempt.

Disabling lifecycle actions pauses queued host issuance and renewal as well as deployment jobs. This also covers older host jobs grouped by identity. Re-enabling the destination resumes the same queued intent; it does not replace its saved configuration or reset its attempt history.

Host-generated keys can still require a management secret. For java-keystore, save the store password in the tenant's secret store and set keystore_password_ref to its secret:// reference. Issuance, renewal, and rollback borrow that password over the authenticated agent channel for one job attempt; the certificate's private key stays on the host. Password values never travel in the queued intent or audit event.

The reference must belong to the reviewed destination configuration. Missing secrets stop a renewal before signing or writing; disabled or reassigned targets cannot release their management credentials. Retrying secret resolution does not consume an attempt's single redemption, but a successful redemption cannot be replayed. Older queued renewal jobs derive their reference names from their saved target configuration without changing the queued policy or target revision. Both the control plane and host agent need this capability to complete password-protected host issuance and restore.

Deliver into a Kubernetes Secret from inside the cluster

A k8s_secret destination is the kubernetes.io/tls Secret an Ingress or Gateway names, written by a host-role agent running inside that cluster. The agent generates the key in the cluster, sends only a CSR, writes tls.crt and tls.key, and reads the Secret back: the install succeeds only when the cluster holds the certificate just written. The key never leaves the cluster and the control plane never generates it, so the destination requires "executor": "agent"; one saved without it is refused. Custody records key_origin=host_agent and key_storage=kubernetes_secret. When verify_address names the listener that serves the Secret, such as the ingress controller's Service, the agent then handshakes it and the signed host receipt records what it served. Without one the deploy is recorded as executed, not verified.

{
  "executor": "agent",
  "required_agent_id": "<the cluster agent's UUID>",
  "namespace": "payments",
  "secret_name": "payments-web-tls",
  "verify_address": "ingress-nginx-controller.ingress-nginx.svc:443",
  "verify_server_name": "payments.example.com"
}

Run the agent in the cluster with a host enrollment, --relay-claim, and a host exec profile (--host-exec-profile) that lists the namespaces it may write. It needs no host roots:

{ "kubernetes_secrets": { "namespaces": ["payments", "edge"] } }

The agent reaches the cluster API through its pod's service account, which needs get, create and update on Secrets in those namespaces (a Role and RoleBinding in each). A target in a namespace the profile does not list is refused before any key is generated. A dry run checks the namespace, asks the cluster API for the Secret and probes verify_address; it writes nothing. Rollback restores the previous certificate and key into the Secret from the agent's encrypted predecessor ledger, so keep the agent's --key and --host-rollback-dir on persistent storage.

A k8s_ingress_gateway discovery source bound to the agent creates these destinations for you, one per TLS block; see Discovery and inventory. As for any host-executed first issuance, the source's owner must be ready for deployment (a current ownership attestation, or an ownership exception) before the agent gets any work; until then each resource is reported not delivered, naming the owner and how to attest it.

Have the appliance generate its key

An F5 BIG-IP (a single device), NetScaler, A10 Thunder, PAN-OS firewall or Cisco ISE node can make the private key itself and send only a certificate request. Declare "key_generation": "appliance" on the destination and name the network relay that reaches the appliance in required_agent_id; the destination dialog in Connectors offers both as a choice. At each renewal that relay asks the appliance for an ECDSA P-256 key and a CSR in an object named for the renewal job (trstctl-<first eight characters of the destination ID>-j<job>), has the control plane sign the CSR, and installs the certificate against the key the appliance already holds. No key crosses the relay or the control plane, and none is sealed to a deploy job. Custody records key_origin=appliance and key_storage=appliance, exportable, because these are software keys an appliance administrator can export.

{
  "endpoint": "https://ns.example.com",
  "username": "trstctl",
  "password_ref": "secret://connectors/netscaler-password",
  "key_generation": "appliance",
  "required_agent_id": "<the network relay's UUID>"
}

• F5: a single BIG-IP only. Each peer of an HA pair keeps its own crypto objects and would make a different key for one certificate, so a destination with peer_endpoint is refused; keep control-plane keys there, which reach both peers.
• PAN-OS: the destination needs ssl_tls_profile and commit_admin. The firewall's certificate object keeps its job name, the profile is set to it in a partial commit, and readback exports the certificate the profile names (never its key) to fingerprint it.
• NetScaler and A10: the key file and the CSR file sit side by side on the appliance, and the certkey or client-SSL template binds the certificate to that key file.
• Cisco ISE: the destination needs node_hostname and certificate_roles. ISE names each request itself, so the relay keeps ISE's request ID for the renewal on the control plane before it asks for a signature, and an attempt picked up again reuses that request instead of making another. The signed certificate is bound to the request under its fingerprint name with those roles; only the leaf is sent, so the node must already trust the issuing CA.

Readback works as it does for sealed keys. A rollback binds the predecessor certificate back with the key the appliance made for it, found from the predecessor's renewal job; it never reads the key, and a predecessor whose key object is gone is refused rather than bound to another key. A renewal the relay picks up again after losing it asks the appliance for the same object and gets the same CSR back, so one renewal never leaves two keys. PAN-OS keeps no CSR once the certificate for it is imported, so a PAN-OS renewal picked up after that import stops and says so; check the firewall and start a new renewal. The same holds on ISE: a request is turned into its system certificate when the certificate is bound, so a Cisco renewal picked up after its bind stops and says so.

Activate Tomcat PEM certificates

The tomcat destination writes the certificate and key paths already referenced by Tomcat's SSLHostConfig. Its fixed tomcat-tls-reload host action activates that TLS context; stock catalina.sh reload is not a supported Tomcat command. Configure verify_address and verify_server_name to independently check the certificate Tomcat serves after activation.

Use the shipped Manager client and host-profile example in Reload a Java application, replacing the example's logical_name with tomcat-tls-reload. Grant the PEM directories in allowed_roots; no keystore, store password or reload_action target field is needed for the PEM connector. The Manager password stays in its private host file. Configure the executing host agent's --host-exec-profile; the agent-bound destination uses that host's authority.

Every retry activates TLS even if the files already match, since an earlier attempt may have stopped after writing them. If activation fails, the connector restores the complete previous certificate/key pair and activates that pair again. It reports a missing predecessor or failed recovery explicitly. A failed attempt remains failed even when the predecessor is recovered.

Reload a Java application

Writing a keystore does not make an application use it. Set reload_action to a named action in the host's exec profile, and configure verify_address and verify_server_name so the agent checks the certificate served after that action. The console's Java destination example includes these fields. Omit reload_action only when file delivery is the intended outcome or the application itself watches the keystore; a file write alone is never proof of a successful TLS switch.

Both JKS and PKCS#12 preserve alias; it must match the application's configured key alias. PKCS#12 aliases use BMP characters without NUL, up to 1024 UTF-8 bytes. Use a randomly generated printable ASCII store password for stock Java PKCS#12 compatibility. A wrong format or incompatible password fails before the host requests signing. The store password reference and the application's management password serve different purposes.

For stock Tomcat, use the shipped agent's one-shot Manager client. Tomcat's catalina.sh has no native TLS reload command. The Manager TLS reload API reloads certificate files for one existing SSLHostConfig; it does not reparse server.xml. Install the Manager application and a dedicated manager-script account, restrict its access to loopback, and place its password in an owner-only file with safe parent-directory permissions. This setup is performed once by the host administrator.

For a destination with "reload_action":"java-tls-reload", the corresponding host profile action can be:

{
  "logical_name": "java-tls-reload",
  "command": "/usr/local/bin/trstctl-agent",
  "logical_args": [],
  "args": [
    "--tomcat-reload-url", "http://127.0.0.1:8080/manager/text/sslReload",
    "--tomcat-reload-user", "trstctl-manager",
    "--tomcat-reload-password-file", "/etc/trstctl/tomcat-manager.password",
    "--tomcat-reload-tls-host", "_default_"
  ],
  "pass_args": false,
  "timeout_seconds": 20
}

The host profile also needs the keystore directory in allowed_roots. The target selects only the action's name, never executable arguments or a password. The helper accepts literal loopback HTTP(S) addresses at /manager/text/sslReload, refuses redirects, and requires Tomcat's acknowledgment for the exact TLS host. HTTPS uses normal certificate validation. The password file contains one line (up to 4096 bytes); errors do not include credentials or Manager response bodies. Other Java applications can use their own fixed, operator-approved reload action.

The Java connector preserves the previous file before a reload. On failure it restores that file and runs the same action again, while reporting the original deployment as failed. A missing predecessor, failed restore, or failed recovery reload is reported explicitly. Replaying a deployment repeats the activation even when the encoded keystore bytes are unchanged, because an earlier attempt may have stopped between writing and reloading. The host validates its action profile before requesting another certificate; the final listener check remains required to prove which certificate the application actually serves.

verify_address plus verify_server_name tell the agent which listener to re-handshake after the reload, so a receipt is backed by an independent TLS check. The expectation is built from the bytes the agent actually wrote, the leaf and the issuers installed with it, and deploy and rollback verification apply one rule: a different leaf, missing or unexpected names, or a certificate outside its validity window fails. A chain-only mismatch — the deployed leaf served with other issuers than were written, as a TLS terminator serving its own chain does — passes: the receipt reason is endpoint_serving_deployed_identity_chain_mismatch (rollback: rolled_back_and_reverified_chain_mismatch) and names the mismatch, and an endpoint-chain-mismatch discovery finding records the listener until it is triaged. Set "strict_chain": true in the target configuration to fail a chain-only mismatch instead (endpoint_serving_chain_mismatch, verify_failed). For MySQL, preview, post-deployment checks, and endpoint verification first read the server greeting and send MySQL's SSLRequest. Select the mysql connector even when the listener uses a nondefault port; the agent does not guess the protocol from the port number. A server that declines TLS fails verification. This check observes the certificate without sending a database username, password, or query; application availability still needs a separate authenticated SQL check.

MySQL requires a mysql-tls-reload action in the operator's host profile. Map it to the stock mysql client running ALTER INSTANCE RELOAD TLS, which activates the server's current certificate and key files. mysqladmin reload refreshes grant tables and does not activate new TLS files. Existing profiles must replace that old action before enabling MySQL deployments with an updated agent.

For example, this host action connects through the local Unix socket and keeps the reload account's password in a private client option file:

{
  "logical_name": "mysql-tls-reload",
  "logical_args": [],
  "command": "/usr/bin/mysql",
  "args": [
    "--defaults-file=/etc/trstctl/mysql-reload.cnf",
    "--protocol=SOCKET",
    "--socket=/run/mysqld/mysqld.sock",
    "--execute=ALTER INSTANCE RELOAD TLS"
  ],
  "pass_args": false,
  "timeout_seconds": 15
}

The client file supplies the user and password and must be readable only by the agent's service account. The MySQL account needs CONNECTION_ADMIN; the SQL statement and connection arguments belong to the operator profile, never target JSON. Mirror this action in connectors.local_profiles, using timeout: "15s" there. Use a MySQL version supporting ALTER INSTANCE RELOAD TLS; preserve its default rollback-on-error behavior. Do not add NO ROLLBACK ON ERROR, which can disable encryption after a failed TLS reload.

The connector repeats TLS activation even when the file bytes are unchanged. If activation fails, it restores the previous certificate/key pair and attempts to activate that pair, while reporting the original deployment failure. Missing predecessors, failed file restoration, and failed recovery activation are reported separately. The listener check still must match the intended certificate before the deployment can be called verified. See MySQL's TLS reload contract.

Use executor: agent whenever the identity's CA is external: an external CA answers asynchronously, and the endpoint lifecycle preview refuses control-plane key custody for an external CA on a host connector rather than risk a certificate whose key no longer exists. The partner lab's working Apache target is:

{"executor":"agent","required_agent_role":"host",
 "required_agent_id":"<enrolled-host-agent-UUID>",
 "cert_path":"/lab/tls/apache.crt","key_path":"/lab/tls/apache.key",
 "verify_address":"127.0.0.1:10443","verify_server_name":"apache.partner-lab.example.com"}

For a host-executed Traefik destination, also set config_path to the existing file-provider YAML or TOML document that references cert_path and key_path, for example /etc/traefik/dynamic.yml. Traefik does not reliably reload when only those referenced certificate files change. The connector rewrites the same document after installing the files to notify the watcher, preserving its content. The document must be inside the agent's allowed roots and readable and writable there. Preview refuses a missing config_path before any certificate is installed.

Every *_ref is a secret://name or secret://name?version=N reference in the same tenant, and every profile points to an operator-owned connectors.local_profiles entry, which maps the connector's logical command (nginx, apachectl, caddy, powershell, netsh, haproxy, systemctl, postfix, doveconf, doveadm, pg_ctl, mysql-tls-reload, rabbitmqctl, or tomcat-tls-reload) to one absolute executable; general shells and symlinked executables are rejected. IIS's powershell action additionally requires an exact logical_args allowlist and a separate args list with pass_args: false, so the requested argv is only ever compared, never forwarded as a tenant-derived byte to -Command. That control-plane entry is the admission and dry-run copy; the --host-exec-profile file is the authority enforced where the effect runs, so operators keep their roots and actions aligned. To add a target trstctl doesn't ship, follow the connector authoring guide.

Host rollback never sends predecessor key material back through the control plane. Run the host agent with --relay-claim --host-exec-profile <file>; it keeps active plus one predecessor in an encrypted local ledger at --host-rollback-dir (by default host-rollbacks beside the agent key). A rollback is routed to that exact enrolled agent, serialized with deploys for the target, restores files, runs the allowlisted reload, and re-handshakes verify_address. The migrated appliance families use object re-bind instead: f5, kemp, netscaler and a10 always, fortigate, paloalto and cisco when the target names its binding (for cisco, node_hostname with certificate_roles: a rollback moves the roles back to the predecessor's ISE system certificate), and GCP Certificate Manager when the target names a certificate_map_entry. AWS ACM and Azure Key Vault have no executable rollback because their APIs cannot address an installed object separately from uploading one; their support rows say so, the refusal names the reason, and no rollback-shaped success is recorded.

Confirm deployments from an external observer (A180)

trstctl's own agents and relays verify what a listener serves. An external observer — probectl, a synthetic monitor, a load balancer's probe — can confirm it independently from where clients stand. An operator registers it with POST /api/v1/observers (notifications:write): a name, an optional https events_url, and an optional secret:// reference whose value signs each event. The observer reports with a discovery:write token to POST /api/v1/observations: the endpoint (host:port), its vantage, the time, reachability, health, and the served leaf and chain fingerprints.

trstctl compares each report with what its newest endpoint verification expects at that address and records a verdict — match, mismatch, unreachable, or unexpected when nothing it verifies names the address. A mismatch raises an external-observation-mismatch finding; an unhealthy or unreachable endpoint within seven days after the renewal that produced the certificate it should serve raises endpoint-unhealthy-after-rotation. A match is independent evidence on the certificate's assurance record, listed apart from trstctl's own verification.

When a renewal completes or a connector rollback restores a certificate, the same transaction queues a rotation.completed or rollback.completed event for every enabled observer with an events URL: the certificate the endpoints should now serve, the one before it, the endpoints and the connector targets. The outbox posts it with X-Trstctl-Signature: sha256=<HMAC>; delivery is at least once. The full contract for an observer's side is in External observers. CLI: trstctl observers register|list|update, trstctl observations record|list.

Pitfalls & limits

• Serving status: the SDK and all 24 shipped connectors (initial + appliance) are constructed from connectors.enabled and wired into the served outbox path through server.Deps.ConnectorRegistry; signed WASM connector plugins are a second served path for third-party code. Tenant-scoped target CRUD, test, deploy, rollback, and identity binding are served; target mutation needs the matching native connector enabled or a signed plugin owner.
• Host availability: host-family deploys have no control-plane fallback. If no enrolled host-role agent with connector.deploy enabled and a usable host profile is polling, the row remains pending. Operations shows the queue; this is a safe refusal, not a request to place the target's filesystem on the control-plane host.
• Host target testing: connector.test is separately enabled. It checks target reachability and local authority without granting connector.deploy; a green test is a plan, not proof that a certificate was installed.
• Cloud target execution and testing: AWS ACM, Azure Key Vault, and GCP Certificate Manager are network-relay work: an enrolled network relay claims their deploys, previews and (for GCP) re-bind rollbacks and redeems the target's credential references for one attempt, and the control plane refuses the deploy while a relay is enrolled. A tenant with no network relay keeps the control plane's egress-guarded client as the executor, so previews then run on the bounded control-plane outbox worker with the exact saved target revision and a one-attempt secret lease for an authenticated, provider-specific read-only operation. Provider I/O never occurs in the request handler, and a preview never calls the connector's deploy method.
• Target evidence: selecting a target in Connectors shows one time-ordered deploy, listener-verification, and rollback timeline. Agent verification rows name the certificate identity that produced the signed observation; a delivery receipt alone does not claim that the listener is serving the new certificate.
• Grants are deny-by-default: an ungranted operation fails with ErrDenied — that's the safety net, not a bug, if a connector seems to do nothing.
• Appliance connectors need reachable management endpoints and import-only credentials.
• Appliance-generated keys: each renewal leaves its job-named key object (and, except on PAN-OS, its CSR) on the appliance, so predecessors stay available for rollback; nothing prunes them yet. The appliance operations are proven against the management-API doubles, not hardware.
• Idempotency is keyed on the fingerprint: redeploying the same certificate is a safe no-op, but a connector must converge to the same state.

Reference

• SDK: Connector{Name, Capabilities, Deploy}, Sandbox{WriteFile, Send, Exec, Request}, Registry, Conformance.
• Capabilities: fs.read, fs.write, net.dial, process.exec (path/host prefix-constrained).
• Initial connectors (F7): nginx, apache, caddy, haproxy, iis, traefik, envoy, postfix, aws-acm, azure-keyvault, gcp-certificate-manager, java-keystore, f5; first-party native targets also include postgresql, mysql, rabbitmq, elasticsearch, and tomcat, taking the shipped production connector catalog to 24 entries before signed third-party plugins.
• Appliance connectors (F27): netscaler, a10, kemp, cisco, fortigate, paloalto — covering F5/BIG-IP, Citrix ADC/NetScaler, A10 Thunder/AX, and Kemp LoadMaster, delivered through the same outbox plus native-registry/signed-plugin path as the initial set.
• Outbox destination: connector.deploy.
• Guide: Authoring a connector.

See also

Lifecycle & PQC (what triggers deployment) · Extensibility & plugins (the capability sandbox model) · Connector authoring guide · glossary: certificate, outbox, plugin / WASM sandbox

Covers: F7, F27