SSH — replace standing SSH keys with short-lived certificates

What it is

Most SSH access works by copying a user's public key into a server's authorized_keys file. That scales badly and ages dangerously: keys pile up, nobody remembers whose they are, and removing access means hunting them down across every host. An SSH certificate replaces that model: you trust one SSH certificate authority, which signs short-lived certificates saying "this user may log in as alice until 5 p.m." No per-host key copying, automatic expiry, central control.

This page covers trstctl's three SSH pieces: the CA that signs host and user certificates (F43), the agent that safely configures hosts to trust it (F44), and attestation-gated short-lived user certificates tying SSH access to verified identity (F45).

Why it exists

Standing SSH keys are one of the most common audit findings and breach vectors: orphaned keys grant access nobody tracks, and offboarding rarely removes every key. SSH certificates fix the structural problem: access expires on its own, trust is centralized in the CA, and you grant exactly the principals and time window each session needs. The hard parts are changing host trust without locking yourself out, and ensuring only the right identity can get a certificate — what F44 and F45 address.

How it works

The SSH certificate authority (F43)

trstctl's SSH CA signs two kinds of OpenSSH certificate: host certificates (so clients verify a server without trust-on-first-use prompts) and user certificates (so servers authorize logins without a stored key). Each certificate carries principals, a validity window, and optional critical options and extensions.

All signing goes through the single crypto path — one SignSSHCertificate operation taking an opaque signer handle — so the CA key lives in an HSM, held in the isolated signing service, never in the API process or in the clear. An issuance profile bounds the maximum TTL and allowed certificate types; serial numbers increment under a lock; every issuance is recorded as an immutable ssh.cert.issued event in its own bounded lane. The CA also maintains a key revocation list (KRL): revoke by serial or key ID, then distribute a snapshot to hosts, pulling back a certificate before it expires.

The operator workflow is served two ways: OpenSSH-compatible protocol endpoints (/ssh/ca, /ssh/issue/user, /ssh/issue/host, /ssh/krl) and a guarded product API used by the CLI and console. POST /api/v1/ssh/certificates/preview validates and normalizes an exact host or user request without allocating a serial, writing an event, changing the KRL, making a network call, or calling the signer. It shows the requested and effective TTL (default 1 hour, hard maximum 24 hours), deduplicated principals, public-key and authority fingerprints, applied options/extensions, the one future signer call, and the revocation path. POST /api/v1/ssh/certificates revalidates the same contract and issues exactly one certificate behind an Idempotency-Key; a stable retry returns the first result instead of signing again. The protocol routes POST /ssh/issue/user, POST /ssh/issue/host and POST /ssh/revoke run the same workflow under the same guard: they need an Idempotency-Key, ABAC deny rules and the per-tenant rate limit apply, issuance needs certs:issue, revocation needs certs:write, and errors come back as application/problem+json.

Direct user certificates may carry only the login names listed in protocols.ssh_user_principals (TRSTCTL_PROTOCOLS_SSH_USER_PRINCIPALS, comma-separated). Matching is exact, so root is allowed only if you list it. With the list empty, direct user certificates are refused with a message saying how to enable them. Host certificates and attested user certificates, whose principals are bound to verified evidence, are unaffected. The eval profile defaults the list to eval-user.

The direct user-certificate path allowlists only source-address and force-command critical options plus known OpenSSH session extensions. Host certificates reject all critical options and extensions. Both paths accept one public key and never accept or return its private key. Status and revocation remain available at GET /api/v1/ssh/status and POST /api/v1/ssh/certificates/revoke; revocation appends an immutable, tenant-scoped ssh.cert.revoked event before publishing the updated KRL snapshot. On every control-plane start, trstctl rebuilds the KRL from those events before serving SSH. A malformed matching event stops startup instead of publishing a partial or empty revocation list. Before answering GET /ssh/krl, each replica catches up with the event log (at most once every two seconds), so a revocation recorded on any replica is listed within about two seconds. If it cannot catch up, it answers 503 and the host keeps its last KRL.

SSH deployment & trust configuration (F44)

For a host to accept the CA's certificates, it must trust the CA's public key, written into TrustedUserCAKeys and referenced from sshd_config. Editing sshd_config on a live fleet is exactly where people lock themselves out, so trstctl's agent follows a hard rule: additive-only, validated before it takes effect, and rolled back automatically on any failure.

The agent saves the current file contents in memory, adds a missing CA line once, and writes changes atomically (write-temp-then-rename). It then validates (sshd -t), reloads, and health-checks that sshd still accepts connections. An apply failure restores the saved contents and reloads the prior config; reload and health commands are operator-supplied, required, and run as validated argv lines with shell metacharacters rejected (--ssh-trust-reload-cmd, --ssh-trust-health-cmd) — reload success alone isn't proof of health. Removing trust is never implicit: RemoveCATrust needs an explicit confirmation flag. A failed restoration leaves an unclear host state that needs operator attention. The library emits ssh.trust.added, ssh.trust.removed, ssh.trust.rolled_back, and ssh.trust.rollback_failed when an audit sink is connected. The current one-shot agent command does not connect that sink or deliver these events to the control plane; record its observed result through the rollout handoff below. That record is an operator assertion, not automatic proof of the host change.

When both files already contain the requested trust, rerunning the command leaves their contents unchanged but repeats validation, reload, and the health check. A previous process may have stopped after writing files and before reloading SSH. A failed retry reports its failed stage and leaves the files untouched; it cannot restore an earlier process's memory-only backup. Keep an independent recovery session and backup until both existing and certificate-based access have been verified. Durable crash rollback and automatic audit delivery are not provided by this one-shot command.

The one-shot requires --ssh-trust-ca-key and --ssh-trust-tenant; enrollment flags do not enroll this operation. Set --ssh-trust-keys-file to the first global TrustedUserCAKeys path, including directives loaded through Include. A conflicting path or a Match block before global CA trust is refused before any file changes. See the complete rollout example.

The control plane also has a served handoff for this high-blast-radius path: POST /api/v1/ssh/trust-rollouts records the source, target hosts, CA fingerprint, reload/health commands, rollback plan, status, and an explicit confirmed=true acknowledgment; POST /api/v1/ssh/hosts/retire records retirement evidence once migration completes. This endpoint remains a manual evidence record; it cannot dispatch jobs or create verified fleet receipts.

Approved fleet trust changes (A226)

The Fleet SSH trust changes panel in SSH access → Trust rollout previews an exact roster, requests independent approval, starts the approved change, and shows each signed host receipt. The matching API is /api/v1/ssh/trust-fleet/{preview,approvals,rollouts}; the CLI is trstctl-cli ssh trust-fleet {preview,request-approval,start,list,show}.

Each enrolled host must separately enable --ssh-trust-fleet and --ssh-trust-confirm, stage the selected public CA, and choose its local file paths and validation, reload, and health commands. The remote job carries none of those commands or paths and cannot turn local confirmation on. Preview keeps opted-out, stale, missing, or misconfigured hosts visible as skipped. Approval binds the exact public key, host policies, roster, and deadline; a changed ready host requires another preview and approval. The default quorum is two distinct reviewers, with no self-approval.

A dedicated one-worker host loop executes ssh.trust.apply. It serializes local SSH changes with KRL installation, adds trust, validates, reloads, and checks health, then independently reads the CA and effective global trust path back. The signed receipt binds the enrolled host, job attempt, plan, policy, hashes, and observation time. A queued job is pending. A checked restoration after a failure is attention_required, never successful rollout. Served completion also requires the matching immutable source event; changing a database row cannot manufacture a verified effect.

Local backups are memory-only. Process or machine loss can interrupt recovery; keep an independent access path and backup. External administrators must not edit these files concurrently. The health command defines the runtime proof: configure a pinned, real login check for the access you need. Receipts describe the observation time, not perpetual host health. The fleet path adds user-CA trust only; host-certificate installation, trust removal, and authorized_keys pruning need their own explicit workflow. See the fleet trust runbook for setup, approvals, receipt recovery, and the exact proof limits.

Hosts under this SSH CA (A115)

Key inventory alone cannot say whether a host is under the CA: a raw public key can be the key inside a CA-signed host certificate. The host's sshd configuration can. Run the enrolled host agent with --inventory-ssh-sshd-configs /etc/ssh/sshd_config and it reports, for each configuration, the fingerprints of the CAs in its TrustedUserCAKeys file, its RevokedKeys list (KRL version and digest, a plain key list, or a missing file), each HostCertificate it presents with the signing CA, key ID, principals and validity (the CA signature is verified on the host), and AuthorizedKeysFile. GET /api/v1/ssh/fleet (trstctl ssh fleet) judges every reporting host under host_trust, and SSH access shows it as Hosts under this SSH CA:

User certificates: this CA only, this CA and others, another CA, or none.
Host certificate: this CA (valid), renew soon (under a third of its validity left), expired, not yet valid, another CA, a signature that does not verify, a user certificate configured as a host certificate, or none.
Revocation list: the current list (its digest matches what /ssh/krl serves), another list, a plain key list, a configured file that is missing (sshd then refuses every key), or not enforced.
authorized_keys: off when AuthorizedKeysFile none leaves certificates as the only way in; on otherwise.

The agent reads the global section the way sshd resolves it (first value wins, HostCertificate accumulates, Include files in place) rather than running sshd -T, so Match blocks are counted and flagged, not applied. The posture is reported when the agent connects: restart it after changing sshd_config. The control plane names each host from the agent's certificate, never from the report.

Attestation-gated short-lived user certificates (F45)

The most powerful pattern: issue an SSH user certificate only to a caller who proves identity first. This issuer runs an attestation check (the same chain used for workload identity), then derives principals from the verified attestation and calls the SSH CA. The authenticated certs:issue caller is recorded as the issuer and must differ from the attested subject. This request does not collect a second person's approval. It rejects unbound principals, supports OpenSSH source-address and force-command critical options, fails closed on attestation failure, defaults to a 15-minute TTL (capped by the profile), and binds the attestation via an immutable ssh.attested_cert.issued event: access short-lived and provably tied to a specific CI job or cloud instance — no standing keys.

Review the exact request first with POST /api/v1/ssh/attested-user-certs/preview or trstctl ssh preview-attested-user. The preview reads tenant trust, validates and normalizes the public key, authenticated issuer, principals, lifetime, source addresses, and forced command, then returns proof and key fingerprints, the isolated-signer action, and recovery instructions. It performs no proof verification, write, external call, audit emission, or signer call; proof verification remains execution-only because a proof may contain one-time evidence.

Execution is served at POST /api/v1/ssh/attested-user-certs and by trstctl ssh issue-attested-user; the request carries an attestation method, base64 payload, SSH public key, optional key ID, principals, TTL, source-address allowlist, and force-command policy. The response is the certificate plus serial, key ID, expiry, constraints, and the attestation record — the private key never crosses the API or UI. The console retains one request-scoped idempotency key after an uncertain response; its Retry unchanged request action recovers the original result instead of signing again. For CLI recovery, set TRSTCTL_IDEMPOTENCY_KEY to a stable value and reuse the same command body. A changed request with that key is rejected with HTTP 409.

Agentless trust collection

Host-agent inventory remains the default. If a machine cannot run an agent, create an explicitly enabled Agentless SSH trust source in Discover. Select an enrolled network relay, a declared scope, and up to 100 exact host:port bindings. Each binding needs a non-root account, a SHA256 server host-key pin obtained through a trusted administrative channel, and a secret:// reference to a supported PKCS#8 RSA/ECDSA key. A raw host-key pin continues to bind that same key when the server presents it in a host certificate; this does not imply that the login client validated the certificate's CA or principals.

Install the repository's deploy/ssh/trstctl-ssh-collect-v1 as root at /usr/local/libexec/trstctl-ssh-collect-v1 (mode 0755). It requires Python 3 and /usr/sbin/sshd. Install a root-owned, group/other-unwritable policy at /etc/trstctl/ssh-collection-v1.json:

{
  "config_path": "/etc/ssh/sshd_config",
  "allowed_public_files": [
    "/etc/ssh/trusted_user_ca_keys",
    "/etc/ssh/revoked.krl",
    "/etc/ssh/ssh_host_ed25519_key-cert.pub"
  ]
}

The policy permits only public files already named by the five allowlisted sshd -T settings. It never returns private HostKey bytes and cannot request arbitrary files or commands. The sshd -T process may internally validate the configured host keys; only the allowlisted public profile leaves the machine. A denied, missing, relative, oversized or invalid public file has an explicit coverage state. The helper bounds time, file count and bytes, suppresses stderr, strips public-key comments and refuses private-key material. It does not follow final-component symlinks. Configure the actual canonical public paths.

Use a dedicated account, for example trstctl_observe, with no writable policy, helper, configuration, public trust files, authorized-key file or parent directory. Store its public authentication key in an administrator-owned authorized-key file. Restrict that account with Match User trstctl_observe and a forced command /usr/bin/sudo -n /usr/local/libexec/trstctl-ssh-collect-v1; disable forwarding, PTY, X11, tunnels, user environment changes and user rc scripts (PermitUserRC no). If root access is required for sshd -T, sudoers must permit only the exact no-argument helper:

trstctl_observe ALL=(root) NOPASSWD: /usr/local/libexec/trstctl-ssh-collect-v1 ""

The OpenSSH daemon manual describes the -T/-C distinction and forced-command restrictions. Validate any SSH configuration change with sshd -t, retain an independent administrative session and follow the host's tested rollback procedure. Never replace an existing trust policy or broaden general login access for collection. The helper is a one-shot read, not a host agent or a remote shell. A source cannot choose its command or paths. Repeated requests run only this fixed profile.

Enable ssh.trust.collect in agent.claimable_job_kinds and run the selected network relay with --relay-claim. Its local egress policy must name the actual SSH destinations. Private RFC1918 targets need explicit source permission as well as declared scope; loopback is a separate owned-test opt-in. Metadata, link-local, multicast and other reserved destinations stay blocked at dial time, including after DNS resolution. The relay never falls back to an unpinned host or a different credential.

Preview/preflight only validates the plan; it makes no SSH connection and queues no job. Save it, then use the source's Run action, the existing trstctl discovery plans preview, sources create, sources preflight, runs start and runs get commands, or the discovery source/run API. The immutable command and same-transaction outbox bind the exact source, relay, targets and reference names. Redemption needs that relay's authenticated live lease and happens once per attempt. Keys are encrypted at rest and over mTLS, borrowed into locked byte buffers, and erased after the bounded attempt; they necessarily exist in the SSH client's memory during authentication. They never enter source configuration, events, job payloads, logs or public reports.

The relay uses eight workers, 20 seconds per SSH attempt and a 90-second total read window while maintaining its lease. Every requested target gets an outcome; queued targets not reached before cancellation remain failed, never silently omitted. A signed report must match its command and include the exact public readback for observed targets. The 1 MiB job-report bound can make a large target report_too_large; reduce the source's host set or inspect that host locally. Connection/authentication, helper, permission and pin failures require correcting the prerequisite and explicitly rerunning; a failed setup attempt is terminal.

SSH access shows the remote address and the collecting relay separately, with agentless provenance. It checks the retained immutable observation before serving those facts; a changed or missing projection proof is unavailable until source history is restored/replayed. Ordinary host inventory cannot declare agentless provenance. The observation is the on-disk configuration as sshd -T resolves it, including global defaults/Includes. It does not prove what a running daemon has loaded, evaluate connection-specific Match contexts, inventory per-user keys or modify trust/reload SSH. A failed public-file read stays unknown, even when the collection job successfully records that failure. Host-agent inventory continues through its existing signed report path.

Use it

Stand up the SSH CA, distribute its public key to hosts via the agent, then issue short-lived user certificates. The CA's public key goes into a host's trust config like this (what the agent writes, additively):

For a container deployment, bind the served SSH workflow and the shared attestation mint to one tenant. This exposes the workflow; it does not invent a trusted identity source:

TRSTCTL_PROTOCOLS_SSH_ENABLED=true
TRSTCTL_PROTOCOLS_SSH_TENANT_ID=11111111-1111-4111-8111-111111111111
TRSTCTL_PROTOCOLS_SSH_USER_PRINCIPALS=deploy,oncall
TRSTCTL_ATTESTED_ISSUANCE_ENABLED=true
TRSTCTL_ATTESTED_ISSUANCE_TRUST_DOMAIN=example.org
TRSTCTL_ATTESTED_ISSUANCE_DEFAULT_TTL=10m
TRSTCTL_ATTESTED_ISSUANCE_MAX_TTL=1h

Add an enabled, tenant-scoped public trust source through /api/v1/workloads/attester-trust-sources. The same source gates both workload SVIDs and attested SSH user certificates. If no matching source exists, both mints fail closed; the console does not show a fake list of available attestors.

# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/trusted_user_ca_keys

# /etc/ssh/trusted_user_ca_keys  (the CA public key in authorized_keys form)
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5... trstctl-ssh-ca

A user certificate is then issued with an attestation-bound principal and a short TTL (e.g. 15 minutes); the user connects normally and sshd validates the certificate against the trusted CA without any stored key.

trstctl ssh status
trstctl ssh preview \
  --type host \
  --public-key "$(cat /etc/ssh/ssh_host_ed25519_key.pub)" \
  --key-id edge-1.internal \
  --principals edge-1.internal,edge-1 \
  --ttl-seconds 86400
trstctl ssh issue \
  --type host \
  --public-key "$(cat /etc/ssh/ssh_host_ed25519_key.pub)" \
  --key-id edge-1.internal \
  --principals edge-1.internal,edge-1 \
  --ttl-seconds 86400
trstctl ssh trust-rollout \
  --hosts edge-1.internal \
  --ca-fingerprint SHA256:... \
  --reload-cmd 'systemctl reload sshd' \
  --health-cmd 'ssh -o BatchMode=yes localhost true' \
  --rollback-plan 'restore backup, reload sshd' \
  --status health_passed \
  --confirm
umask 077
python3 - <<'PY'
import base64, json, pathlib
proof = pathlib.Path('/var/run/trstctl-proof/token').read_bytes().strip()
public_key = (pathlib.Path.home() / '.ssh/id_ed25519.pub').read_text().strip()
request = {
    'method': 'k8s_sat',
    'payload_base64': base64.b64encode(proof).decode('ascii'),
    'public_key': public_key,
    'principals': ['web'],
    'source_addresses': ['10.0.0.0/24'],
    'force_command': '/usr/local/bin/deploy',
    'ttl_seconds': 900,
}
with pathlib.Path('ssh-attested-user.json').open('x') as out:
    json.dump(request, out)
PY
trstctl ssh preview-attested-user -f ssh-attested-user.json
trstctl ssh issue-attested-user -f ssh-attested-user.json
trstctl ssh revoke --serial 42 --reason 'revoked'
trstctl ssh retire-host --host edge-1.internal --reason 'replaced'

Run the attested example where the projected token and requester key are available. The CLI refuses a group- or world-readable request file on Unix, and -f keeps the proof out of process arguments. The approver response field names the authenticated issuer; callers cannot name somebody else. Use a separate approval workflow if your policy requires two people.

Pitfalls & limits

Never hand-edit trust on a live host. Use the agent so the validate-reload-health-check-rollback safety net applies — a bad manual sshd_config edit can lock you out, and trstctl won't remove existing trust without explicit confirmation.
Serving status: the SSH CA is served by the running control plane (protocols.ssh.enabled, default off): cert issuance at /ssh/..., the OpenSSH binary KRL at /ssh/krl (sshd's RevokedKeys consumes it), and workflow API/CLI coverage for effect-free direct issuance preview, idempotent host/user issuance, status, trust rollout evidence, attested user cert issue, KRL revocation, and host retirement. The CA key stays in the isolated signing service, never the API process, with every step recorded as an immutable event and tenant data isolated at the database layer. SSH host-key discovery is also served via ssh discovery sources on the outbox worker; privileged trust rewrites still need the explicit agent-safe rollout workflow — see Current limitations.
Short TTLs require renewal. That's the security benefit, but plan the renewal path for long-running sessions.
KRL enforcement requires host delivery. Publishing a revocation updates /ssh/krl. An opted-in host agent can poll that endpoint with --ssh-krl-watch, check the served tenant over pinned HTTPS, and install a newer version at an already configured RevokedKeys path. The explicit --ssh-krl-apply operation handles a single, digest-pinned artifact. Both paths validate the KRL with OpenSSH, check sshd's effective path and config, retain the predecessor, and run a known-good login probe; a failed update restores the old file. The watcher skips reload on an unchanged list. Configure it on every relying host; the control plane does not push lists. A watcher run beside an enrolled host-role agent signs a receipt onto each heartbeat: the digest and version read back from its RevokedKeys file and how its latest attempt ended, which the control plane checks against the agent's certificate before recording. GET /api/v1/ssh/krl/receipts (trstctl ssh krl receipts) places each reporting host against the served list as current, behind, refusing (with what the served list would drop), failing, or stale; SSH access shows the same view under Revocation list on hosts. Existing SSH sessions are not terminated by a KRL update. A host keeps its installed KRL if the control plane is unavailable or cannot finish a verified event catch-up; it retries the download. A restored or replacement control plane counts its own KRL versions, so the agent compares what the lists revoke as well as their versions: it refuses a list whose version does not advance the installed one (except that a list revoking nothing, such as the empty one ssh-keygen -k writes when a host is prepared, yields to a list of the same version, so a new host reads current before the first revocation), and it refuses any list, however new, that would stop revoking a serial, key ID, key, or key hash the installed list revokes (a revocation for any CA keeps one scoped to a single CA). The installed list stays in force and the refusal counts what the served list would drop. To move hosts to a restored or replacement control plane, adopt the list they hold there: POST /api/v1/ssh/krl/adoptions (trstctl ssh krl adopt -f old.krl) takes the predecessor's binary KRL as served or as a host installed it, records every revocation it holds (serials and serial ranges, key IDs, revoked keys and key hashes; revocations the predecessor scoped to one CA are carried for any CA, and the response counts them), and moves this list's version past the predecessor's. Hosts take the new list on their next poll. Adopting a list already carried at a newer version records nothing. One adoption reads at most 4 MiB and 100,000 entries, and refuses a list with signatures, a revoked entry that is not an SSH key, or a key ID OpenSSH and JSON cannot both carry exactly.

Reference

CA operations: IssueUserCert, IssueHostCert, AuthorityKey (for TrustedUserCAKeys / @cert-authority), KRL.RevokeSerial, KRL.Distribute.
Served API/CLI: POST /api/v1/ssh/certificates/preview, POST /api/v1/ssh/certificates, GET /api/v1/ssh/status, POST /api/v1/ssh/trust-rollouts, POST /api/v1/ssh/attested-user-certs/preview, POST /api/v1/ssh/attested-user-certs, POST /api/v1/ssh/certificates/revoke, POST /api/v1/ssh/krl/adoptions, GET /api/v1/ssh/krl/receipts, POST /api/v1/ssh/hosts/retire; trstctl ssh preview|issue|status|trust-rollout|preview-attested-user|issue-attested-user|revoke|krl adopt|krl receipts|retire-host.
Agent config: SSHDConfigPath, TrustedUserCAKeysPath, AllowUnconfirmedRemoval (default false).
Attested issuance: AttestedUserCertIssuer.Issue (method+payload → cert).
Configuration: protocols.ssh.enabled, protocols.ssh.tenant_id, protocols.ssh_user_principals.
Events: ssh.cert.issued (key ID, serial, principal names, validity window and the calling actor), ssh.cert.revoked, ssh.krl.adopted (one part of an adopted predecessor list), ssh.attested_cert.issued, ssh.trust.added, ssh.trust.removed, ssh.trust.rolled_back, ssh.trust.rollback_failed.
Standard: OpenSSH certificate format (PROTOCOL.certkeys).
Design deep-dive: SSH trust-rewrite design.