Incident response & just-in-time access — contain compromise, gate access

What it is

Sometimes a credential leaks, a CA is compromised, or someone needs emergency access right now. This page covers the four workflows trstctl provides for those moments: a credential-compromise workflow that re-issues and revokes a leaked credential and everything downstream of it; fleet re-issuance to replace every certificate from a compromised CA; just-in-time (JIT) issuance that grants access only after approval; JIT privileged access sessions for Postgres and SSH; and break-glass emergency signing for when the control plane itself is down.

The mental model: this is the fire department and the keymaster combined. The compromise workflow and fleet re-issuance are the fire response (contain and rebuild without leaving anyone locked out); JIT is the keymaster who only hands out a key after a second person signs off; break-glass is the sealed emergency key behind glass that takes two officers to use.

Why it exists

The worst time to invent a process is during an incident. A leaked key needs to be replaced and every credential that depends on it rotated, in the right order, without creating an outage. A compromised CA can mean thousands of certificates to re-issue — impossible by hand, dangerous without health checks. And standing, always-on access is itself a risk: JIT replaces "everyone has access all the time" with "access is granted, approved, and expires." Each workflow is built to be safe under pressure: ordered, audited, idempotent, and reversible.

How it works

Credential compromise workflow (F31)

When one credential identity is compromised, the danger is everything it can reach. The served workflow starts with a read-only blast-radius snapshot from the graph, then executes the containment path idempotently — every state-changing request takes an Idempotency-Key, so a retry never applies the change twice: it creates a replacement identity, issues it, deploys it through the connector outbox, and only then revokes the compromised identity. The result is recorded as an immutable incident.execution.recorded evidence pack — with the replacement id, revocation queue status, connector delivery receipt, failed-target list, rollback references, and a sealed audit bundle — and its outbound deliveries are journaled first so a crash can't drop them.

This is deliberately stricter than probectl's guarded-remediation pattern. probectl proposes and records remediation; trstctl's remediation actually executes issue, deploy, and revoke work after a human operator trigger. The served routes are part of Core and require no commercial license. All deployments require RBAC (incidents:write and certs:issue for replacement issuance), policy approval where configured, and idempotency before anything mutates.

The same incident surface can open a ServiceNow / ITSM workflow after the operator has enough evidence. POST /api/v1/itsm/servicenow/tickets records an immutable itsm.ticket.requested event, then enqueues itsm.servicenow in the outbox in the same tenant-scoped transaction. The outbox worker resolves token_ref only when it is ready to call the ServiceNow Table API; the token value is never stored in the event log, outbox payload, UI, or audit response. The request is idempotent like every other mutation: replaying the same Idempotency-Key returns the same queued receipt instead of creating a second ticket request. The instance_url, token_ref, and allow_private_endpoint values must match an operator-approved ServiceNow binding in configuration; the route fails closed rather than sending a configured token to a caller-chosen URL. Private ServiceNow bindings also require private_egress_cidrs in operator configuration and the caller must hold the dedicated egress:private permission.

Detection is served on the Discovery side, not hidden inside remediation. A credential_compromise source accepts metadata-only ITDR, honeytoken, secret-scanner, IdP, SaaS, or threat-intel signals and emits compromised_credential findings tagged CAP-ITDR-02 and OWASP NHI2. Those findings carry credential references and evidence refs only; the raw stolen token or secret body never enters the control plane. Operators can use those findings as the evidence that justifies the incident execution path above.

OAuth consent abuse is served through the same Discovery spine. An oauth_grant source still emits ordinary oauth_grant inventory findings for CAP-OAUTH-01, and it emits an additional oauth_grant_abuse finding tagged CAP-ITDR-03 only when the grant export has concrete abuse evidence such as provider threat signals, dangerous wildcard / .default scopes, offline_access plus high-privilege admin consent, explicitly unverified publisher evidence, ownerless privileged third-party admin consent, or suspicious redirect URIs. The finding stores evidence references and source event ids, not OAuth client secrets, access tokens, or refresh tokens.

Supported tables are incident, change_request, and sc_task. Production endpoints should be HTTPS; allow_private_endpoint is accepted only when the configured binding explicitly permits that private/eval endpoint and grants the destination CIDR.

For a broader response packet, POST /api/v1/incidents/response-integrations/dispatch records response.integration.dispatched and fans out one tenant-scoped outbox row per destination: Splunk HEC (response.splunk), Jira issue creation (response.jira), Slack/operator notification (notification.response), and ServiceNow Table API (itsm.servicenow). Each destination carries only endpoint metadata plus token_ref; token bytes are resolved by the worker at delivery time and wiped after the outbound request. ServiceNow destinations reuse the same operator-approved binding check as direct ITSM ticket creation. Splunk and Jira token_ref values must be listed by the operator in TRSTCTL_OUTBOUND_ENV_CREDENTIAL_REFS; their private destinations additionally require both egress:private and a destination CIDR grant on the dispatch request. This is a served dispatch and evidence path, not a promise that trstctl manages the customer's Splunk correlation searches, Jira automation rules, Slack app installation, or bidirectional ServiceNow state sync.

Fleet re-issuance for CA compromise (F32)

If an X.509 CA is compromised, every active identity whose current certificate is signed by that issuer must be replaced, and every exact trust consumer must be visible before the first estate effect. Historical identity-to-issuer links do not expand the current blast radius after a completed migration. trstctl freezes the compromised issuer, signer-backed replacement authority, exact certificate/SPKI TRUSTS stores and hosts, ordered cohorts, exact enrolled agents, deployment-target revisions, predecessor certificate/revocation authority, and rollback paths into one H2 plan. Subject-only trust matches remain visible as candidates but never authorize work. The plan's digest and H3 projection commit before H2 can enqueue its first agent action.

Each cohort installs the replacement authority, waits for the host agent's lease-bound signed trust readback, mints a successor from a host-generated CSR, deploys it, and waits for the signed live-listener transcript. Only that proof releases revocation of that member's exact predecessor; every revocation receipt must land before the next cohort starts. A failed trust/live gate automatically restores only the current unrevoked cohort and removes its replacement trust. Completed earlier cohorts are not rolled back after their predecessors have been revoked. Every action is idempotent, journaled in the same transaction as the H2 event projection, and executed in the bounded fleet lane. The predecessor revocation gate publishes the signed CRL from that predecessor's exact issuing authority before it reports success. After a crash between the revocation event and CRL publication, the next startup or freshness sweep detects the newer revoked serial and republishes the CRL; operators should still verify the signed CRL or OCSP answer with a stock client.

Three run-level health gates are computed on every read rather than asserted, and a gate an operator asserted is never overwritten. Replacement deployment comes from verification receipts (one failed replacement fails it; one unprobed replacement keeps it unevaluated). Graph enumeration re-reads the live store as its completeness oracle: an identity that still holds an active certificate signed by the compromised CA but is not in the run fails the gate and is named in graph_enumeration_missing_identity_ids. Revocation publication narrows the revocation-health observations to the compromised issuer: a stale or unreachable CRL/OCSP endpoint fails it, a fresh artifact older than the run's last change keeps it unevaluated, and a fresh or expiring observation after that change passes it; revocation_publication carries the counts, and no observation means no verdict.

Status: compromised-issuer fleet re-issuance is served through POST /api/v1/incidents/fleet-reissuance-runs, with list/get evidence at GET /api/v1/incidents/fleet-reissuance-runs{,/{id}}, pause/resume/rollback evidence at POST /api/v1/incidents/fleet-reissuance-runs/{id}/{pause,resume,rollback}, and a signed evidence export at GET /api/v1/incidents/fleet-reissuance-runs/{id}/evidence. Start accepts issuer_id, replacement_authority_id, mode, reason, rollback_ref, and ordered cohorts; each member supplies identity_id, the exact enrolled agent_id, and a public trust_anchor_path. live is the production response. game_day additionally requires incidents:game-day, and H2 refuses any member whose frozen owner/target environment is not explicitly non-production. It is therefore a safety boundary, not a display label. non_production is accepted as an explicit non-production environment spelling for both the accountable owner and target; production still fails the game-day gate. Preview and Start verify that each current active predecessor was cryptographically signed by the named compromised issuer's exact CA certificate, not merely another authority beneath a shared parent. After a successful replacement, the historical identity-to-issuer graph edge may remain for audit, but it cannot authorize a second migration of the replacement certificate; the repeated plan returns a non-ready preview and Start returns conflict without effects. Terminal success or rollback is mirrored to incident.fleet_reissuance.recorded only with a compact-JWS audit export that verifies offline. Pause/resume and restart retain the same H2 cursor and immutable bindings. The legacy POST /api/v1/incidents/executions mutation now returns conflict with this H2 route as guidance; historical execution reads remain available. CLI parity is trstctl-cli incidents fleet-reissuance preview|start|list|get|pause|resume|rollback|evidence. Incident execution and fleet run lists return newest-created records first. Their opaque cursors preserve that order across pages even when an older run changes status. The console's situation summary compares the latest records from both lists before offering recovery guidance. In the console's Fleet & break-glass run table, Export downloads trstctl-fleet-evidence-<run-id>.json with the compact signed JWS, rollback references, failed targets, run ID, and export time. The page also shows the export result; operators can verify the saved JWS offline and retain the file with their incident record.

POST /api/v1/incidents/fleet-reissuance-runs/preview accepts the exact Start body without an idempotency key. It checks the current issuer, exact certificate/SPKI trust-store and host relationship, affected identities, cohort bindings, and game-day authorization and environment. It returns ready, named blockers, and the observed exact/candidate scope. It writes no event, queues no work, and calls no signer or external target. Run it before Start; Start repeats the checks because a preview cannot reserve an estate that may change between calls.

The /incidents console configures the same contract as a three-step journey; it does not ask an operator to paste UUIDs or hand-write cohort JSON. First choose a served X.509 issuer and active replacement authority, then build ordered waves from the affected identity and enrolled-agent rosters. The final step names the exact issuer, authority, mode, wave order, and identity-to-agent assignments before start. An empty issuer, authority, agent, or affected-identity roster blocks progress with a named prerequisite instead of guessing. The advanced exact-request disclosure remains available for audit and API comparison, while the primary path keeps the immutable-plan, canary-first, signed-gate, and predecessor-after-proof safety order visible in plain language. The review step calls the served preview and disables Start while it is blocked or unavailable. Clicking Start rechecks the preview before submitting the mutation.

Just-in-time issuance with approval (F33)

F33 has three related lanes, and the distinction matters. The ordinary self-service certificate lane opens one seven-day request and requires one different principal with certs:issue to approve it. The attested ephemeral lane is dual-control by default (2 required, configurable for m-of-n), has a shorter approval window, and queues its approver notification through the transactional outbox. The PAM lane opens a short-lived database or SSH session instead of minting a general-purpose certificate. All three lanes block self-approval. PAM requires a configurable quorum of distinct reviewers with access:approve before a target grant or SSH signature can start.

On /request, the requester first reviews an exact, effect-free server preview bound to the tenant owner, active profile version, requester, subject, and optional requester-held CSR. Preview writes nothing, calls no signer or CA, and explains that submission opens a request only. On /approvals, a different principal can review the subject, purpose, owner, profile, decision expiry, immutable request ID, and audit link in one dialog. Approve and deny take an Idempotency-Key; denial requires a reason and is terminal. Approval still mints nothing: it reveals the separate prepare/issue/complete step. If that step fails, the request remains approved and retries the same deterministic identity and issuance key instead of minting a duplicate.

The generic operation-review surface keeps those domains separate: certificate review requires certs:issue, secret review requires secrets:write, and managed-key review requires keys:approve, and PAM review requires access:approve. List responses contain only the caller's authorized domains and paginate with an opaque composite cursor. Approve and deny re-check the request's exact kind, action, resource, and intent digest while the request row is locked. A denial closes only the immutable request; it does not mutate the requested resource.

For privileged-access management, the same JIT model opens short-lived sessions instead of standing database or shell access. A requester sends the exact target, role, reason, attestation proof, TTL, and SSH public key when applicable to POST /api/v1/access/session-requests, with a UUID request_id and an Idempotency-Key. The response returns the immutable approval_request_id and intent_digest, but no credential. Distinct custodians find it through GET /api/v1/approval-requests and approve that digest with POST /api/v1/approval-requests/{id}/approvals; the requester cannot approve it. The original requester can read the current non-secret count and status at GET /api/v1/access/session-requests/{approval_request_id} with access:write. Other requesters receive 404, even in the same tenant. The console refreshes this exact status while its review dialog is open and lets the operator refresh it manually. It asks for custodians only while review is pending; approved review prompts activation, and denied, expired, superseded, or consumed review explains that the operator must close the dialog and inspect or submit a new request. This read never creates a credential or grants reviewer access. The requester then sends the same fields and both returned approval identifiers to POST /api/v1/access/sessions. Changing a field or the configured target invalidates the review. A PostgreSQL provider reattachment after a control-plane restart also changes its configuration revision; request a new review if activation did not finish before that restart. The activation consumes approval once before granting a scoped Postgres login role or signing an OpenSSH user certificate, returns the one-time credential, and records a tenant-scoped session row. A configured PostgreSQL target references a tenant-bound dynamic-secret provider, whose administrator DSN is loaded from a 0600 file: or tenant secret:// reference. Issuance and removal run through its durable outbox; the PAM expiry worker records completion only after provider removal completes. PostgreSQL also enforces the credential deadline with rolvaliduntil if the worker is unavailable. SSH access ends at the certificate valid_before time. The event trail is filterable by pam.session.activation_requested, pam.session.started, and pam.session.expired; credential material is not written into those events. For early incident containment, POST /api/v1/access/sessions/{id}/revoke requires access:write, an Idempotency-Key, and a nonempty reason. It first records pam.session.revocation_requested with the authenticated actor and moves the session to revoking. The worker marks it revoked only after the PostgreSQL provider confirms role removal, or after the tenant's served SSH KRL includes the exact PAM certificate serial and key ID. The latter confirms publication, not that every host downloaded the KRL: sshd must use RevokedKeys with the current GET /ssh/krl artifact. The console shows these two states and polls readback. pam.session.revoked records completion. Native expiry remains an independent deadline; a stopped worker cannot extend a PostgreSQL role beyond rolvaliduntil or an SSH certificate beyond valid_before. If PostgreSQL provider removal exhausts delivery retries, the session becomes revocation_failed with a closed, non-secret error class and failure time. This does not certify target removal. Repair the target, then submit the same POST /api/v1/access/sessions/{id}/revoke operation with a fresh Idempotency-Key and a new reason, or use Retry target removal in the console. The retry records another pam.session.revocation_requested event and queues a new provider command. Its result returns to revoking until native role removal is confirmed. Exact-key replays return the original attempt; they do not create another removal. pam.session.revocation_failed records each terminal attempt without retaining provider response text. For new sessions, the start event also records the verified attestation method, subject, selectors, claims, and verification time. GET and list responses project those facts after a restart without returning the one-time credential. Sessions created before this evidence field was introduced omit attestation on readback; their old start events retained an ID and audit summary but not the verified selectors, claims, or verification time. The UI identifies that evidence gap instead of displaying a zero verification timestamp. An exact Idempotency-Key retry returns the original credential while its protected HTTP result is retained. Once that cache entry has aged out, the durable session identity rejects the old key instead of signing another SSH certificate; a new access request needs a new key. For SSH, the approved activation also records a metadata-only signing intent before calling the isolated signer. If the control plane dies after signing but before pam.session.started, the recovery worker takes a per-session PostgreSQL lock across replicas, publishes pam.session.ssh_signing_recovery_requested, and revokes the deterministic pam:<session-id> key ID in the served KRL. It records pam.session.ssh_signing_recovered only after that KRL contains the key ID. An unavailable KRL leaves recovery pending and retries; it is not reported as complete. The old approval stays consumed, and a retry cannot sign another certificate. The operator should inspect those audit events, ensure each SSH host has fetched the current KRL, and submit a fresh request for access. The one-time certificate never enters the event log or read model.

Status: the core identity approval gate is served through POST /api/v1/identities/{id}/approvals. The self-service certificate portal is served through /request plus /approvals and the /api/v1/issuance-requests{,/preview,/{id}/{approve,deny,cancel,prepare,complete}} family. It previews and submits a profile-bound x509_certificate request, denies requester self-issue, blocks RA self-approval of privileged issue, rotate, or revoke actions, accepts a distinct approval, then mints through the signer-backed issuance outbox and records certificate inventory evidence. The direct review dialog and the history panel are two views of the same event-projected request; failed issuance stays visible with an explicit safe-retry action. Ephemeral/JIT credential issuance is served when configured through POST /api/v1/ephemeral plus POST /api/v1/ephemeral/{id}/approvals, where {id} is the genuine approval_request_id and the body carries the same UUID as request_id plus its matching intent_digest; PAM sessions are served through POST /api/v1/access/session-requests, GET /api/v1/access/session-requests/{id}, POST /api/v1/access/sessions, GET /api/v1/access/sessions, GET /api/v1/access/sessions/{id}, and POST /api/v1/access/sessions/{id}/revoke when pam.enabled is true. A registrar adds targets through the console or POST /api/v1/access/targets, reads them with GET /api/v1/access/targets, and disables tenant targets with POST /api/v1/access/targets/{target_type}/{id}/disable. These mutations require access:targets.write; access:write does not grant it. The broker resolves attestors from the existing tenant-managed workload trust-source API at request time and refuses a method with no enabled tenant trust. Operator target configuration and tenant registration bind each target to one tenant. PostgreSQL targets require an existing tenant PostgreSQL dynamic-secret provider and an explicit role allowlist using readonly or writer; an unlisted role is rejected before a credential is created. SSH targets require a host, port, and principal list; only the user role is valid, and an empty principal list grants nobody. An SSH PAM target must belong to the tenant of the served SSH protocol and KRL; the broker refuses to start with an unmatched target. The ephemeral path verifies the attestation first, writes the approval request and outbox notification intent in the same tenant transaction, blocks requester self-approval, then mints a short-TTL credential only after a distinct approver records approval. CLI parity is trstctl-cli identities approve issue|rotate|revoke, trstctl-cli ephemeral issue, and trstctl-cli ephemeral approve; PAM sessions use trstctl-cli access sessions request, trstctl-cli access sessions request-status <approval_request_id>, trstctl-cli access sessions open, trstctl-cli access sessions list, trstctl-cli access sessions get, and trstctl-cli access sessions revoke <id> -f revoke.json where the JSON body contains {"reason":"incident containment"}. Target commands are trstctl-cli access targets list, get <target_type> <id>, register -f target.json, and disable <target_type> <id> -f reason.json.

Break-glass procedures (F34)

If the control plane is reachable during an incident, POST /api/v1/breakglass/issue serves online emergency issuance gated by an m-of-n operator quorum. The caller first sends the exact CSR/reason/TTL to POST /api/v1/breakglass/issue-ceremonies/preview. This effect-free check opens no ceremony, appends no event, records no idempotency key, calls no signer, and performs no external request. It returns the request and CSR fingerprints, effective lifetime, the signed CSR common name and every requested DNS, IP, email, and URI identity, safe deployment-readiness checklist, configured threshold and roster count (never operator names), exact execution effects, recovery steps, and verification steps. The supplied emergency subject must exactly equal the signed CSR common name; preview, ceremony start, online issue, and the offline issuance service reject a mismatch before signing. Custodians must review the full identifier set because the certificate carries the CSR's SAN identities, not just its common name. The caller then opens the unchanged request as a ceremony at POST /api/v1/breakglass/issue-ceremonies; distinct configured operators approve it with their own authenticated tokens, and the execution body contains only the ceremony id—not approver names. The server matches the roster to immutable ca.ceremony.approved event actors in the same locked transaction, so sub-quorum, cross-tenant, altered-request, and reused-ceremony attempts fail closed. The escrow signing key is a persisted purpose-constrained handle into the separate, isolated signing service, never caller-supplied key material in the API process. The result is a self-verifying signed bundle — anyone can verify it offline (signature + chain to the CA), and the served route reconciles it into the hash-chained audit log as immutable breakglass.issued before responding. If the control plane was unreachable, the isolated recovery signer uses a separately provisioned v2 public kit and two custodian signatures over one exact CSR, tenant, deployment, reason and TTL. It can issue while the API and database are stopped. After recovery, operators call POST /api/v1/breakglass/reconcile; the server verifies the complete proof against the independently pinned kit and records it in a v3 audit event. The disconnected runbook gives the steps. A bundle that fails verification stops the batch, so a forged emergency issuance can't be silently absorbed.

The same configured authority supports ceremony-bound CA rotation at POST /api/v1/breakglass/rotate and target-CA cross-signing at POST /api/v1/breakglass/cross-sign. Rotation creates a fresh signer-held key, preserves the predecessor policy lane, emits new-by-previous and previous-by-new cross-certificates for overlap/rollback, and replays the active handle after restart. Both results are public-certificate-only receipts. If a response is lost, an operator with issuers:read can list recent receipt summaries with GET /api/v1/breakglass/authority-receipts?limit=20 and recover the exact rotation or cross-sign result with GET /api/v1/breakglass/authority-receipts/{ceremony_id}. The list omits certificate bodies; the exact response includes the same PEM certificates and request digest returned by the mutation. The matching CLI commands are trstctl-cli breakglass authority-receipts list and trstctl-cli breakglass authority-receipts get <ceremony-id>.

In the console

The /incidents screen is the response console: a served blast-radius preview for the compromised identity, replacement-before-revoke execution with the resulting evidence, automated remediation playbooks for revoke / rotate / NHI right-size, a CAP-REM-02 owner self-remediation queue where bound owners can accept least-privilege right-size recommendations without broad incident authority, a SIEM/SOAR/chat/ITSM response-dispatch form for Splunk, Jira, Slack, and ServiceNow, a ServiceNow ITSM ticket form that queues the Table API call through the outbox, and a guided emergency certificate access workspace. That workspace collects a structured request, shows the exact zero-effect server preview, reports deployment-owned signer and quorum readiness without exposing the operator roster, opens the request-bound ceremony, waits for independently authenticated approvals, executes only after quorum, and gives the operator a verification checklist. Its advanced recovery panel folds offline-issued, quorum-approved bundles back into the event log (/api/v1/breakglass/reconcile). The emergency CA maintenance section opens rotation and cross-sign ceremonies for an exact reason/lifetime or target public CA certificate. The console shows the approval digest, refreshes the independently authenticated quorum, and disables execution if the reviewed fields change. An operator can resume a pending ceremony after a browser restart with its ID and the original fields. The rotation lifetime accepts 1 through 315,360,000 seconds in both the console and API, so an operator can choose the same long-lived CA successor through either path. The bounded result history recovers public certificates and the request digest after a lost response or cold restart; signer private keys never enter the console. The self-service approvals inbox at /approvals blocks self-approval of your own request. See The web console.

Use it

Credential compromise is served through REST, CLI, and the console:

compromised-issuer.json names the exact reviewed H2 plan; the API rejects a caller-supplied subset when it does not equal the issuer's active identity denominator:

{
  "issuer_id": "11111111-1111-4111-8111-111111111111",
  "replacement_authority_id": "22222222-2222-4222-8222-222222222222",
  "mode": "live",
  "reason": "intermediate signing key exposure",
  "rollback_ref": "restore the exact predecessor for the failed cohort",
  "cohorts": [
    {
      "id": "canary",
      "ordinal": 1,
      "members": [
        {
          "identity_id": "33333333-3333-4333-8333-333333333333",
          "agent_id": "44444444-4444-4444-8444-444444444444",
          "trust_anchor_path": "/etc/trstctl/trust/replacement-root.pem"
        }
      ]
    }
  ]
}

# This legacy mutation now refuses with H2 guidance; list/get still read history.
trstctl-cli incidents executions execute -f incident.json
trstctl-cli incidents fleet-reissuance start -f compromised-issuer.json
trstctl-cli incidents fleet-reissuance pause 33333333-3333-4333-8333-333333333333 -f pause.json
trstctl-cli incidents fleet-reissuance evidence 33333333-3333-4333-8333-333333333333
trstctl-cli incidents response-integrations dispatch -f response-dispatch.json
trstctl-cli itsm servicenow tickets create -f servicenow-ticket.json
trstctl-cli remediation playbooks
trstctl-cli remediation playbooks run nhi-right-size -f right-size.json
trstctl-cli remediation playbook-runs list --playbook_id nhi-right-size
trstctl-cli remediation owner-actions list
trstctl-cli remediation owner-actions accept right-size-aWRlbnRpdHkvMTEx -f accept-owner-action.json
trstctl-cli incidents executions list --identity_id 11111111-1111-1111-1111-111111111111
trstctl-cli incidents executions get 22222222-2222-2222-2222-222222222222

right-size.json names the identity whose usage-backed posture finding supplies the allowed scope delta. The connector must match an operator connectors.right_size binding, and target is the entitlement resource at that provider:

{
  "target_identity_id": "11111111-1111-1111-1111-111111111111",
  "reason": "remove the unused deployment grant",
  "connector": "least-privilege",
  "target": "payments-deployer",
  "remove_scopes": ["deploy:write"],
  "recommended_scopes": ["deploy:read"]
}

{
  "identity_id": "11111111-1111-1111-1111-111111111111",
  "reason": "private key export detected",
  "replacement_name": "payments-api-incident-replacement",
  "connector": "nginx",
  "target": "edge/prod/payments",
  "delivery_rollback_ref": "restore previous fullchain"
}

Dispatch the same incident response packet to Splunk, Jira, Slack, and ServiceNow:

trstctl-cli incidents response-integrations dispatch -f response-dispatch.json

response-dispatch.json:

{
  "title": "Contain compromised payments credential",
  "summary": "Rotate, revoke, page responders, and open investigation.",
  "severity": "critical",
  "correlation_id": "incident-2026-06-25",
  "evidence_refs": ["incident/22222222-2222-2222-2222-222222222222"],
  "destinations": [
    {
      "id": "splunk",
      "provider": "splunk",
      "endpoint_url": "https://splunk.example.com/services/collector",
      "token_ref": "env:TRSTCTL_SPLUNK_TOKEN"
    },
    {
      "id": "jira",
      "provider": "jira",
      "endpoint_url": "https://jira.example.com",
      "project_key": "SEC",
      "issue_type": "Task",
      "token_ref": "env:TRSTCTL_JIRA_TOKEN"
    },
    { "id": "slack", "provider": "slack", "channel": "security-incidents" },
    {
      "id": "servicenow",
      "provider": "servicenow",
      "instance_url": "https://example.service-now.com",
      "table": "incident",
      "token_ref": "env:TRSTCTL_SERVICENOW_TOKEN"
    }
  ]
}

Queue a ServiceNow incident ticket from the same response surface:

export TRSTCTL_SERVICENOW_INSTANCE_URL=https://example.service-now.com
export TRSTCTL_SERVICENOW_TOKEN_REF=env:TRSTCTL_SERVICENOW_TOKEN
trstctl-cli itsm servicenow tickets create -f servicenow-ticket.json

servicenow-ticket.json:

{
  "instance_url": "https://example.service-now.com",
  "table": "incident",
  "token_ref": "env:TRSTCTL_SERVICENOW_TOKEN",
  "short_description": "Rotate exposed TLS private key",
  "description": "trstctl incident response queued replacement-before-revoke remediation.",
  "category": "security",
  "urgency": "1",
  "impact": "2",
  "correlation_id": "incident-2026-06-25"
}

Equivalent REST call:

export TRSTCTL_CA_FILE=/etc/trstctl/pki/control-plane-ca.pem
curl -fsS --cacert "$TRSTCTL_CA_FILE" -X POST "https://trstctl.example.com/api/v1/itsm/servicenow/tickets" \
  -H "Authorization: Bearer $TRSTCTL_TOKEN" \
  -H "Idempotency-Key: incident-2026-06-25-servicenow" \
  -H "Content-Type: application/json" \
  -d '{"instance_url":"https://example.service-now.com","table":"incident","token_ref":"env:TRSTCTL_SERVICENOW_TOKEN","short_description":"Rotate exposed TLS private key","description":"trstctl incident response queued replacement-before-revoke remediation.","category":"security","urgency":"1","impact":"2","correlation_id":"incident-2026-06-25"}'

The caller needs incidents:write. The returned receipt names the outbox row and idempotency_key; the worker marks that row delivered only after ServiceNow accepts the Table API request.

Online break-glass issue is API-served when the signer-backed break-glass issuer is configured:

curl -fsS --cacert "$TRSTCTL_CA_FILE" -X POST "https://trstctl.example.com/api/v1/breakglass/issue-ceremonies/preview" \
  -H "Authorization: Bearer $TRSTCTL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"request_id":"bg-001","subject":"recovery.svc.example.test","csr_der":"...base64-csr...","reason":"regional outage","ttl_seconds":900}'

curl -fsS --cacert "$TRSTCTL_CA_FILE" -X POST "https://trstctl.example.com/api/v1/breakglass/issue-ceremonies" \
  -H "Authorization: Bearer $TRSTCTL_TOKEN" \
  -H "Idempotency-Key: incident-2026-06-25-bg-ceremony" \
  -H "Content-Type: application/json" \
  -d '{"request_id":"bg-001","subject":"recovery.svc.example.test","csr_der":"...base64-csr...","reason":"regional outage","ttl_seconds":900}'

# Each command uses a different configured operator's token.
trstctl-cli ca ceremonies approve <ceremony-id>
trstctl-cli ca ceremonies approve <ceremony-id>

curl -fsS --cacert "$TRSTCTL_CA_FILE" -X POST "https://trstctl.example.com/api/v1/breakglass/issue" \
  -H "Authorization: Bearer $TRSTCTL_TOKEN" \
  -H "Idempotency-Key: incident-2026-06-25-bg-issue" \
  -H "Content-Type: application/json" \
  -d '{"ceremony_id":"<ceremony-id>","request_id":"bg-001","subject":"recovery.svc.example.test","csr_der":"...base64-csr...","reason":"regional outage","ttl_seconds":900}'

Break-glass reconciliation remains API-served after an offline ceremony:

curl -fsS --cacert "$TRSTCTL_CA_FILE" -X POST "https://trstctl.example.com/api/v1/breakglass/reconcile" \
  -H "Authorization: Bearer $TRSTCTL_TOKEN" \
  -H "Idempotency-Key: incident-2026-06-25-bg-reconcile" \
  -H "Content-Type: application/json" \
  -d '{"bundles":[{"request_id":"bg-001","subject":"recovery.svc.example.test","cert_der":"...base64...","reason":"regional outage","approvals":["op1","op2"],"issued_at":"2026-06-25T17:00:00Z","signature":"...base64..."}]}'

The caller needs certs:issue; audit readers can then confirm the result with GET /api/v1/audit/events?type=breakglass.issued.

Open a short-lived Postgres session:

curl -fsS --cacert "$TRSTCTL_CA_FILE" -X POST "https://trstctl.example.com/api/v1/access/sessions" \
  -H "Authorization: Bearer $TRSTCTL_TOKEN" \
  -H "Idempotency-Key: incident-2026-06-25-pg-readonly" \
  -H "Content-Type: application/json" \
  -d '{"target_type":"postgres","target_id":"pg-main","role":"readonly","reason":"production incident 42","method":"k8s_sat","payload_base64":"...","ttl_seconds":900}'

The response includes the session id, expiry, and a one-time Postgres DSN. Store it only in the process that will use it. PostgreSQL's native role validity ends at the reviewed session deadline even if the background worker is unavailable. After expires_at, the worker also drops the generated database role and records pam.session.expired. ttl_seconds may be omitted to use the configured default; a negative value or a value above the configured maximum returns HTTP 422 before any target credential is created.

Open a short-lived SSH session:

ssh-keygen -t ed25519 -N "" -f /tmp/pam_id
curl -fsS --cacert "$TRSTCTL_CA_FILE" -X POST "https://trstctl.example.com/api/v1/access/sessions" \
  -H "Authorization: Bearer $TRSTCTL_TOKEN" \
  -H "Idempotency-Key: incident-2026-06-25-ssh" \
  -H "Content-Type: application/json" \
  -d '{"target_type":"ssh","target_id":"ssh-edge","role":"user","reason":"production incident 42","method":"k8s_sat","payload_base64":"...","ssh_public_key":"'"$(cat /tmp/pam_id.pub)"'","ssh_principal":"alice","ttl_seconds":900}'

Write the returned ssh.certificate value to /tmp/pam_id-cert.pub and connect with the matching private key. The target host must trust the trstctl SSH CA through its TrustedUserCAKeys configuration.

The lower-level library shapes remain useful for tests and future batch workflows:

// Compromise library: preview the blast radius, then remediate idempotently
report := incident.Preview("cert:abc123")            // read-only: what's affected
_, err := incident.Remediate(ctx, "cert:abc123", "idem-key-xyz")

// JIT: request, then two distinct approvers (dual control) → auto-issue
approval.RequestIssuance(ctx, approval.RequestSpec{ID: "req-001",
    Resource: "cert:db-tls", Requester: "alice", RequiredApprovals: 2})
approval.Approve(ctx, "tenant1", "req-001", "bob")
approval.Approve(ctx, "tenant1", "req-001", "carol")  // quorum met → issues

Blast-radius preview reads the same credential graph you can query directly; incident execution also appears in /incidents in the console. JIT notifications use the notification integrations.

Pitfalls & limits

Serving status: historical credential-compromise execution evidence (F31) remains readable through /api/v1/incidents/executions, trstctl-cli incidents executions *, and /incidents, while its unsafe direct mutation refuses with H2 fleet guidance; automated remediation playbooks (CAP-REM-01) are served through /api/v1/remediation/playbooks, /api/v1/remediation/playbooks/{id}/runs, /api/v1/remediation/playbook-runs{,/{id}}, trstctl-cli remediation playbooks*, and /incidents. The revoke and rotate playbooks take the same lifecycle gate as an identity's own revoke and issue (certs:issue, the ABAC overlay, the default-deny policy, dual control); the rotate playbook decides both its issue and its revoke before it creates the replacement. A playbook carries no exact approval, so under dual control it refuses with 403 and names POST /api/v1/identities/{id}/transitions, which does. Both revoke with an RFC 5280 revocation_reason (identity-revoke defaults to unspecified, credential-rotate to superseded), so the revocation reaches CRL and OCSP; the run keeps the operator's free-text reason, and keyCompromise goes through the identity transition, which contains the key on its hosts first. The rotate playbook works on an X.509 identity bound to a deployment target (anything else is refused with 409, since nothing could verify the replacement). It issues a replacement for the original's DNS name and target and returns a running run (replacement_awaiting_verification); it neither marks the replacement deployed nor revokes the original. Its internal remediation.rotate.verify command waits for the replacement's signed, verified delivery: the endpoint verification an enrolled host agent or network relay signs. Only then is the replacement marked deployed and the original revoked, and the run reads executed (replacement_verified_and_original_revoked, citing the verification). If the signed verification finds the endpoint not serving the replacement, or none arrives within verification_timeout_seconds (default 3600, 300 to 604800), the original stays valid, the run reads failed (replacement_verify_failed_original_kept or replacement_unverified_original_kept), and a rotation-replacement-unverified finding names the replacement, the original and the target. Owner-driven self-remediation (CAP-REM-02) is served through /api/v1/remediation/owner-actions, /api/v1/remediation/owner-actions/{id}/accept, trstctl-cli remediation owner-actions *, and the /incidents console; NHI right-size runs require usage-backed CAP-POST-01 posture evidence and queue connector.right_size through the outbox. A configured tenant/connector binding applies the least-privilege scope removal to the external entitlement API, reads the effective scopes back, and records a delivered/failed connector receipt; a binding whose executor is network runs that same conversation on an enrolled network relay inside its segment and records the relay's signed evidence; CA-compromise fleet re-issuance (F32) is served through /api/v1/incidents/fleet-reissuance-runs, trstctl-cli incidents fleet-reissuance *, and the /incidents console; compromised-credential / stolen-token detection (CAP-ITDR-02) is served through credential_compromise Discovery sources, runs, and findings; malicious / abused OAuth-grant detection (CAP-ITDR-03) is served through oauth_grant Discovery sources, runs, and oauth_grant_abuse findings; SIEM/SOAR/chat/ITSM response dispatch (CAP-REM-03) is served through /api/v1/incidents/response-integrations/dispatch, trstctl-cli incidents response-integrations dispatch, and /incidents; ServiceNow / ITSM ticket creation is served through /api/v1/itsm/servicenow/tickets and the /incidents console. JIT issuance is served. Online m-of-n break-glass issue/rotation/cross-signing is conditionally served when its signer handle, tenant, authenticated operator roster, and threshold are configured; offline-bundle reconciliation is served at /api/v1/breakglass/reconcile.
Order matters in remediation. The reissue-before-revoke ordering is deliberate; don't shortcut it, or you risk an outage mid-incident.
JIT needs real approvers configured and a notifier wired, or requests will sit in awaiting-approval until they expire.
PAM sessions need configured targets and attestors. Postgres targets need a tenant-bound provider with an administrative DSN reference that can create and drop scoped roles. SSH targets need hosts that trust the trstctl SSH CA; trstctl does not weaken host sshd trust on your behalf.
Break-glass is a last resort. It trades the control plane's guarantees for offline availability; reconcile the bundles promptly so the audit log is complete.

Reference

Compromise: /api/v1/incidents/executions, incident.execution.recorded, Workflow.Preview, Workflow.Remediate (replacement→deploy→revoke).
Playbooks: /api/v1/remediation/playbooks, /api/v1/remediation/playbooks/{id}/runs, remediation.playbook_run.recorded, and connector.right_size outbox delivery plus authenticated entitlement readback for right-size; revoke/rotate use the lifecycle state machine. The operator binding is under connectors.right_size in Configuration.
ITSM: /api/v1/itsm/servicenow/tickets, itsm.ticket.requested, itsm.servicenow outbox delivery; token material by token_ref only.
Response integrations: /api/v1/incidents/response-integrations/dispatch, response.integration.dispatched, and outbox fan-out to response.splunk, response.jira, notification.response, and itsm.servicenow; token material by token_ref only.
Fleet: /api/v1/incidents/fleet-reissuance-runs, trstctl-cli incidents fleet-reissuance *, migration.run.recorded, and incident.fleet_reissuance.recorded — exact-H1 planned, trust-before-leaf, signed-live-gated, revoke-after-verify, resumable, and JWS-sealed.
JIT: RequestIssuance, Approve, Deny; default RequiredApprovals: 2, self-approval blocked.
PAM: /api/v1/access/session-requests and /api/v1/access/sessions; distinct access:approve reviewers; Postgres scoped login roles; OpenSSH user certificates; pam.session.activation_requested, pam.session.started, pam.session.expired.
Break-glass: ceremony/execution pairs at /api/v1/breakglass/issue-ceremonies + /issue, /api/v1/breakglass/rotation-ceremonies + /rotate, and /api/v1/breakglass/cross-sign-ceremonies + /cross-sign; the isolated trstctl-signer offline kit, signed custodian requests, and POST /api/v1/breakglass/reconcile form the disconnected path. The older IssueOffline library helper is not an operator signing path.
Events: incident.*, response.integration.dispatched, fleet.*, approval.*, pam.session.*, breakglass.issued.