Discovery & inventory — find every certificate, key, and secret you already have

What it is

Before you can manage credentials, you have to know they exist. Discovery is how trstctl finds the credentials already scattered across your infrastructure — certificates, SSH keys, and secrets — and the inventory is the single, tenant-scoped list it keeps them in.

Think of it like a building's master key register: someone has to walk every floor, write down every lock and every key, and keep that register current as locks change. trstctl is that walker and that register, for machines.

In the console, open Discover. The page starts with the credentials that need a decision. Choose Run scan to use an existing discovery source or add your first one.

trstctl discovers credentials five ways, and each suits a different corner of your estate: scanning the network from outside, asking a trstctl Edge runtime what a host can see from inside, pulling inventory straight from cloud provider APIs, reading SSH key files and trust config, and connecting to external secret stores. Everything they find lands in one inventory.

Why it exists

Almost every organization has more machine credentials than it can name, and the ones nobody remembers are the ones that cause outages and breaches: the certificate that expires on a forgotten load balancer at 3 a.m., the SSH key a contractor left behind, the API token hard-coded in a script five years ago. You cannot rotate, revoke, or risk-score a credential you do not know about.

Discovery turns "we think we have a few hundred certs" into a precise, queryable list — the foundation every other trstctl feature builds on. Risk scoring, drift detection, the credential graph, and lifecycle automation all read the inventory.

How it works

The inventory (F1) — the source of truth that is actually a projection

The inventory is a PostgreSQL table of certificate metadata — subject, SANs, issuer, serial, SHA-256 fingerprint, key algorithm, validity window, where it's deployed, and lifecycle status. It never stores a private key.

Nothing writes to that table directly — a core trstctl design rule (event sourcing). When a certificate is discovered or issued, the orchestrator appends a certificate.recorded event to the append-only, tamper-evident log, and a projector reads that event and builds the table row. The table is a projection — a derived view you could delete and rebuild from the log — which is why trstctl can survive a database loss: the truth is the event log, and the inventory is a fast index into it.

Ingestion is idempotent: the row key is (tenant_id, fingerprint), so seeing the same certificate twice refreshes one row instead of creating a duplicate, and the ingest API requires an Idempotency-Key so a retried request can't double-record. Certificate parsing routes through the single isolated cryptography path, so the inventory code itself never touches the low-level X.509 libraries directly.

GET /api/v1/certificates/health serves a tenant-wide expiry dashboard from that same projection: total and lifecycle-status inventory counts, expired and 7/30/90-day expiry bands, source breakdown, and the soonest-expiring certificates. Total and source counts preserve revoked and superseded history. Expiry action counts, bands, and the soonest-expiring list include only active certificates, so a retired predecessor does not raise a new expiry warning. The expiry bands sum to the active count. A short-lived certificate issued through the approved JIT workflow remains in inventory and the separate expected_ephemeral band, even after a later discovery observation changes its source label. Its planned expiry is excluded from renewal alarms, the expiry worklist and alert delivery; request a fresh attested credential through Workloads instead. The active count is the retained lifecycle status, so it includes a JIT record until revocation or retirement even after its validity window closes. Issued, manually imported, and discovery-fed active certificates all count, including one found on a load balancer after a different CA issued it.

Find a certificate across the inventory

Open Certificates and use Search certificates to find a known credential without loading unrelated pages first. Search checks the authenticated tenant's subject, SANs, issuer, serial, fingerprint, lifecycle status, and deployment location. The list stays paginated; its loaded count is not a total search-result count.

The console waits briefly after typing, cancels obsolete reads, and keeps the search field available while a request runs or returns no matches. Changing the search, page size, or expiry window starts a new page sequence. The issuer, profile, team, and environment controls still filter the matching rows loaded in the console.

API clients use GET /api/v1/certificates?q=<text>&limit=20. Matching is a case-insensitive literal substring; a complete certificate UUID also matches. Percent signs and underscores are literal characters, not wildcards. The trimmed query must be valid UTF-8, contain no NUL, and have at most 256 characters; invalid queries return 400. Omit q or pass whitespace to use the ordinary inventory query. For subsequent pages, send next_cursor as cursor with the same q and expiring_before. The existing page-size limit is 1–100. Search runs before both pagination and the expiry filter's result limit; it does not retrieve private keys, issuance responses, or custody receipts. Broad substring searches can scan the tenant's metadata; this endpoint does not establish a large-estate latency guarantee. When expiring_before is set, the result contains active certificates only. Omit it to search historical revoked and superseded rows as well.

Recover the certificate behind a scanned row

Some paths record what a certificate says without the certificate itself: a network scan without agent_channel.probe_scanned_certificates, a cloud listing, an older import. The certificate detail says so (certificate_bytes_held: false), and such a row cannot take part in authority agreement checks, which identify a certificate by the issuer and serial inside its bytes. Certificate bytes → Look up in this tenant's CA records (certificates recover-der <id> with {"method": "ca_ledger"}) finds a mint of the tenant's CAs that retained the certificate by the row's serial; Read from host:port ({"method": "endpoint"}) reads the certificate that address serves from the control plane — for a private address the console sends that one address as the allowed CIDR, which needs egress:private. Either way the bytes are attached only when they hash to the row's fingerprint and the row held none; the answer says recovered, already_held, not_found, mismatch (the endpoint serves another certificate) or unreachable, and the detail then shows where the bytes came from.

Network discovery (F2) — scanning from an enrolled network relay

Network discovery connects from an explicitly enrolled agent with the network role to IP/port ranges you define, performs a normal TLS handshake, captures the certificate each host presents, and records its metadata. No software is installed on the targets — it sees exactly what any client on the network would see.

The scanner runs in its own bounded lane: a bounded pool of workers (default 16, queue 256). When the queue fills, it slows the producer instead of dropping targets or exhausting the pool the API needs, so a big scan can never starve the rest of the system. The handshake and certificate parsing both go through the single isolated cryptography path, and the scanner applies the shared SSRF guard and a reserved-IP denylist before dialing expanded CIDRs, so a scan cannot be turned into a loopback, RFC1918, link-local, or cloud-metadata probe.

Operators can safely save a network, ssh, or adcs source before its relay is enrolled. Saving records configuration; it does not scan. Plan preview and GET /api/v1/discovery/sources/{id}/preflight return ready, the exact connection origin, and blocked_reasons from the same server-owned readiness rule. If no non-offboarded network-role relay has sent an authenticated heartbeat within two configured heartbeat intervals, manual and scheduled run admission both refuse the run before a run event or outbox row is created. A stale relay may have an expired channel certificate; re-enroll that exact agent and wait for a fresh heartbeat. The console explains the fix instead of leaving an impossible run silently queued.

Readiness also checks the server's agent dispatch allowlist. A healthy relay is not enough when discovery.run (or the source’s exact adcs.inventory / adcs.inventory.enrollment_groups / adcs.inventory.endpoint_probe kind) is absent from agent_channel.claimable_job_kinds: preview and saved-source preflight name that exact configuration gap, and run admission returns a conflict before creating an event or outbox row. The console therefore cannot report “ready” for work the server is configured never to hand out.

Once at least one eligible relay is online, the run executes from the outbox worker. External probes are journaled first and delivered at-least-once, so a temporary relay disconnect after admission is durable and retryable instead of being done inline by the request handler. Readiness proves that an eligible relay exists; it does not claim that a particular relay is connected at every millisecond.

What a run tells you, target by target

A completed run carries one outcome per assigned target in target_results (succeeded, failed, blocked, or rejected, with the reason for anything that did not succeed). A partial run's error names the targets that failed or were blocked, so Discover → Runs shows which listener is the problem instead of a bare count. Listeners that need an application-level upgrade to TLS are handled without a protocol hint: when a reachable target rejects a bare TLS ClientHello, the scanner retries with the PostgreSQL SSLRequest negotiation and inventories the certificate the server then presents.

Running the same source again refreshes its findings instead of duplicating them: a listener observed by a later run keeps its one finding (and its triage decision) with first_seen_at, last_seen_at, seen_count, and run_id moved to the latest run, so the console and the open-finding counts describe the estate once.

Recover a failed or partial run without erasing the failure

Open Discover → Runs and choose Retry on a failed or partial run. The console first rechecks the source's current relay and scope prerequisites. If the source is ready, POST /api/v1/discovery/runs/{id}/retry creates a new queued run with the same source, schedule binding, and dry-run mode. The original terminal run is never changed or deleted. The replacement stores retry_of_run_id, so the table, API, event replay, and a page reload still answer “which failure did this replace?” The console shows newest runs first; Load older runs follows the bounded history. API clients can request GET /api/v1/discovery/runs?order=recent&limit=50 and pass each returned next_cursor with the same order to page by creation time and run ID. The CLI exposes the same mode with trstctl discovery runs list --order recent --limit 50 and accepts the returned cursor with --cursor. Omitting order keeps the original ID-order contract.

This is intentionally not available for queued, running, or succeeded runs. An active run must finish; a successful run can be repeated from its source as a fresh scan. Every retry requires an Idempotency-Key, so a browser, proxy, or human sending the same request again receives the first replacement instead of queueing duplicates. Headless operators use trstctl-cli discovery runs retry <failed-run-id> and get the same server rule. If recovery is blocked, restore or enroll the required network-role relay, then retry with a new idempotency key; never rewrite the old run or its outbox row.

If a relay's signed result says it discovered more credentials than it actually included, the control plane fails that run and imports no findings from it. It retains the signed receipt, marks the exact agent job failed, and acknowledges the terminal report so one bad sealed result cannot block every later job after a restart. Discover → Runs and the API show the count mismatch and the need to retry. Use the same Retry action above after the relay is updated; the old failed run remains in the audit trail.

Continuous monitoring rollup

GET /api/v1/discovery/monitoring (CLI: trstctl discovery monitoring) is a single read-side rollup over the tenant's discovery sources, enabled schedules, last runs, findings, and certificate inventory — it creates no new state and joins the same discovery.* and certificate.recorded projections other endpoints already read. Each source row shows whether it's scheduled, the monitoring interval, the latest run status, current execution readiness, the exact blocker and connection origin, finding counts, and pointers to /api/v1/certificates and /api/v1/discovery/findings. The console keeps this exact rollup under Monitoring and exact scan evidence. The credentials needing a decision appear first; operators can still expand the underlying CT, drift, coverage, source, schedule, run, and repository-path proof when they need to verify or troubleshoot it.

Agent-based discovery (F3) — what each host can see from the inside

A network scan only sees what a host presents on a port. Plenty of credentials never appear on the wire: a certificate sitting in a file, in a PKCS#11 token, in the Windows certificate store, or in a Kubernetes Secret. The trstctl agent runs on the host and enumerates those local sources, then reconciles what it finds into the inventory over its mutually-authenticated (mTLS) channel. Each source is independent: if one token errors, the agent records the error and keeps going, so one broken source can't hide the rest.

The agent enrolls into the control plane with a one-time bootstrap token (POST /enroll/bootstrap); GET /api/v1/agents then lists it, and POST /api/v1/agents/{id}/offboard emits an agent.offboarded event, keeps the row visible as a tombstone with actor/reason evidence, and causes the agent's mTLS certificate to be rejected on future heartbeat, renewal, and inventory RPCs. POST /api/v1/agents/{id}/cert-revocations takes the exact client-certificate serial, SHA-256 fingerprint, or both. The control plane commits an independent tenant-scoped denial before publishing either removal event, so an already-open mTLS connection is refused on its next RPC even if the event projection fails. The pending command retains the exact event for restart recovery; the projected tombstone and revocation row remain rebuildable from the event log. This agent channel checks its own denial on every RPC and renewal; it does not wait for CRL or OCSP propagation to terminate agent access.

In Agents → Fleet status and safe actions → Revoke certificate, identify the exact client certificate with its serial or SHA-256 fingerprint. On the agent host, openssl x509 -in agent.crt -noout -serial -fingerprint -sha256 reads those values without exposing the private key. The console requires at least one selector and explains that the next agent request is refused, including a request on a connection opened before revocation. If the certificate cannot be inspected, Offboard agent denies the whole agent identity. Operators can use the same selector with the API or CLI; an empty selector does not mean “current certificate.”

Enrollment is deliberately two steps in Agents → Add agent:

1. Review exact enrollment plan asks POST /api/v1/agents/enrollment-tokens/preview what the token would allow. The server returns the trimmed agent identity, signed certificate roles, public agent address, TLS name the new agent must verify, required permissions, and any blockers. This call creates no token, event, job, or agent connection.
2. Mint one-time token appears only while that exact plan is ready and unchanged. Editing the identity or either role hides the plan and requires a new review. Minting then calls the separate idempotent token route, and the token is shown once; the browser does not save it to local or session storage.

Leaving the identity blank means any agent identity in the caller's tenant may use the token. Selecting Network relay also requires agents:relay.grant; ordinary agents:write authority cannot silently turn a host agent into a relay. If the public agent address or TLS name is missing, preview says exactly what must be configured and minting stays unavailable. Headless operators use trstctl agents enroll-token-preview with the same JSON body before trstctl agents enroll-token.

Changing an enrolled agent's roles. The roles live in signed SANs on the agent's certificate, so they change at a renewal, never by editing a row, and no re-enrollment is needed. An operator holding agents:relay.grant records the complete role set with Agents → Fleet status and safe actions → Change roles, POST /api/v1/agents/{id}/roles with {"roles": ["host", "network"], "reason": "…"}, or trstctl agents roles grant <agent-id>. The grant is an agent.roles.granted event; agents:write alone cannot record one. The agent's next heartbeat asks it to renew, the renewal over the agent channel signs exactly the granted roles and records which grant it signed (agent.cert.renewed, schema 2, carries the new certificate's roles), and the agent reconnects with the new certificate. The agent list shows the change as pending ("Next renewal: …") until a renewal signs it and applied, with the new certificate's serial, after; the roles column keeps showing what the presented certificate carries. An agent built before this release ignores the request and takes the roles at its scheduled renewal, with about a third of its 24-hour certificate left. Removing the network role works the same way; the old certificate stays valid until it expires, so revoke it as well when the change is urgent. An offboarded agent cannot take new roles.

A renewal never takes roles from the CSR. A CSR that asks for a role the renewal is not signing is refused with the role named, on the agent channel and on the HTTP renewal endpoint (/enroll/renewal), which carries the presented certificate's roles over unchanged and does not apply grants.

The agent's own discovery sources are filesystem, pkcs11, windows-store, k8s-secret, trust-store, and private-key — the read has to happen on the host, since only it can see its own files, tokens, Windows store, trust stores, browser profiles, or in-cluster Secrets. An enrolled agent sends metadata-only ReportInventory batches over the same mTLS channel it uses for heartbeat and renewal. Its first authenticated heartbeat must register the agent before an inventory batch; otherwise the RPC asks it to heartbeat first. The server rejects inline secret-looking metadata keys, caps batch size, and ingests in the bounded agent lane so a noisy fleet cannot starve the API. GET /api/v1/agents advertises the accepted source kinds for each enrolled endpoint, and the Agents console shows the same capability list.

Only a completed inventory run whose recorded executing agent ID matches the receiver-stamped finding can create graph trust-store or service-dependency edges. Manual imports remain visible as discovery findings but cannot claim agent authority. Generic POST /api/v1/discovery/sources declarations cannot create the reserved agent kind or an agent: source name; enrolled reporters create these records through the authenticated inventory channel. The CT monitoring setup also refuses an agent: name, so a watchlist cannot reserve or replace a host's source. CT updates return a conflict if their requested name belongs to another source kind or to a different CT source ID; operators must choose an unclaimed name. A generic source declaration cannot change the kind bound to an existing name either; its preview reports that ownership conflict before the operator starts the mutation. The secretscan:, secret-repo:, and secret-third-party: source names are also reserved for their product workflows. Configure those scans through the Secrets tool or its dedicated API, not by creating a generic discovery source or CT monitor under a product-owned name. The weak-key-audit: name belongs to the inventory weak-key audit, whose runs appear in run history (see Weak keys). Repository and third-party scan source names end with their stable source ID. That keeps otherwise identical labels separate when a repository/ref contains a colon or when two third-party artifacts share a provider, source, and event.

The selected-agent panel links to View discovery runs and View discovery findings, so collector claims can be checked against the server's actual evidence. An offline or stale agent cannot provide current evidence. First check its service and network path. If the endpoint has the wrong certificate or capability role, revoke that certificate or offboard the agent, then enroll it again with a newly reviewed one-time token. Offboarding never deletes the old row: the actor, reason, and time remain as a tombstone, so recovery cannot erase the failure that caused it.

For Linux certificate files, the shipped agent can inventory public certificate roots at startup with --inventory-cert-roots, reporting references, fingerprints, and certificate metadata only — never private keys or secret values.

Kubernetes TLS Secrets. A cluster is frequently the largest single population of certificates an organization holds, and the one nobody has an inventory of. Run the agent in-cluster with --inventory-k8s-secrets and it enumerates the kubernetes.io/tls Secrets in its own namespace through the in-cluster service account, reporting each Secret's certificate metadata over the same mTLS inventory path as every other source.

The exact contract: it reads tls.crt — public certificate material — and never tls.key; no key bytes cross the agent channel. It needs list access to Secrets in that namespace. A Secret whose certificate does not parse is skipped rather than failing the pass, so one malformed Secret does not cost you the inventory of the rest; an unreachable API server is an error rather than an empty result, because reporting "no certificates" for a cluster you could not reach is worse than reporting nothing.

One agent can cover more than its own namespace. --inventory-k8s-namespaces all lists every namespace the service account may list and inventories each one; --inventory-k8s-namespaces payments,edge names the namespaces to read. Each finding still carries its namespace and Secret name, and only tls.crt is read. The RBAC is the operator's choice: deploy/kubernetes/rbac-inventory-namespaces.yaml is the opt-in, read-only cluster grant (list namespaces, list Secrets) for all; for a named list, bind a namespaced Role with list on Secrets in each namespace. A namespace the service account cannot list is reported as an error naming that namespace, never as an empty inventory, and the readable namespaces are reported first, so a missing binding shows up as a gap in the run rather than as a clean cluster.

PKCS#11 tokens. HSMs and smart cards hold the certificates an organization cares most about and inventories least. The default agent image is statically linked and cannot load a PKCS#11 module, and it says so: --inventory-pkcs11-module there fails with "built without cgo" instead of reporting an empty token, and the shipped census the agent sends on heartbeat does not list pkcs11. The release publishes a second agent image, ghcr.io/ctlplne/trstctl-agent-pkcs11 (built from deploy/docker/Dockerfile.agent-pkcs11, make agent-pkcs11-image locally), with cgo, glibc, SoftHSM and OpenSC's opensc-pkcs11.so on board; vendor modules such as libykcs11.so or an nShield/Luna Cryptoki are mounted in. Run that image with --inventory-pkcs11-module /usr/lib/softhsm/libsofthsm2.so (or the vendor path), optionally --inventory-pkcs11-token <label> to pick one token and --inventory-pkcs11-pin-file for tokens whose certificate objects are not public.

The contract matches the other collectors: a read-only session, objects of class CKO_CERTIFICATE only, CKA_VALUE and CKA_LABEL read, never a private-key search and never C_Sign or an export; a token or PIN that cannot be opened is an error, never an empty inventory. Findings carry the token and object label as their location (not the module path, which differs per host for the same physical token). The repository's SoftHSM container test loads the real module, writes a certificate and a key pair onto a token with pkcs11-tool, and checks that the agent reports the certificate and nothing about the key.

CA, trust-store & private-key discovery

GET /api/v1/ca/discovery rolls up CA estates: configured public upstream CAs, configured private upstream CAs, and imported private CA hierarchy authorities, all in one tenant-scoped response with source id, source kind, public/private scope, status, and pointers such as /api/v1/external-cas/{id}/issue or /api/v1/ca/authorities/{id}/issue. It intentionally omits certificate PEM and private-key bytes — operators use it to see which CAs are connected, then follow the referenced route for issuance or import.

Trust-store discovery is a separate agent collector, because a trusted CA is not the same thing as a deployed service certificate. The agent reads public trust anchors from OS trust directories, Java cacerts/JKS files, NSS profile exports, browser profile exports, and Windows trust-store enumerators, tagging each finding with trust_store_kind (os, java, nss, browser, or windows) and private_key_present=false. It also reports the certificate SHA-256 fingerprint and SPKI SHA-256 public-key identity. The graph counts a store as trusting a managed issuer only when one of those exact identities matches; a subject-name-only match is a separately visible unverified candidate, not an authoritative edge. This lets the control plane answer "what does this host trust?" without ever moving a key or mistaking two same-name roots for the same authority. The control plane stamps the host from the reporting agent's verified mTLS identity, so several stores or anchor paths on one machine remain several stores across one host.

Finding exposure. GET /api/v1/discovery/findings?distrust_entry=<entry name> lists the findings whose reader matched a distrust-list entry, and ?key_strength=weak those whose key was rated weak (CLI trstctl-cli discovery findings list --distrust_entry "<entry name>" or --key_strength weak). The MCP distrust_exposure and weak_key_exposure tools answer from these and the rated certificate inventory.

Removing a distrusted root. When the control plane's distrust list names a trust-store anchor itself (for example emSign ECC Root CA - G3, by key), the finding can be removed from the host through trust distribution: POST /api/v1/discovery/findings/{id}/remove-distrusted-root with the finding's fingerprint and anchor_path as reviewed (connectors:write; CLI trstctl-cli discovery findings remove-distrusted-root --force). The removal is queued for the exact host agent that found the anchor, which must run with --relay-claim and a --host-exec-profile whose allowed_roots include the anchor's directory. The agent reads the file itself (the control plane never holds its bytes), and removes it only when it holds exactly that one certificate and the agent's own distrust list names it: the shipped entries plus any operator list given with --distrust-file. The receipt ends as trust_anchor_removed, trust_anchor_absent, trust_anchor_removal_refused (a bundle, another certificate, a root the agent's list does not name, or a path outside the profile or behind a symlink) or trust_anchor_removal_failed. A bundle such as /etc/ssl/certs/ca-certificates.crt is never edited: remove that root with the host's trust tool (update-ca-certificates, update-ca-trust or the platform equivalent). Only OS trust-store files are removable this way; Java keystores, NSS databases, browser exports and the Windows store need their own tools. A removed source file reaches bundles built from it the next time the host's trust tool runs.

Private-key-material discovery answers the companion question: "what sensitive key files exist here?" Point the agent at canary directories with --inventory-private-key-roots. It reads each regular file locally, classifies PKCS#8, PKCS#1 RSA, SEC1 EC, OpenSSH, and encrypted private-key containers through the isolated cryptography boundary, wipes the file buffer after inspection, and reports only the path, format, algorithm, file-mode metadata, and — when the key is parseable — a fingerprint derived from the public key. Encrypted keys are located and tagged as encrypted, but no passphrase is requested and no private bytes ever reach the control plane.

SSH credential discovery (F42) — keys and standing access

SSH is where forgotten access hides. trstctl inventories SSH credentials two ways: a network-side SSH handshake captures each host's host key, and the on-host agent reads host keys, user keys, authorized_keys grants, known_hosts trust anchors, and each configured sshd_config's trust posture (A115): the fingerprints of the CAs in its TrustedUserCAKeys file, its RevokedKeys list (KRL version and digest, a plain key list, or a missing file), each HostCertificate it presents (signing CA, CA signature verified or not, key ID, principals, validity), and AuthorizedKeysFile. It reads the global section the way sshd resolves it (first value wins, HostCertificate accumulates, Include files in place) and counts Match blocks rather than applying them. The posture is reported when the agent connects, so restart the agent after changing sshd_config. The same agent path locates and classifies SSH and TLS private-key files as metadata-only findings instead of copying key bytes into the control plane.

On-host SSH collection is explicit, not a surprise filesystem crawl. Configure only the paths the agent may read with --inventory-ssh-host-key-globs, --inventory-ssh-user-key-globs, --inventory-ssh-authorized-keys, --inventory-ssh-known-hosts, and --inventory-ssh-sshd-configs. Every flag is empty by default. For example:

trstctl-agent ... \
  --inventory-ssh-host-key-globs '/etc/ssh/ssh_host_*_key.pub' \
  --inventory-ssh-authorized-keys '/home/*/.ssh/authorized_keys' \
  --inventory-ssh-known-hosts '/etc/ssh/ssh_known_hosts,/home/*/.ssh/known_hosts' \
  --inventory-ssh-sshd-configs /etc/ssh/sshd_config

The agent reports the result over its existing mTLS inventory RPC. The control plane derives the tenant from the verified agent certificate, appends discovery events, and projects the metadata into GET /api/v1/ssh/fleet. Operators can read the same view with trstctl ssh fleet or in Machine & Workload Trust → SSH trust.

Two flags make the result actionable. StandingAccess marks an entry that grants persistent login (an authorized_keys line). Orphaned marks a standing-access grant whose comment field is blank — meaning nobody can say whose key it is. An orphaned standing-access key is exactly the thing a security team wants surfaced. Only the fingerprint is ever stored, never private key material (held in wipeable memory and zeroed after use).

The control plane serves ssh discovery source/run/finding records and executes non-invasive SSH host-key scans from the discovery outbox worker. Source configs accept explicit targets (host:port) or CIDRs plus ports, run on a bounded worker lane, and use reserved-address filtering unless the operator explicitly allows private or loopback diagnostic targets.

Discovery source kinds, one shape

Eight more source kinds follow an identical pattern: create a source with a kind and a config, queue a run, read back metadata-only findings. Rather than walk through that shape eight times, here is what each one actually finds:

All eight run through the discovery outbox worker, normalize their input into metadata-only findings carrying a stable provenance string (<kind>:<key-parts>), and append the standard discovery.* audit events. Several also stamp a capability tag used for compliance/ITDR mapping (for example CAP-ITDR-02, CAP-ITDR-03) — see Observability & risk for how findings feed risk scoring and posture. Every finding also becomes a node in the credential graph with its provenance and risk score; a related bridge ingests leaked-credential findings from scanners (gitleaks, trufflehog) into the same graph, again excluding the secret value structurally.

One worked example — the shape is identical for the other seven kinds; change kind and config (the create → run → findings CLI flow is the same one shown for network under Use it below):

{
  "kind": "oauth_grant",
  "name": "quarterly-oauth-consent",
  "config": {
    "grants": [
      {
        "provider": "okta",
        "app_id": "0oa-payments",
        "principal": "payments-bi-export",
        "resource": "google-workspace",
        "scopes": ["drive.readonly"],
        "consent_type": "admin",
        "third_party": true,
        "owner": "finance-platform"
      }
    ]
  }
}

For oauth_grant specifically: a grant with provider threat signals, a dangerous or .default scope, offline_access combined with high-privilege admin consent, an unverified publisher, a missing owner on a privileged grant, or a suspicious redirect URI additionally emits an oauth_grant_abuse finding tagged capability=CAP-ITDR-03. Ordinary high-risk-but-owned grant inventory stays oauth_grant only, so trstctl does not count inventory as detection.

Unified NHI inventory — every credential kind, one denominator

GET /api/v1/nhi/inventory (nhi:read) is a read-side projection, not a new state store, over identities, certificate inventory rows, API-token metadata, enrolled agents, and discovery findings, normalized into one item shape spanning certificates, SSH keys, secrets, API keys, OAuth apps, tokens/PATs, service accounts, IAM roles, webhooks, workload IDs, and agents. Each item preserves its provenance (identity, certificate_inventory, access_api_token, agent_fleet, or discovery_finding) plus public metadata — owner, status, fingerprint, risk score, source system, external reference — and never returns secret values, private keys, or raw API tokens. The dashboard's NHI inventory summary consumes this API directly, so the first screen counts all eleven kinds instead of only the managed identities table.

Managed NHI decommissioning rides the same denominator: POST /api/v1/nhi/decommission selects tenant-local managed identities by departure, vendor-term, or inactivity signal and drives the normal event-sourced revoke/retire lifecycle transitions.

Ownership attribution — the canonical treatment

Every non-human identity needs an accountable owner, not a discovery note that says "owner: unknown." GET /api/v1/ownership/attribution (nhi:read) resolves NHIs to an owner two ways: managed identities resolve their stored owner_id directly, while discovery-fed rows resolve the metadata-only owner, team, or vendor name captured at discovery time against registered owners. Anything that doesn't resolve stays marked orphaned rather than silently counted as accountable — that feeds tenant governance reporting and the orphaned-owner signal risk scoring weighs. Each resolution carries an evidence ref (such as discovery.finding:<id> or inventory:<id>) back to the record that produced it, so an operator can trace an attribution to its source. The same data is available via trstctl-cli owners attribution and the Owners console; Policy & governance is the main consumer that turns this into compliance posture.

Discovery triage

Findings are immutable evidence, but their operator triage state is mutable and event-sourced. triage_status starts as unmanaged; the state model also includes investigating, managed, dismissed, and (A106) remediated and accepted_risk. POST /api/v1/discovery/findings/{id}/claim marks a finding managed, and POST /api/v1/discovery/findings/{id}/dismiss dismisses it with a reason. Both are tenant-scoped, idempotent mutations guarded by discovery:write.

POST /api/v1/discovery/findings/{id}/resolution (A106, discovery:write) records a resolution with a reason and an action: investigate (an assignee and an optional future due_at; repeating it reassigns), remediated (an evidence_ref for the fix), accept_risk (an expires_at within one year; the leader's lifecycle sweep reopens the finding when the acceptance expires, as system:risk-acceptance-expiry), or reopen (from remediated or accepted risk). Each action refuses the other actions' fields. A finding reports overdue when its investigation is past due and regressed when it was seen again after its fix was recorded. The NHI shadow and stale posture leave a fix that held and an unexpired acceptance out of the open posture and count them (remediated, accepted_risk, resolved); the CT and drift dashboard summaries count every status (triage_counts), plus overdue and regressed findings. On the Posture page each CT-log and drift row has a Resolve action with these choices, limited to the moves its status allows.

GET /api/v1/discovery/findings filters on the server by run_id, source_kind (the kind of the source a finding was recorded under, for example ct_log or drift), kind, and triage_status (unmanaged includes findings never triaged), and pages by cursor. The CT-log and drift dashboards (GET /api/v1/discovery/ct-monitoring and GET /api/v1/discovery/drift-remediation) take triage_status, cursor, and limit (1-100, default 50): their findings are one page of their own source kind with a next_cursor, and their summary counts every finding of that kind on the server.

A finding produced without an explicit ID gets one deterministic ID from its exact tenant, run, kind, reference, and fingerprint. This matters when the outbox retries a partly completed scan: the retry records the same observation identity instead of a new UUID that collides only after restart replay. Older histories can contain both payload IDs for one natural observation. The projection keeps those IDs as aliases, chooses the earliest immutable observation (then lexical ID for an exact-time tie) as the canonical row, and resolves later triage events through either alias. A duplicate whose source, provenance, risk, or metadata differs is not merged: replay fails with the tenant, natural key, both IDs, and differing fields rather than silently changing the evidence.

GET /api/v1/nhi/posture/shadow is the shadow posture view: it pages through tenant-scoped discovery findings, excludes anything already claimed as managed or dismissed, counts unmanaged and unregistered external NHIs by kind and surface, and returns recommendation/evidence refs only — never credential values. trstctl-cli nhi posture shadow reads the same view.

In the console

The /discovery screen is named Find unmanaged credentials because that is the operator's job, not the database feature underneath it. Its default reading order is:

1. What needs attention — unmanaged count, high-risk count, and credential types.
2. Credentials to review — one plain-language row action opens the finding.
3. Claim — the main next step turns a reviewed finding into managed inventory.

When creating an identity from a TLS finding with one distinct observed subject alternative name, the console suggests that name rather than the certificate's full distinguished name such as CN=web.example.com. This lets endpoint enrollment reuse the requested identity for the same hostname and owner. The name remains editable; when a certificate has multiple names, review which endpoint this identity represents before creating it. Creation and claiming do not issue or deploy a certificate.

Less-common rotate, revoke, decommission, remediate, and dismiss actions are grouped under More actions. Exact fingerprints, internal finding/source/run IDs, provenance, evidence references, raw kinds, the shadow-NHI projection, CT monitoring, and drift details remain available through explicit evidence disclosures. This keeps the first decision understandable while preserving the technical proof needed for an audit or investigation. See The web console.

Use it

The certificate inventory (F1) is live today. Drive it from the CLI:

# list certificates, newest first, paginated
trstctl-cli certificates list --limit 50

# list only certificates expiring within a window
trstctl-cli certificates list --expiring-before 720h

# show estate-wide expiry/source health, including imported and discovered certs
trstctl-cli certificates health

# ingest a certificate you already have (idempotent)
trstctl-cli certificates ingest -f ./server.pem

Those map to the served REST routes GET /api/v1/certificates, GET /api/v1/certificates/health, and POST /api/v1/certificates (the latter requires an Idempotency-Key header). For certificates ingest, -f accepts a PEM certificate directly as shown above. To set an owner, deployment location, or source during import, pass a JSON file with pem, owner_id, deployment_location, and source fields instead. An import keeps the signed certificate, recorded as an observation, so revocation health watches the CRL and OCSP endpoints it names (including an AD CS LDAP distribution point covered by an AD CS discovery source); importing a certificate trstctl issued updates its provenance and never replaces the public chain trstctl retained. The CLI supplies a fresh idempotency key for either form; repeat an exact request with the same explicit key when recovering from an uncertain response.

Network discovery is live too:

cat > segment.json <<'JSON'
{"name":"edge","ranges":["10.0.0.0/24"],"staleness_hours":24,"excluded":false}
JSON
trstctl-cli discovery segments create -f segment.json

cat > source.json <<'JSON'
{"kind":"network","name":"edge-tls","config":{"segment":"edge","targets":["10.0.0.10:443"],"allow_rfc1918":true}}
JSON
trstctl-cli discovery sources create -f source.json
trstctl-cli discovery sources list
trstctl-cli discovery sources preflight <source-id>

cat > run.json <<'JSON'
{"source_id":"<source-id>"}
JSON
trstctl-cli discovery runs start -f run.json
trstctl-cli discovery runs get <run-id>

Copy the run ID from the start response. Read that exact run until its status is succeeded. A queued or running response is still pending work; a failed run requires inspecting its error and target results before retrying. Then read the findings for that completed run:

trstctl-cli discovery findings list --run_id <run-id>
trstctl-cli nhi posture shadow

This example deliberately permits an owned RFC1918 target inside the declared segment. Without allow_rfc1918, private IPv4 targets remain blocked. In the console, select Allow approved RFC1918 targets for that same explicit scope. This permission does not allow metadata, link-local, multicast, or other prohibited addresses. A separate allow_loopback admits loopback targets (127.0.0.0/8, ::1) for an authorized local diagnostic, such as a relay checking a listener on its own host; it widens nothing else. Hostnames must be declared by their exact name; an IP-only segment does not implicitly authorize a hostname that resolves into it. Preflight checks known literal-address blockers without dialing; DNS and connectivity are checked by the relay during the run.

The segment command maps to POST /api/v1/discovery/segments; declare that bounded estate scope before creating any network or SSH source. The remaining commands map to POST|GET /api/v1/discovery/sources, GET /api/v1/discovery/sources/{id}/preflight, POST|GET /api/v1/discovery/schedules, POST|GET /api/v1/discovery/runs, GET /api/v1/discovery/runs/{id}, POST /api/v1/discovery/runs/{id}/retry, GET /api/v1/discovery/findings, POST /api/v1/discovery/findings/{id}/claim, and POST /api/v1/discovery/findings/{id}/dismiss. The shadow posture command maps to GET /api/v1/nhi/posture/shadow.

A discovery schedule name belongs to one schedule within a tenant. Retry a lost create response with the same Idempotency-Key to recover that creation; a separate create with the same name returns 409 before another event is recorded. Use GET /api/v1/discovery/schedules to find the existing schedule.

Open Discover → Schedules to pause future scans, resume a paused schedule, or retire it permanently after reviewing the exact schedule name. Pausing and retiring leave prior runs and findings available. A stale scheduler decision is checked again at run admission, so it cannot queue a new scheduled run after pause or retirement. Retirement keeps the schedule visible as a tombstone and refuses resume; create a new schedule with a new name if monitoring should start again. The same controls are available through POST /api/v1/discovery/schedules/{id}/pause, /resume, and /retire and the trstctl-cli discovery schedules commands. GET on the schedule ID reads the current state, including retired_at. Each mutation needs an Idempotency-Key; retry the same key after a lost response to recover the original result.

To see enrolled agents that perform local discovery:

trstctl-cli agents list

# On an enrolled host, report public certificate files the agent can see.
trstctl-agent --enroll-url https://localhost:8443 \
  --bootstrap-token-file ./trstctl-bootstrap-token \
  --server localhost:19443 \
  --server-name localhost \
  --name edge-agent-1 \
  --ca-bundle ./trstctl-ca.pem \
  --inventory-cert-roots /etc/ssl,/etc/pki/tls/certs \
  --inventory-os-trust-roots /etc/ssl/certs,/etc/pki/ca-trust/source/anchors \
  --inventory-java-trust-stores "$JAVA_HOME/lib/security/cacerts" \
  --inventory-nss-trust-roots "$HOME/.pki/nssdb/exported-roots" \
  --inventory-browser-trust-roots "$HOME/.config/chromium/Default/exported-roots" \
  --inventory-private-key-roots /etc/ssl/private,/etc/ssh

# Then read the projected discovery inventory and graph from the control plane.
trstctl-cli discovery findings list
trstctl-cli graph nodes

--enroll-url is the control-plane base URL, not /enroll/bootstrap; the agent adds the endpoint. The console uses this same contract on Agents and in the setup wizard. When the console itself is intentionally served from loopback HTTP for local evaluation, its generated command adds --allow-insecure-loopback-enrollment. The binary rechecks that the resolved destination is loopback before every dial. Non-loopback HTTP remains refused.

When you find a credential you didn't expect, follow it into the credential graph to see what it can reach, or into risk scoring to see why it matters.

Pitfalls & limits

Be precise about what runs in the server today versus what ships as tested library code awaiting control-plane wiring (this matters for an honest evaluation — see also Current limitations):

NHI policy compliance lives in Policy & governance; CT-log monitoring and drift detection live in Observability & risk — all three read the same discovery projections, but they aren't this page's features.

Other gotchas: a network scan only sees what a host presents on a port at scan time — pair it with agent-based discovery for the full picture. Cloud discovery needs read-only credentials with list/get permission on the relevant service. Secret-store discovery records references, so a finding tells you a secret exists and where, not what it is.

Reference

• CLI groups: the discovery-relevant commands live under discovery (sources, schedules, runs, findings, ct-monitoring, drift-remediation, monitoring), plus certificates, agents, and nhi. trstctl-cli has many more command groups outside discovery; trstctl-cli <group> --help lists the current subcommands for any of them.
• Served routes: GET|POST /api/v1/certificates, GET /api/v1/certificates/{id}, GET|POST /api/v1/discovery/sources, GET /api/v1/discovery/sources/{id}/preflight, GET|POST /api/v1/discovery/schedules, GET /api/v1/discovery/schedules/{id}, POST /api/v1/discovery/schedules/{id}/pause|resume|retire, GET|POST /api/v1/discovery/runs, GET /api/v1/discovery/runs/{id}, POST /api/v1/discovery/runs/{id}/retry, GET /api/v1/discovery/findings, POST /api/v1/discovery/findings/{id}/claim, POST /api/v1/discovery/findings/{id}/dismiss, GET /api/v1/agents, GET /api/v1/nhi/posture/shadow, GET /api/v1/ownership/attribution, POST /api/v1/agents/enrollment-tokens, POST /api/v1/agents/{id}/offboard, GET /api/v1/graph, POST /enroll/bootstrap.
• Agent channel: AgentService.ReportInventory over the mTLS agent gRPC listener when agent_channel.enabled is true.
• Config: TRSTCTL_LIFECYCLE_RENEW_BEFORE (default 720h) sets the expiry window the inventory and lifecycle treat as "renew soon".
• Served discovery source kinds: network, ssh_trust (explicit agentless SSH trust through a pinned network relay; setup and limits), cloud_certificate, cloud_secret, nhi_cross_surface, oauth_grant, service_account, nhi_behavior, credential_compromise, k8s_ingress_gateway, ct_log, drift, manual, plus metadata-only ssh, secret_store, api_key, and agent.
• Discovery source kinds (agent): filesystem, pkcs11, windows-store, k8s-secret, trust-store, private-key.
• Agent inventory flags: --inventory-cert-roots, --inventory-os-trust-roots, --inventory-java-trust-stores, --inventory-java-trust-store-password, --inventory-nss-trust-roots, --inventory-browser-trust-roots, --inventory-private-key-roots, --inventory-ssh-host-key-globs, --inventory-ssh-user-key-globs, --inventory-ssh-authorized-keys, --inventory-ssh-known-hosts, --inventory-ssh-sshd-configs.
• Audit events: certificate.recorded, discovery.source.upserted, discovery.schedule.upserted, discovery.schedule.paused, discovery.schedule.resumed, discovery.schedule.retired, discovery.run.queued, discovery.run.started, discovery.finding.recorded, discovery.run.completed, secretscan.finding.

See also

Policy & governance (ownership attribution's main consumer) · Observability & risk (scoring what you discover) · Graph, query & AI (what a credential can reach) · Secrets · Current limitations · glossary: certificate, fingerprint, bulkhead, event sourcing

Covers: F1, F2, F3, F42, F49, F35, F36

Nested AD CS enrollment links

Discovery’s AD CS source wizard can opt into read-only nested group reads. Select an upgraded network relay, enable Read nested enrollment groups, and provide the explicit membership domain naming context. Preview checks both the server’s job allowlist and that relay’s fresh enabled executor census. Posture displays the observed users/computers, canonical SIDs, security-group paths, trustee, observation time and completeness. An incomplete or refused read gives a recovery action and no verified member list. This does not evaluate a full Windows token or authorize issuance. The default direct-ACL reader stays available.

The nested enrollment runbook includes exact source JSON, CLI operations, bounds, matching-rule verification and the Windows forest proof boundary. The focused served journey exercises real Samba AD, vaulted single-attempt credentials, mTLS signed results, immutable projections, CLI readback, tenant isolation and replay. The final image/browser/restart qualification follows the release replay.

AD CS enrollment endpoint candidates

The AD CS source wizard can opt into fixed-path probes derived from directory hostnames. Enable Probe enrollment paths from directory hostnames, select an upgraded network relay and enter explicit DNS suffixes. Preview checks the relay capability without contacting LDAP. Candidate counts become known during the queued directory read; private destinations need an additional approved CIDR grant.

Posture shows configured, LDAP-derived and deduplicated combined origins, the latest page's total and offsets, per-service coverage and unattempted outcomes. The next run advances the retained cursor. Empty endpoints on a different page mean unassessed, and unreachable endpoints remain unverified. Findings and compliance evidence carry those limits; HTTP reachability does not prove IIS Extended Protection or successful SCEP enrollment. The service-view ceiling is visible when reached. The endpoint probing runbook explains exact configuration, bounds and recovery. Windows IIS/NDES qualification requires the Windows environment; a private Samba/HTTP/TLS fixture is separate proof.