Thalix VaultDocumentation
Browse documentation
Operators
publishedimplementedEdit this page

Runbook: incident investigation via correlation IDs

Every tv command surfaces a correlation ID, in both success and failure output. It is the thread that ties one operation's CLI output, service-boundary evidence, and database audit event together — when those records exist. See Audit and observability for the authoritative boundary contract this runbook walks through.

Step 1 — Get the correlation ID from the reporter

Ask the user or the failing script for the exact JSON error they received, or the "Correlation ID:" line printed under a human-readable error. Every failure carries one:

{
  "schemaVersion": 1,
  "error": {
    "code": "FILE_ARCHIVED",
    "message": "This file is archived.",
    "correlationId": "<the id to investigate>"
  }
}

If all you have is "the upload failed around 2pm," you don't yet have enough to investigate precisely — get the correlation ID first; approximate time ranges below are a fallback, not the primary tool.

Step 2 — Look it up in the audit trail

tv admin audit list --correlation "<the id to investigate>"

This is the fastest path when the operation reached a database-attributable decision. A single correlation ID can legitimately return zero, one, or several audit events — for example, a file access check that also touched an authorization-denial record. Read every returned event's action, result, targetType/targetId, and details together, in occurredAt order, to reconstruct what actually happened.

Step 3 — Widen the search when the correlation ID alone is not enough

If the correlation ID returns nothing (a purely local failure never reached the network — see the next section) or you need more context around it, filter by what you already know instead:

tv admin audit list --actor person@example.com --after 2026-08-20T13:55:00Z --before 2026-08-20T14:05:00Z
tv admin audit list --target-type file --target-id <file-id>
tv admin audit list --action file. --result denied

Every filter composes, so narrow with whichever facts you have (actor, a rough time window, target, an action prefix, or --result denied to find every authorization refusal in a window).

Step 4 — Know when "no record" is the correct answer

Not every correlation ID implies a database record exists. A correlation ID supports lookup; it does not promise every layer produced one:

  • A local usage error (a bad flag, a missing required argument) never reaches the network — its correlation ID is generated purely on the client and has no server or database record at all.
  • Poll exhaustion or local session expiry during tv login retains the last request's correlation ID, not a synthesized new one — look up that ID, not a guess at "whenever the CLI eventually gave up."
  • A persistence failure after a successful token issuance (the network round-trip succeeded, writing the credential file locally failed) retains the token-response correlation ID — the server-side event for that ID will show a successful authentication even though the client ultimately reported failure. This is the correct, expected reconciliation, not a contradiction to chase further.
  • Routine authorization_pending/slow_down device-login polls are intentionally quiet and do not create individual audit noise — do not expect one audit row per poll.

Step 5 — Confirm exceptional access separately

If the investigation concerns a file an administrator accessed without owning or holding a grant on it, that access is recorded as its own audit event carrying the --admin-reason text verbatim (see Audit and administration) — look it up the same way, filtered by --target-type file --target-id <file-id>, and read the reason directly rather than assuming intent.

Worked shape (illustrative, not a captured transcript)

A user reports tv file update failed. They supply this error:

{"schemaVersion":1,"error":{"code":"FILE_ARCHIVED","message":"This file is archived.","correlationId":"c-9f2a-…"}}
  1. tv admin audit list --correlation "c-9f2a-…" — likely empty, because FILE_ARCHIVED is a precondition failure returned before any audit-worthy mutation is attempted; the RPC rejects before writing a row.
  2. Fall back to tv admin audit list --target-type file --target-id <file-id> and read the most recent event: if the last entry is a file.archived action, the explanation is simply that the file was already archived — direct the user to Restore from archive rather than treating this as a bug.

Continue with Audit and administration or Deactivation.