Runbook: stale upload reconciliation

Object storage and the SQL catalog cannot share one transaction. A client that streams bytes but never calls back to commit leaves a file_versions row stuck in upload_state = 'pending' and a possibly-orphaned object in storage. Neither state is ever readable — tv file get and search only ever see committed versions — but stale pending rows and orphaned objects should still be found and cleaned up periodically.

Implementation status: implemented and exposed to operators. tv admin reconcile wires the administrator-gated public.reconcile_stale_file_uploads RPC to a command and cross-checks each result against a live storage listing, so the report distinguishes a genuine orphaned object from a version whose upload never reached storage. The RPC is also exercised directly in tests/integration/cli-files-lifecycle.test.mjs and tests/security/cli-files-security.test.mjs.

Detect stale pending uploads

Run tv admin reconcile as an admin/global_admin account (the command checks the caller's role itself and refuses a user-role token with FORBIDDEN):

tv admin reconcile [--stale-after <seconds>] [--json]

--stale-after is the age threshold in seconds (default 3600, one hour); only pending rows older than that are reported. The result is a read-only report, never a mutation:

{
  "schemaVersion": 1,
  "command": "admin reconcile",
  "stalePending": [
    {
      "fileId": "...",
      "versionId": "...",
      "storageKey": "...",
      "createdAt": "2026-08-20T12:00:00Z",
      "objectExistsInStorage": true
    }
  ],
  "correlationId": "..."
}

stalePending is ordered oldest-first and scoped to the caller's own organization. An empty array means nothing needs attention. objectExistsInStorage is the per-entry storage cross-check the command performs automatically: true means an object actually landed in the bucket; false means the upload never reached storage.

Clean up

1. For each storageKey with objectExistsInStorage: true, the upload's commit call never arrived (client crash, network failure, or a completed-but-uncommitted upload). The object is orphaned — safe to delete directly from storage once you've confirmed no in-flight client is still working on it (compare createdAt against your --stale-after threshold; the command already only reports rows past that age).
2. For each storageKey with objectExistsInStorage: false, the upload never got as far as storage. The pending row itself is inert — it was never readable and never will be — but if you want it removed for hygiene rather than left as a permanently-pending row, that requires a direct, audited SQL statement against file_versions, since no RPC exists to fail/cancel a pending version out-of-band today (fail_file_upload requires the original uploader's own live session, not an administrator override).
3. Record what you did (which keys, which rows, why) outside tv — there is no CLI audit event for this manual cleanup path yet, since it bypasses the RPC surface entirely. See Incident investigation for how correlation IDs tie this back to the original failed upload attempt if the client reported one.

Cadence

Run this as a scheduled check (daily is reasonable for MVP scale) rather than only reactively — --stale-after should comfortably exceed your largest expected upload's realistic duration plus retry window, so a still-in-flight legitimate upload is never mistaken for orphaned.

Continue with Restore from archive or Incident investigation.