Runbook: failed release and rollback

This runbook covers the two different "rollback" concerns for the standalone signed-installer path (#12): a single client recovering from a bad update, and an operator preventing a bad release from ever reaching more clients. It assumes the tooling in Releasing tv — this page is the incident-response companion to that build/sign/publish runbook, not a replacement for it.

Client-side: tv update --rollback

tv update --rollback [--json]

Every standalone install retains exactly one prior version. --rollback atomically restores it and re-verifies it starts up correctly before considering the rollback successful — a corrupt or unbootable retained version fails the rollback rather than leaving the install in an unknown state. If no previous version is retained (a fresh install, or a second consecutive rollback), it refuses with RELEASE_NO_PREVIOUS_VERSION rather than doing nothing silently. Rollback only applies to a standalone install; a private-Git checkout has no retained version and refuses --rollback with UPDATE_ROLLBACK_NOT_SUPPORTED — see Install and update from a private Git checkout.

A failed or interrupted tv update (network failure, signature mismatch, checksum mismatch, platform mismatch, downgrade attempt, or a failed health check after activation) always leaves the currently active version fully intact — tv update never partially replaces the running binary. This means a client experiencing a bad update almost always doesn't need --rollback at all: the update simply didn't apply, and tv version --json still reports the previous (working) version. --rollback is for the case where an update did apply and the newly-active version itself turned out to be broken.

Operator-side: publish immutability

scripts/release/publish.mjs refuses to overwrite a stable tag (v<version>) that already exists on the releases repository:

refusing to overwrite the existing immutable release v1.2.3 on thalixinc/thalix-vault-releases. Cut a new patch version instead -- a published stable tag is never replaced.

This is deliberate: once a version is published, tv update's downgrade-rejection (RELEASE_DOWNGRADE_REJECTED) and signature/checksum verification only work as a safety net if the manifest a client already fetched can never later point somewhere different. You cannot "fix" a bad stable release by re-publishing the same version. Recovery is always forward:

1. Fix the defect.
2. Cut a new, higher version (v1.2.4), sign and publish it normally per Releasing tv.
3. Clients on the bad v1.2.3 self-heal on their next tv update, since the manifest for stable now points at v1.2.4 and the update engine's normal verification path applies unchanged.
4. If the bad release is actively harmful (not just broken), you cannot force-pull it back from already-updated clients — there is no remote-kill or forced-downgrade mechanism by design (a forced downgrade would itself be exactly the attack RELEASE_DOWNGRADE_REJECTED exists to prevent). The only lever is publishing the fixed version quickly and communicating out-of-band.

The beta channel is the one exception: it uses a single mutable channel-beta tag by design, so a beta build can be replaced directly — beta is explicitly not the immutability guarantee stable gives you.

Verifying which version a fleet is on

tv version --json

origin.source: "standalone" reports the active binary's path and whether a previous version is retained; there is no fleet-wide inventory command in tv today, so cross-machine rollout tracking is currently an operator-side concern outside this CLI.

Continue with Releasing tv or Incident investigation.