Upgrade Workbench

This document covers moving an existing Workbench install from one tagged release to the next. It does NOT cover the first-run install (see the bootstrap guide) or ad-hoc image-tag swaps in development (the procedure here assumes a production install with persistent state worth keeping).

Spec: install-docs-v1 § C.8 (DOC-R100..R106).

Supported upgrade shape: single-tenant single-host, stop the running container, pull the new tag, start a new container against the same persistent volume. SQLite migrations apply automatically on the new binary's first startup; the per-release list of migrations new in each tag lives in the release notes under that tag's section header.

The persistent volume name and contents are documented in Configuration — that's the same volume you mount at /var/lib/workbench/ in the Compose example.

---

1. Back up first

Take this step seriously. Workbench's entire mutable state — SQLite database, license artefact, workspace seed — lives in the workbench_data volume. The recovery story across a failed upgrade depends on having a known-good tarball of that volume from before you pulled the new image.

Three things must survive every upgrade — preserve all three:

• The workbench_data volume (SQLite + license artefact).
• The identity volume (/var/lib/arq — instance_id, install_secret, activation files).
• The master key (WORKBENCH_MASTER_KEY / WORKBENCH_MASTER_KEY_FILE). It is not in either volume — it lives in your secret manager, and the workspace seed inside the SQLite database is wrapped with it. Restoring the volumes with the wrong (or a missing) master key leaves every stored credential permanently undecryptable. The new container must start with the same master key. Custody + recovery: Security posture § Master key.

The supported backup is a stopped-container snapshot. SQLite is WAL-mode by default, but a clean shutdown guarantees the on-disk file copy is consistent.

# 1. Stop the container so the SQLite file is quiesced.
docker compose stop workbench

# 2. Snapshot the named volume to a tarball. Substitute the actual
#    Docker volume name for your install — Compose typically
#    prefixes it with the project name.
docker run --rm \
  -v workbench_data:/data:ro \
  -v "$(pwd):/backup" \
  alpine \
  tar czf "/backup/workbench-$(date +%Y%m%d-%H%M%S).tar.gz" -C /data .

The resulting tarball name follows the convention workbench-YYYYMMDD-HHMMSS.tar.gz so support tickets reference recognisable artefacts.

Keep this tarball until the upgraded container has been running healthily for at least one operator-driven smoke pass (license visible, databases listed, a fresh analyzer import accepted). Only then is rollback unlikely enough to justify deleting it.

Online backup (no downtime)

If a maintenance window is inconvenient, the binary can take a consistent copy of the SQLite database while it is running, with no stop. The backup verb issues a SQLite VACUUM INTO against the live database and writes a single, self-contained .sqlite file — no -wal/-shm sidecars are needed to restore it.

docker exec workbench workbench backup \
  --sqlite-path /var/lib/workbench/workbench.sqlite \
  --out "/var/lib/workbench/backups/workbench-$(date +%Y%m%d-%H%M%S).sqlite"

The source path defaults to WORKBENCH_SQLITE_PATH when --sqlite-path is omitted, so on a standard install the flag can be dropped. The verb refuses to overwrite an existing --out (exit 3), so a timestamped destination never clobbers a prior backup. Exit codes are listed in Operating Workbench.

The online copy is a drop-in database file: to restore it, stop the container, replace workbench.sqlite in the volume with the copy, and start the container (see § 7). The identity volume (/var/lib/arq) holds small static files — back those up with a plain file copy or the volume tarball above; they do not need VACUUM INTO.

When to use which. The online backup is ideal for routine, low-friction snapshots and for taking a fresh copy immediately before a patch without downtime. The stopped-container snapshot remains the belt-and-braces pre-upgrade artefact, because it also captures the license artefact and workspace seed in one tarball for the rollback in § 7.

---

2. Record the running digest (so you can roll back)

Before pulling, capture exactly which image bytes are running. The tag you might be running could be replaced upstream; the digest is content-addressed and pinned.

docker inspect workbench --format '{{.Image}}' > /tmp/wb-current.digest
cat /tmp/wb-current.digest
# → sha256:<...>

If the upgrade goes wrong, this is the exact reference you pin in Compose for the rollback step.

---

3. Pull the new tag

docker compose pull workbench

This downloads the new image into the local Docker engine without touching the running container.

If you verified the previous image with cosign per verification.md, repeat the verification against the new digest BEFORE step 4. The signing identity and OIDC issuer expectations are unchanged across releases.

---

4. Start the new container against the same volume

# Bring the new image up. The persistent volume is preserved.
docker compose up -d workbench

# Watch the migration apply on first boot.
docker compose logs --since 1m workbench | grep schema_migrations

The new binary runs SQLite migrations forward in-place. Existing findings, reports, license artefact, database registrations, and analyzer tokens are preserved through the migration.

---

5. Verify

Three checks confirm the upgrade succeeded:

# 5a. /healthz returns 200 with a healthy shape.
curl -fsS http://127.0.0.1:8080/healthz | jq .

# 5b. The running binary's version matches the new tag.
docker inspect workbench --format '{{.Config.Image}}'

# 5c. Migrations applied. The `schema_migrations` table records
#     every version. Compare the highest version with the
#     migration list in CHANGELOG.md for the new tag — they
#     should match.
docker exec workbench sqlite3 \
  /var/lib/workbench/workbench.sqlite \
  "SELECT version, name, applied_at FROM schema_migrations ORDER BY version"

If step 5a returns a non-200 status, or step 5b shows the old image, or step 5c shows fewer migrations than the new tag's CHANGELOG section lists, treat the upgrade as failed and proceed to § 7 (Rollback).

---

6. Migrations — how they work

Workbench applies SQLite schema migrations forward only on every startup. The migration runner:

• Reads the current schema_migrations table to find which versions have already applied.
• Runs every unapplied migration in version order, inside its own transaction.
• Records the new version + name + applied-at timestamp on success.

The migration set ships in the image as Go source (see internal/workbench/store/migrations.go). There is no operator-tunable migration knob; you cannot skip a migration or hold the schema at an old version while running a new binary.

Per-release list. Every CHANGELOG section for a tag calls out the new migrations introduced in that tag with the marker migration vN (e.g. migration v7 adds integration_targets...). This is a mechanical CI gate — see DOC-R095 in the install-docs-v1 specification. A release that introduces a migration without a CHANGELOG callout fails the gate before merging.

The GitHub Release page for each tag inherits the CHANGELOG section verbatim and adds a relative link back to this page (per DOC-R096). So operators reading the release page always see both the migration list and the upgrade procedure.

---

7. Rollback

If /healthz does not turn green after step 4, or if the upgraded SPA misbehaves in a way the Troubleshooting table can't explain:

# 1. Stop the new container.
docker compose down workbench

# 2. Pin the Compose file's image reference to the recorded
#    previous digest from /tmp/wb-current.digest. Substitute the
#    actual repository ref:
#
#      image: ghcr.io/elevarq/workbench@sha256:<previous-digest>

# 3. Start the old image against the SAME persistent volume.
docker compose up -d workbench

The SQLite store does not auto-rollback. If the new release's migrations advanced the schema past what the old binary knows, the old binary refuses to start and exits with store_error (exit 3), logging a clear line:

database schema vN is newer than this binary supports (vM); restore a backup or upgrade the binary

This is a deliberate, early refusal — the binary will not run against a schema it cannot fully understand. You can confirm the live schema version at any time from /healthz (the schema_version field). See the exit-code table in Operating Workbench. At that point the ONLY recovery path is to restore the volume from your backup-before-upgrade tarball:

# 1. Ensure no Workbench container is running against the volume.
docker compose down

# 2. Wipe the volume (destructive — be certain).
docker volume rm workbench_data
docker volume create workbench_data

# 3. Unpack the backup tarball into the fresh volume.
docker run --rm \
  -v workbench_data:/data \
  -v "$(pwd):/backup" \
  alpine \
  sh -c 'cd /data && tar xzf /backup/workbench-YYYYMMDD-HHMMSS.tar.gz'

# 4. Start the old image (still pinned to the recorded digest).
docker compose up -d workbench

After restore, /healthz should return 200 and licensing.cache_state: "fresh" (the restored license artefact re-loads on startup). Findings, reports, database registrations, and analyzer tokens are back as of the backup point.

This is why § 1 says to take the backup step seriously. Without it, a backward-incompatible migration is unrecoverable.

---

8. Where to find the per-release migrations list

Three surfaces, all kept in sync:

If they disagree, the SQLite table is authoritative — that's what actually ran. The CHANGELOG and Release page are documentation.

---

Next

• Troubleshooting — closed failure → action mapping if the upgrade didn't produce a healthy /healthz.
• Operating Workbench — back to day-to-day operation.
• Verification — repeat cosign verify against the new digest before exposing the upgraded container.