T-A97O-backup-prune-operation

State: planning/draft · Impact: medium · Complexity: medium

Goal

The backup store only grows. A user who deletes 1 GB from the vault still carries that 1 GB in the store forever. This task adds the operation that drops history older than a cutoff, so content deleted before the cutoff leaves the disk.

Today

crates/determined-backup has no retention. Each snapshot is a plain git commit on main in the bare store at <durable home>/com.sizerllc.determined/vaults/<vault id>/vault-backup/store.git. Every git call goes through git.rs as git --git-dir=<store> --work-tree=<vault> …. Snapshots and restores claim the vault-backup job key in port.rs, so they never overlap within one process. Reads claim no key.

Proposed

BackupPort gains prune(vault_id, keep_since), implemented in crates/determined-backup and exposed as an addressed POST in api/v1/backup.rs, so ontogen generates the route, IPC command and client. It rebuilds the store as a shallow clone that keeps every snapshot from the boundary on, then swaps it in.

What a prune keeps

The rule: every vault state from keep_since onward stays restorable.

• keep_since is an absolute instant. T-94OR computes it as now minus keep_days × 24 hours. It is not a calendar day in any timezone.
• Snapshot times are committer dates, the %cI that status.rs already reads. They compare as instants, so the timezone offset in the commit does not matter.
• The boundary is the newest snapshot whose committer date is strictly before keep_since. It holds the vault’s state at keep_since, so it is kept, with every snapshot after it.
• The boundary is found by walking git log on main newest first and stopping at the first commit dated before keep_since. Log order decides, not a date sort, so a clock that went backwards cannot drop a newer snapshot.
• If no snapshot is older than keep_since, or the boundary is already the oldest snapshot, there is nothing to drop. The prune is a no-op and says so.

Steps

1. Take the store lock (see below), then claim the vault-backup job key. A prune never runs during a backup or a restore.
2. Find the boundary. The depth is the number of snapshots from HEAD down to and including it.
3. git clone --bare --depth=<depth> file://<store> <store>.new. The file:// URL is required: with a plain path git ignores --depth and prints a warning. Depth counts commits, so the boundary is kept exactly.
4. Copy the old store’s config file over the new one. The clone does not carry the store’s user.name, user.email and commit.gpgsign pins. ensure_store sets those only on first init, so without the copy later commits use the user’s global identity or fail. The copy also drops the remote.origin entry the clone adds, which points at the old store.
5. Copy the old store’s index file into the new store. The clone has no index. Without it the next git add -A re-hashes the whole vault. The index matches HEAD after the last snapshot, and HEAD is kept, so its blobs are all present.
6. Swap: rename store.git to store.git.old, rename store.git.new to store.git, then remove store.git.old. Re-run ensure_store, which rewrites info/exclude.
7. Report the store’s bytes before and after, the number of snapshots dropped, the boundary’s sha and date, and the time of the prune.

Kept snapshots keep their shas, so links to them and a recent safety snapshot’s Undo keep working. A dropped sha fails rev-parse --verify and reads as an unknown snapshot.

Two entry points

prune::prune(cfg, keep_since) does the work and claims no job key. The port method claims the key and calls it. T-94OR’s scheduled job already holds the key, so it calls prune::prune directly. A second jobs.begin on a live key would refuse it.

The store lock

The job key works within one process only. port.rs says “Separate platform processes may still contend on the store’s Git lock.” Reads (get_status, get_list, get_changes, get_tree, get_file) claim no key at all. Both gaps matter for a swap. A read between the two renames finds no store.git. A snapshot in another process calls ensure_store, which would init_bare an empty store.git in the gap and hide the history behind it.

So the crate adds a cross-process lock: an OS file lock on vault-backup/store.lock, beside store.git and not inside it, since the store directory is replaced.

• A prune holds it exclusively for its whole run, clone included, so no other process can commit a snapshot the clone would miss.
• Snapshots, restores and ensure_store hold it exclusively.
• Every read holds it shared. A read that starts during a prune waits for the prune to finish.

Crash recovery

Recovery runs only under the exclusive store lock, at the start of a prune, a snapshot or a restore. Reads never recover. It handles three leftovers:

• store.git is missing and store.git.old exists: the crash came between the renames. Rename store.git.old back.
• Both exist: the second rename finished. store.git is the pruned store, so remove store.git.old.
• store.git.new exists: a clone did not finish. Remove it.

Because ensure_store takes the lock, it can never create an empty store.git in front of a store.git.old.

The boundary in responses

The boundary snapshot has no parent after the prune. Git treats it as a root commit. diff-tree --root and log --numstat then list every file in it as added. Its subject still carries the original counts, such as snapshot: +1 ~1 -0. This task marks the boundary in the list and changes responses (for example a boundary: true field, read from the store’s shallow file), so the UI can say “oldest kept snapshot” instead of showing a misleading all-added change list.

Approach

1. Add prune.rs with the boundary search, clone, config and index copy, and swap. Keep every git call in git.rs; add a helper there for the clone, which, like init_bare, takes no --git-dir/--work-tree.
2. Add the store lock. Take it in ensure_store, snapshot, restore, prune and every read.
3. Add crash recovery under the lock, as above.
4. Read the boundary from <store>/shallow and add it to the list and changes responses.
5. Add the port method, the wire request and result types, and the API endpoint. Record the result in the job log, as restore does.
6. Unit and integration tests in integration_tests.rs with dated commits.

Areas

Area	Note
solutions/determined/crates/determined-backup/src/	New prune.rs, clone helper in git.rs, store lock and recovery in config.rs, boundary flag in history.rs, port method and read locking in port.rs
solutions/determined/crates/determined-core/src/ports/backup.rs	BackupPort::prune
solutions/determined/crates/determined-core/src/wire/	Prune request and result types
solutions/determined/crates/determined-platform/src/api/v1/backup.rs	The addressed POST endpoint

Acceptance criteria

• AC-1: After a prune, every kept snapshot has the same sha as before. The list shows every snapshot at or after keep_since, plus the boundary, the newest snapshot before it.
• AC-2: With keep_since 30 days back, a note last changed 40 days ago and deleted 20 days ago can still be restored as it was 25 days ago.
• AC-3: A large file deleted before the boundary is gone from the store, and the reported bytes-after is smaller than bytes-before by about its size.
• AC-4: With no snapshot before keep_since, the prune is a no-op. With every snapshot before keep_since, exactly the newest one is kept.
• AC-5: A prune while a backup or restore holds the vault-backup key is refused with a clear error, and the store is untouched.
• AC-6: After a prune, Back Up Now commits on top of the kept history with the vault-backup identity, and restoring a file from the boundary snapshot works.
• AC-7: The list and changes responses mark the boundary snapshot.
• AC-8: A read that starts while a prune holds the store lock waits, then returns the pruned history. It never sees a missing store.
• AC-9: Each leftover (store.git.old alone, store.git.old beside store.git, a stray store.git.new) is cleaned up by the next prune, snapshot or restore, and no history is lost.

Implementation notes

• The store lock and recovery live in a new lock.rs, not in config.rs. ensure_store takes the lock; ensure_store_locked serves callers that already hold it.
• The port claims the vault-backup key before prune::prune takes the store lock, the reverse of step 1. A prune during an in-process backup is then refused at once, without first waiting for the backup’s current git phase to release the lock.
• A snapshot takes the lock once per git phase (stage, model input, commit), not for its whole run, so reads never wait on the model call. A prune from another process between stage and commit makes that snapshot’s commit fail; it cannot corrupt the store.
• A restore plans under the exclusive lock, not the shared one, so it repairs an interrupted swap before it resolves the sha.
• After copying the index, the prune runs git reset --quiet --no-refresh in the new store. An index staged but not committed (a snapshot preview leaves one) names blobs the clone does not hold, and the next commit would fail. Entries that match HEAD keep their stat data.
• The new store is checked before the swap: its HEAD must equal the old one, and it must hold exactly depth snapshots.
• git percent-decodes file:// URLs, so the clone escapes % in the store path.
• boundary is absent, not false, on every other snapshot, so existing reports and parity cases are unchanged.

Out of scope

• The retention setting, the automatic prune and the UI (T-94OR-backup-retention-setting).
• Pruning by snapshot count instead of age.
• Off-machine copies. A shallow store cannot be pushed to a fresh remote as it is.

Dependencies

• none

Discovery context

• Spike, 2026-10-01, git 2.54: ten dated commits with a 50 MB file deleted early. A shallow re-clone with --shallow-since took the store from 48 MB to 100 KB. Kept shas were unchanged. This task uses --depth instead, which builds the same kind of shallow store; the first integration test confirms it. log, rev-list, diff-tree, ls-tree, show, cat-file, add -A, diff --cached and commit all worked on the shallow store. A second prune of a shallow store also worked.

---

← Back to Tasks