T-M1VQ-backup-changes-lazy-line-counts

State: planning/draft · Impact: medium · Complexity: medium

Goal

Opening a snapshot’s Changes tab is slow on a large store. The per-file line counts need git diff-tree --numstat, which reads and diffs every changed blob. The snapshot list already dropped numstat for --raw and went from 21.8 s to 0.09 s cold on a 5 GB store. This task gives the Changes view the same speed: files first, counts when they arrive.

Today

• history::list_snapshots runs git log --raw. It compares tree entries only.
• history::changes_for runs two diff-tree calls at once, --name-status and --numstat, and joins them by path in join_changes. The response waits for the slower call, which is numstat.
• BackupPort::get_changes returns one ChangesReport { sha, changes }. Each Change carries linesAdded and linesDeleted. Null means a binary file. api/v1/backup.rs::get_changes passes it through.
• BackupChangesList.vue calls transport.backupGetChanges once, shows DDataState until it returns, then renders each row with its counts. A null count renders as “binary”.
• Parity cases get-changes-* under crates/determined-parity/fixtures/cases/backup/ assert the counts in the one response.

Proposed

The Changes view lists files at once and fills counts in later.

Request	Returns	Cost
get_changes(sha)	{ sha, changes: [{ path, status }] } from diff-tree --name-status only	Tree compare, fast
get_change_counts(sha) (new)	{ sha, counts: [{ path, linesAdded, linesDeleted }] } from diff-tree --numstat	Reads blobs, slow on large stores

• A binary file has null counts in get_change_counts, as today.
• A row shows a small placeholder where counts go until the counts arrive. It must not show “binary” while counts are loading. That label means a loaded null.
• A failed or slow count load leaves the list usable. Rows, the restore buttons and the Browse tab work throughout. A failed load shows a quiet “line counts unavailable” note with a retry. It does not replace the list with the error state.
• Per snapshot or per file is open. Start with one request per snapshot, since it keeps one git call. Per file only pays off if one huge file stalls the rest. Decide in the first step.
• Switching snapshots while counts load drops the stale reply, so counts never land on the wrong snapshot.

Approach

1. Decide the request grain with a timing run on a large store. Default: per snapshot.
2. In history.rs, split changes_for into a name-status read and a change_counts_for numstat read. Drop run_pair there. The client applies counts by path, so join_changes goes away or shrinks to the status parse.
3. Add ChangeCountsReport to wire.rs. Remove the count fields from Change.
4. Add BackupPort::get_change_counts, the api/v1/backup.rs endpoint and the ontogen-generated transport. Validate sha as get_changes does.
5. In BackupChangesList.vue, load files first, then call backupGetChangeCounts and merge counts by path. Track a separate counts status (pending, error, success) and a request token to drop stale replies. Update BackupChange in types.ts.
6. Update the parity cases get-changes-* and add get-change-counts-* cases (follow-up, first snapshot, unknown sha, invalid sha).
7. Add a component test for the split.

Areas

Area	Note
solutions/determined/crates/determined-backup/src/history.rs	Split changes_for; add change_counts_for
solutions/determined/crates/determined-backup/src/wire.rs	Change loses counts; new ChangeCountsReport
solutions/determined/crates/determined-backup/src/port.rs	get_change_counts
solutions/determined/crates/determined-core/src/ports/backup.rs	BackupPort::get_change_counts
solutions/determined/crates/determined-platform/src/api/v1/backup.rs	New endpoint
solutions/determined/crates/determined-parity/fixtures/cases/backup/	Updated and new cases
solutions/determined/experiences/backup/app/components/BackupChangesList.vue	Two-stage load
solutions/determined/experiences/backup/app/types.ts	BackupChange counts become optional

Acceptance criteria

• AC-1: The file list appears without waiting on numstat. get_changes runs no --numstat call.
• AC-2: Counts fill in when get_change_counts returns, and match what get_changes returned before the split, including null for a binary file.
• AC-3: While counts load, rows show a placeholder, not “binary”.
• AC-4: A slow or failed count load leaves the list usable. Rows and restore buttons work, and a failed load shows a retry for counts only.
• AC-5: Switching snapshots while counts load never shows the old snapshot’s counts.
• AC-6: A regression case covers the split: a parity case for each request, and a component test that delays or fails the counts request and checks the list.

Out of scope

• Line counts in the snapshot list. It keeps --raw.
• Caching counts across sessions.
• The boundary snapshot’s all-added list (T-A97O-backup-prune-operation).

Dependencies

• none

Discovery context

• list_snapshots moved from --numstat to --raw: 21.8 s to 0.09 s cold on a 5 GB store. changes_for kept numstat because the Changes view shows counts, so opening a snapshot is still slow there.

---

← Back to Tasks