Duplicate image scanning and safe resolve

State: open/building · Kind: module · Audience: system

Summary

Finds exact and near-duplicate images in a vault and resolves each group by trashing the copies the user did not keep.
Repoints vault links from removed copies to the kept file.

Statement

The crate scans a vault for duplicate images. Exact duplicates are grouped by content hash. Near duplicates (resized or re-compressed copies) are grouped by perceptual hash. It can also rank the vault against one query image. Resolving a group validates the plan against live disk, refuses the whole group if any file drifted, and moves removals to a .trash folder inside the vault instead of deleting them. It then rewrites vault links to the kept file and records an audit log.

What it provides

Exact (sha256) and near (dHash union-find) duplicate grouping with stable group ids
Similarity ranking of the scan report against one image
Vault-wide link index and link retargeting rewrite
Validated keep/remove resolve with .trash and an audit log
Config read from the _system/media-scanner.md note

Underlying implementation

solutions/determined/crates/determined-media/src: scan.rs and group.rs find and group duplicates, similarity.rs ranks, links.rs rewrites links, resolve.rs validates and trashes; port.rs (Media) binds it to the platform; config.rs reads the scanner note

Locations

solutions/determined/crates/determined-media/
solutions/determined/crates/determined-media/src/lib.rs
solutions/determined/crates/determined-media/src/port.rs#Media
solutions/determined/crates/determined-media/src/scan.rs
solutions/determined/crates/determined-media/src/group.rs
solutions/determined/crates/determined-media/src/similarity.rs
solutions/determined/crates/determined-media/src/links.rs
solutions/determined/crates/determined-media/src/resolve.rs
solutions/determined/crates/determined-media/src/config.rs

← Back to Capabilities