Skip to content
sdlc

Vault image asset substrate

State: open/building · Kind: module · Audience: system

Summary

Section titled “Summary”

  • Shared Rust library that selects a vault’s non-note files (images), decodes them, computes perceptual signatures and thumbnails, and caches the results, keyed by path, size and mtime.

  • Serves the ocr, media, photos, macos and platform crates so they share one walk and decode pipeline.

Statement

Section titled “Statement”

A vault holds images and PDFs next to its markdown. This crate walks those files, decodes the images to EXIF-rotated sRGB pixels, hashes them, computes perceptual signatures, compares them, and makes thumbnails. It is a library, not a port: no determined-core port is implemented here. The algorithms reproduce the node originals exactly where they are pure functions (dHash bit order, histogram binning, cache keys). Pixel values differ from the node side because decoding uses the image crate instead of libvips, so near-duplicate grouping is compared at a Hamming threshold, not by hash value.

What it provides

Section titled “What it provides”
  • Vault asset walk: extension allowlist, dot-segment skip, substring exclude, min_bytes, cross-root dedupe, deterministic sort, built on vault-fs
  • Decoder with an ExtendedDecoder boundary for HEIC/AVIF
  • dHash-64, dHash-256 and 64-bin colour histogram signatures from one decode
  • Hamming distance, histogram-intersection colour distance and a blended similarity score
  • Thumbnail and face-crop generation with sha1-keyed on-disk caches
  • Signature cache with a usability predicate and a backend stamp
  • sha256 and sha1 content hashing

Underlying implementation

Section titled “Underlying implementation”
  • lib.rs re-exports the walk, decode, signature, distance, thumbs, cache and hash modules. walk delegates to vault-fs.
  • hash.rs holds sha256_hex, sha256_file and sha1_hex.

Locations

Section titled “Locations”
  • solutions/determined/crates/determined-attachments/
  • solutions/determined/crates/determined-attachments/src/lib.rs
  • solutions/determined/crates/determined-attachments/src/walk.rs#walk_assets
  • solutions/determined/crates/determined-attachments/src/decode.rs#Decoder
  • solutions/determined/crates/determined-attachments/src/signature.rs#signature_of
  • solutions/determined/crates/determined-attachments/src/distance.rs#similarity_score
  • solutions/determined/crates/determined-attachments/src/thumbs.rs
  • solutions/determined/crates/determined-attachments/src/cache.rs#HashCache
  • solutions/determined/crates/determined-attachments/src/hash.rs#sha256_hex

← Back to Capabilities