find-duplicates

Find Duplicates

Surface duplicate files and quantify the reclaimable space. Default scope: a user-supplied path (e.g. ~/Downloads, ~/Pictures). Default tool: rmlint — fast, scriptable, generates a review-able shell script.

Inputs

1. Path — what to scan. Refuse to scan / or anything above $HOME unless user explicitly insists; the false-positive risk is too high (system files, package manager artefacts).
2. Min size (optional) — skip files smaller than this (default 1 MiB; raise to 100 MiB if the user only cares about large dupes).
3. Mode — report only (default) or generate prune script (writes a script the user reviews).

Tool selection

• rmlint if installed — preferred; emits rmlint.sh with reviewable per-file actions.
• Else jdupes (faster than fdupes) — falls back to fdupes if neither is present.
• If none installed: sudo apt install rmlint and proceed.

Run

rmlint:

rmlint --types=duplicates --size=<min>K \
       --output=sh:<scans_dir>/rmlint-<timestamp>.sh \
       --output=json:<scans_dir>/rmlint-<timestamp>.json \
       --output=summary \
       <path>

(<scans_dir> here = the benchmarks_dir from linux-system-optimisation's config, with a /space-audits/ subfolder. Create it if missing.)

jdupes / fdupes fallback:

jdupes -r -S -m <path>      # summary
jdupes -r -S <path> > <scans_dir>/jdupes-<timestamp>.txt

Report

• Total duplicate groups found.
• Total reclaimable bytes if all but one copy of each group is removed.
• Top 20 largest groups — show one representative path + total wasted bytes.
• Suspicious path patterns to flag (don't delete from these — likely intentional):
  • ~/.cache/* (regenerable; prune-dev-clutter handles this)
  • ~/snap/*/common/, ~/.var/app/* (Flatpak per-app stores)
  • */node_modules/* (every project legitimately has its own)
  • */.venv/*, */venv/* (likewise — prune-dev-clutter handles)
  • Anything under .git/objects/

Prune script

If user requested prune-script mode:

• For rmlint — the emitted rmlint.sh is already that. Print the path and tell the user to head it, edit if needed, then run.
• For jdupes — generate a similar script: keep the first file in each group, rm the rest, but comment out any line whose path matches the suspicious patterns above. User uncomments to confirm.

Never run the script automatically. The user reviews and runs it.

Notes

• Hashing is I/O heavy on first run; subsequent runs on the same tree are much faster (rmlint caches).
• For BTRFS / XFS, mention reflinks: rmlint --types=duplicates --keep-all-tagged --merge-directories can be configured to deduplicate via reflink instead of delete (zero-copy on supported FS). Out of scope for the default flow but worth surfacing for users with big duplicate sets they don't want to delete.