eliminate duplicates from hard drives

[lee nadel]

can someonesuggest how to eliminate duplicates files the files are images. thanks

[jp]

It's probably not worth the time.. Let's say you had 1TB of duplicate images which is a ton, since 1TB lasts me 1-2 years at my current rate of shooting and scanning. A new 10TB drive is $300; thus you're looking to conserve $30-40 worth of space at the most extreme.

[jon.oman]

If you are running windows 10 or 11, look at robocopy.

robocopy /? (this will give you some short help)

I do this:

robocopy J:\ K:\ /MIR


Here is a tutorial: https://adamtheautomator.com/robocopy/

Just make sure you back things up BEFORE you run this globally. Test things out first on a set of test directories. It gives you a lot of power, but you can delete things you do not want to if you are not careful!

[Jody_S]

There are utilities that display thumbnails and allow you to pick duplicates for deletion. I used one when I consolidated 10 years' worth of phone backups onto one hard drive. Can't recall which one I used though, and I tend to remove programs like this from my 'puter after use.

[HMG]

It's probably not worth the time.. Let's say you had 1TB of duplicate images which is a ton, since 1TB lasts me 1-2 years at my current rate of shooting and scanning. A new 10TB drive is $300; thus you're looking to conserve $30-40 worth of space at the most extreme.

But you then move the dupes to the new drive. And then, when that drive is replaced, to a new drive. So ad infinitum.

Though I'd be careful about how dupes are detected. File name? File name plus file size? Jpeg vs tiff?

[Graham Patterson]

This can be complicated - why do you have duplicates, and why is it important to know which files are duplicated.

Typically you get duplicates because you make a working copy, which may or may not be modified, you import the source files more than once, or you create a backup when you meant to create an archive (a backup is a duplicate of your working file set, while an archive is usually a set of files than no longer exist on your primary system).


I would take 'duplicate' to mean an exact copy, though with image files there can be many variations from the original, or just file format changes.

To test for duplicates you need to look at the file size, the file format (from the file internals, not the name), and probably something like an MD5 hash or file checksum which is highly likely unique to the binary file content. The file name and the location path are not reliable because pesky humans have a habit of changing them.

A lot of operating systems support links to directories or files. From a cursory scan these can appear to be duplicates, but actually point to the same file(s). Which is a potential problem if you are cleaning up. A lot of server operating systems support de-duplication by leaving links in place of copies. In theory the last reference to a file always points to the actual file.


I would suggest thinking about why you get duplicates, and getting that under control first. Then look at the archive and decide if you need to de-duplicate or organize it. Often the best way is to only extract and organize bit you need as you need them. If you have lots of files in deep and varied directory paths the work to sort it out in retrospect may not be warranted.

When I was working, I was managing servers with quite a bit of image duplication. Often these duplicates were intentional, because a project or publication kept a copy of the source material in the project directory, as well as any modified versions. This helped with archiving, as only the used material needed to be in the archive.

[lee nadel]

thanks for the advice lee

[domaz]

Lightroom (and probably other similar programs) can Import and Skip duplicates. You could leverage this I believe by telling Lightroom to import in Place, ignoring duplicates. Then once imported select all the imported files and have Lightroom move them to a new "clean" folder. Then delete your old folder full of duplicates once you verify everything is as expected.

[jp]

Lightroom (and probably other similar programs) can Import and Skip duplicates. You could leverage this I believe by telling Lightroom to import in Place, ignoring duplicates. Then once imported select all the imported files and have Lightroom move them to a new "clean" folder. Then delete your old folder full of duplicates once you verify everything is as expected.

This is a pretty clever idea!

[maltfalc]

Lightroom (and probably other similar programs) can Import and Skip duplicates. You could leverage this I believe by telling Lightroom to import in Place, ignoring duplicates. Then once imported select all the imported files and have Lightroom move them to a new "clean" folder. Then delete your old folder full of duplicates once you verify everything is as expected.

lightroom only skips previously imported files, not files that are identical to each other within an import.