Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: eliminate duplicates from hard drives

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1

    Join Date
    Jul 1999
    Posts
    71

    eliminate duplicates from hard drives

    can someonesuggest how to eliminate duplicates files the files are images. thanks

  2. #2
    jp's Avatar
    Join Date
    May 2009
    Location
    Maine
    Posts
    5,508

    Re: eliminate duplicates from hard drives

    It's probably not worth the time.. Let's say you had 1TB of duplicate images which is a ton, since 1TB lasts me 1-2 years at my current rate of shooting and scanning. A new 10TB drive is $300; thus you're looking to conserve $30-40 worth of space at the most extreme.

  3. #3

    Join Date
    Dec 2006
    Location
    Minnesota and Massachusetts, USA
    Posts
    587

    Re: eliminate duplicates from hard drives

    Quote Originally Posted by jp View Post
    It's probably not worth the time.. Let's say you had 1TB of duplicate images which is a ton, since 1TB lasts me 1-2 years at my current rate of shooting and scanning. A new 10TB drive is $300; thus you're looking to conserve $30-40 worth of space at the most extreme.
    But you then move the dupes to the new drive. And then, when that drive is replaced, to a new drive. So ad infinitum.

    Though I'd be careful about how dupes are detected. File name? File name plus file size? Jpeg vs tiff?

  4. #4

    Join Date
    Oct 2008
    Location
    Summerville, SC
    Posts
    1,821

    Re: eliminate duplicates from hard drives

    If you are running windows 10 or 11, look at robocopy.

    robocopy /? (this will give you some short help)

    I do this:

    robocopy J:\ K:\ /MIR


    Here is a tutorial: https://adamtheautomator.com/robocopy/

    Just make sure you back things up BEFORE you run this globally. Test things out first on a set of test directories. It gives you a lot of power, but you can delete things you do not want to if you are not careful!

  5. #5
    (Shrek)
    Join Date
    Mar 2011
    Location
    Montreal
    Posts
    2,012

    Re: eliminate duplicates from hard drives

    There are utilities that display thumbnails and allow you to pick duplicates for deletion. I used one when I consolidated 10 years' worth of phone backups onto one hard drive. Can't recall which one I used though, and I tend to remove programs like this from my 'puter after use.

  6. #6

    Join Date
    Mar 2004
    Location
    San Francisco Bay Area
    Posts
    425

    Re: eliminate duplicates from hard drives

    This can be complicated - why do you have duplicates, and why is it important to know which files are duplicated.

    Typically you get duplicates because you make a working copy, which may or may not be modified, you import the source files more than once, or you create a backup when you meant to create an archive (a backup is a duplicate of your working file set, while an archive is usually a set of files than no longer exist on your primary system).


    I would take 'duplicate' to mean an exact copy, though with image files there can be many variations from the original, or just file format changes.

    To test for duplicates you need to look at the file size, the file format (from the file internals, not the name), and probably something like an MD5 hash or file checksum which is highly likely unique to the binary file content. The file name and the location path are not reliable because pesky humans have a habit of changing them.

    A lot of operating systems support links to directories or files. From a cursory scan these can appear to be duplicates, but actually point to the same file(s). Which is a potential problem if you are cleaning up. A lot of server operating systems support de-duplication by leaving links in place of copies. In theory the last reference to a file always points to the actual file.


    I would suggest thinking about why you get duplicates, and getting that under control first. Then look at the archive and decide if you need to de-duplicate or organize it. Often the best way is to only extract and organize bit you need as you need them. If you have lots of files in deep and varied directory paths the work to sort it out in retrospect may not be warranted.

    When I was working, I was managing servers with quite a bit of image duplication. Often these duplicates were intentional, because a project or publication kept a copy of the source material in the project directory, as well as any modified versions. This helped with archiving, as only the used material needed to be in the archive.

  7. #7

    Join Date
    Jul 1999
    Posts
    71

    Re: eliminate duplicates from hard drives

    thanks for the advice lee

  8. #8

    Join Date
    Apr 2009
    Location
    Seattle area, WA
    Posts
    1,107

    Re: eliminate duplicates from hard drives

    Lightroom (and probably other similar programs) can Import and Skip duplicates. You could leverage this I believe by telling Lightroom to import in Place, ignoring duplicates. Then once imported select all the imported files and have Lightroom move them to a new "clean" folder. Then delete your old folder full of duplicates once you verify everything is as expected.

  9. #9
    jp's Avatar
    Join Date
    May 2009
    Location
    Maine
    Posts
    5,508

    Re: eliminate duplicates from hard drives

    Quote Originally Posted by domaz View Post
    Lightroom (and probably other similar programs) can Import and Skip duplicates. You could leverage this I believe by telling Lightroom to import in Place, ignoring duplicates. Then once imported select all the imported files and have Lightroom move them to a new "clean" folder. Then delete your old folder full of duplicates once you verify everything is as expected.
    This is a pretty clever idea!

  10. #10

    Join Date
    Nov 2017
    Posts
    290

    Re: eliminate duplicates from hard drives


Similar Threads

  1. Black points in copies as I eliminate them?
    By Nino Grangetto in forum Darkroom: Film, Processing & Printing
    Replies: 7
    Last Post: 1-Feb-2008, 13:06
  2. How Do You Eliminate Scratches
    By howard s in forum Style & Technique
    Replies: 8
    Last Post: 16-Jan-2002, 19:19
  3. Rebuilding a Tachihara to eliminate play
    By Tim Kimbler in forum Cameras & Camera Accessories
    Replies: 2
    Last Post: 6-Jul-1998, 14:22

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •