Backing Up Digital Images

[neil poulsen]

I'm curious. What are the different methods that photographers are using to back up their digital images? We know that properly processed negatives can last for decades. But, digital presents it's own problems, when it comes to long term storage.

I've heard of a number of methods:

1. Back up on a hard disk drive, and shelf the drive when it's full and replace it with a new hard disk drive.
2. Same as the first method, except use two hard disk drives at one time configured in a raid such that, if one goes, the information is still retrievable from the other.
3. Use CD's to back up digital images. (Or a pair of CD's for reduncancy.)
4. Use DVD's to back up images, because they hold more.

Some comments of what I've heard, keeping in mind that we hear a lot. But, is it all true?

DVD's are not reliable. Sure, they hold more, but that's all that much more to lose if one fails.

CD's are more reliable than DVD's. After checking around, I've been hearing that TDK CD's are the best of the commonly available CD's. One sales person with whom I spoke said that Sony uses TDK CD's for their own software, even though Sony themselves produce a high end CD. Our local ProPhoto Supply has gold based CD's that are supposed to be super reliable. Price is about a dollar per gigabyte.

Using hard disk drives is reasonable, in that the IDE standard by which they communicate with computers is not expected to change. Sure, hard disk drives all fail eventually. But as a backup, they only get used "once", and then they're shelved. Single hard disk drives cost less than a dollar per gigabyte.

As a question, it would seem reasonable that some writers write more reliably than others. So, what are the best drives for writing CD's and/or DVD's? Does the fact that Seagate provide 5-year warrenties on their hard disk drives where others don't mean that their's are the most reliable?

I have images on CD's, some on DVD's, some on hard disk drives, etc. But, I want to standardize on a method of archive that will be reliable. So, I thought it would be worthwhile to ask how others are addresssing this issue.

[Frank Petronio]

I have two 500 gb drives that I manually copy everything too, and occassionally I will toss stuff that I don't want to keep. I don't do huge gb scans, but I figure that I use double the storage space every year -- with scans, work, and raw digital I did about 200 gb in 2006. In 1995 I did >2 gb. Plot the curve...

Everything up until 2005 is on CD/DVD, and all the best/important earlier stuff is also on the HDs. But I don't use DVDs for regular back up anymore, although I do burn them for projects and probably will burn them if I assemble distinct portfolios or sets.

I suspect that we should just count on buying a new $350 drive -- of whatever size $350 buys -- from now until eternity, every six months to a year, and cycling through them. I gave my old 250 gb HD to my daughter, leaving 100 gb of "essential house fire items" onboard so it is off site and she can store her ridiculous iTunes library.

I can't imagine an online storage medium being competitive price-wise yet, but it would be a nice additional option given that a productive full-time pro photographer should easily be doing at least 500 gb per year of new work... But I would still want a local HD for backup, with the online service as a secondary.

But please prove me wrong. Is there some industrial web server company that would allow photographers to ftp hundreds of GBs onto live HDs for a reasonable price?

I can't help but imagine that such an online service is coming, but I don't think it has gotten competitive price-wise yet...? Yet I would still fear a competitively cheap place might go out of business overnight. Which is why I would only see online storage as part of a larger strategy, at least until it matures.

[Marko]

It's not just about backing up photographs, same concerns are valid for any data we generate using our computers.

Tapes and any other open magnetic media is available and relatively affordable, but it is slow, cumbersome and prone to physical defects such as streching, flaking and (de)magnetization.

CD's are standard and cheap, but their standard capacity is progressively diminishing compared to rising file sizes. Besides, they are prone to oxidation.

DVDs can hold much more data, but they are still not standardized and are also prone to oxidation. Any error or inability to read has a potential to disable more data than a CD.

Hard disks, especially external units that connect through USB or Firewire as relatively universal, relatively fast and reasonably fast protocols for data transfer, with low current per-GB prices that are only going down, seems like a rather reasonable compromise. There are potential drawbacks too, since this is also a magnetic medium, although an enclosed one, that relies on mechanical mechanism and electrical connections. Any of those subsystems can fail at any time and thus take away all the data on the disk. The way to guard against that is redundancy and rotation - have at least two identical copies that get copied futher - rotated - onto new units every once in a while.

Online storage may look enticing to users who would rather not be bothered with technicalities, but this is in fact entrusting one's data to some remote, corporate entity that can either experience service and connectivity failures for a number of reasons, data loss or even going out of business. Any of those disruptions would be absolutely out of any user's control.

In short, there is no single, most reliable, most affordable, fastest method. There are only various compromises that may or may not be the most appropriate to each individual situation.

[QT Luong]

I used CDs before I realized how much time I was wasting. I skipped DVDs and went straight to HDDs. Now that I have a dozen primary external HDDs in capacities from 250GB to 500GB (and for each of them 1 or 2 backups), I just cannot see which other current method would be efficient enough. Even just for large data transfers, I just have HDDs shipped. The Seagate are very good drives that tend to run cooler than others. Other manufacturers have several lines of HDDs with varying warranties periods from 1 to 5 years within the same manufacturer. The 5-year warrantied drives are generally considered to be premium quality.

[claudiocambon]

Here is my sense of what to do:

Hard drives are considered more archival than disks, but only if you have a file backed up on at least 2 different drives. Having a file on only one drive is asking for it, as I found out the hard way 18 months ago when a conflict between my EOS 5D and Lexar cards caused my external Fantom Drive to physically break in 2 hours, (who knows how that happened? ) resulting in what Drivesavers 200 dollars later declared a full and total loss. (I had most everything on disk, as it was all recent stuff from the prior six months, which fortunately hadn't failed.) Disk shelf-life is not predictable, and should only be used as a 3rd back-up.

I have been told by many people to use Hitachi or IBM drives, as they perform better, and to avoid Western Digital drives, which fail more often.

A friend has a hot swappable set-up that lets you pop drives in and out; I haven't seen it yet, but it looks to be the wave of the future. That way the individual drives cost a lot less, because you aren't buying the shell, just the drive.

We should probably view drives as an continuous, incremental cost, almost like a film storage expense (print files and Beseler plastic folders?!), and will end up with shelves of them instead of negative binders. Because we outlay so much for the digital camera and the computer, I think we tend to be stingy about other expenses like hard drives because, gosh darn it, digital is supposed to be free from thereon out, but we just have to buy them, it seems.

Apparently drives also need to be run once every few months, as if they were a car.
Then there is the related issue of keeping files up to date with the latest versions of whatever computer program they need to read them. Not sure that that applies to RAW files, but it does to anything formatted in Photoshop, for example.

[Carsten Wolff]

Interestingly, valuable digital files can also be archived on sheetfilm:
Check out www chromacolour com au/digital htm

)

(BTW: I'm not associated with them in any way; just a customer......)

[neil poulsen]

[QUOTE=Marko;215775] CD's are standard and cheap, but their standard capacity is progressively diminishing compared to rising file sizes. Besides, they are prone to oxidation.QUOTE]

I'm not familiar with oxidation as it applies to CD's. Being completely naive, I would think that the plastic covering on CD's wound prevent oxidation. I wonder if that's one of the characteristics that separates good from bad CD's, the rate at which they oxidate?

Does anyone know of any serious data available on accelerated testing for storage media?

It's interesting, I wonder how the cost of film and processing compares with the costs of an archiving system based on redundancy and rotation. Once one has the equipment, is digital less expensive, or has one merely transferred those costs to the future? Purchasing film and processing is fixed. Costs of archiving the same images digitally appears to be perpetual.

[Marko]

I have been told by many people to use Hitachi or IBM drives, as they perform better, and to avoid Western Digital drives, which fail more often.

It's a toss of the dice. I've had IBM drives fail twice on me, one of those incidents was really fatal, taking the entire motherboard with it. Never had one with a WD or a Maxtor. Yet.

Come to think of it, never had a failure with Seagate either, but that's probably because I tend to avoid them ever since I had an opportunity to dissassemble two of their HDs which died on someone else. The sight of several patch wires running down the controller boards as if someone decided to do a last minute bug fix or something like that was not a pretty one to behold.

So, like QT said, look for the warranty - the longer, the better.

A friend has a hot swappable set-up that lets you pop drives in and out; I haven't seen it yet, but it looks to be the wave of the future. That way the individual drives cost a lot less, because you aren't buying the shell, just the drive.

You can have third-party encasings for $50 or less these days. It is a cost, but not a huge one.

We should probably view drives as an continuous, incremental cost, almost like a film storage expense (print files and Beseler plastic folders?!), and will end up with shelves of them instead of negative binders. Because we outlay so much for the digital camera and the computer, I think we tend to be stingy about other expenses like hard drives because, gosh darn it, digital is supposed to be free from thereon out, but we just have to buy them, it seems.

Apparently drives also need to be run once every few months, as if they were a car.

Digital was supposed to dispense with film and processing costs, not all costs associated with photography. As long as you want to produce a paper print, you'll need a machinery with which to do it (computer and printer vs. enlarger and processor) and the proper way to handle it after it is produced (archival and/or framing).

Many people complain that computer costs, including training for using them, are prohibitive, but they forget that computers are multi-purpose equipment. The same computer used for processing images can also be used to run all other business task, finances, correspondence and just plain fun. Those costs should therefore be spread out accordingly.

After all, you buy your car and then you keep paying for maintenance and gas, even though the initial cost was at least on the order of magnitute larger than your computer and your camera combined, don't you?

Life is an expensive proposition, but everything is more or less worth what you pay for it.

Then there is the related issue of keeping files up to date with the latest versions of whatever computer program they need to read them. Not sure that that applies to RAW files, but it does to anything formatted in Photoshop, for example.

RAW files are proprietary for each manufacturer. Once the OEM decides to pull the plug, it's gone. Photoshop is at least an industry-wide standard and they did come up with a universal open standard - DNG. I convert all my RAW files to DNG and keep both versions. The images I think are worth individual attention I also save in layered PSD and TIFF formats as an additional insurance.

When the new version of Photoshop comes out, I read the changes carefully and if there are any major changes to the format and/or file handling, I run a few spot-tests and then decide if I should run a large scale batch conversion. Same procedure essentially as with all the other major software packages, such as MS Office.

[Brian Ellis]

I'm not a pro so I don't have to worry about stock submissions over a long period of time or reorders or things like that. So I don't have the same needs as a pro. But FWIW I have duplicate backups, one to the external hard drive and and one to a DVD. I buy those gold DVDs from Delkin (the ones that are "archival" and will last 300 years - sure). I used to buy gold CDs but began running out of space. The DVDs are very slow and cumbersome to use at least on my system so they're strictly emergency backups, to be accessed only if all else fails. When I want to find an old photograph I go to the external hard drives. This is a very easy system but not as elaborate as some people use.

[Kirk Keyes]

Neil -

RAID 5.

RAID - redundand array of independant disks. There are a couple of commonly used definitions for RAID levels, but the ones you may be interested in for backing up data are:
RAID 1 - mirrored disks. You could have two disks an copy data to both disks so that if there is a failure of one disk, you still have an identical copy of the disk for recovery.

RAID 5- multiple disks with data "striped" across the disks, but with "parity". The striping takes a file and splits it into segements, and writes some of that file on one disk, and some of it on another disk, and maybe some of it on a 3rd disk. This is what RAID 0 does, and it is done for speed of data access, not backup because if one disk fails, you loose part of the file. What RAID 5 adds to the striping is the parity - it has each stripe mirrored on another disk. RAID 5 is often set up with 5 disks, so if you have a file split into 4 segments, the first part could be on disk 1 and 2, the second part would be on disk 2 and 3, the third part would be on disk 3 and 4, and so on. SO if one disk failed, say disk 2, you had the first and second segment written on it, but they were also written on disks 1 and 3 repectively. When the disk fails, the system informs you that you had a disk failure, you replace that disk and the system can then rebuild the array of backup data from disks 1 and 3 and restore what whad on disk 2. It's pretty slick.

Go to a local shop that knows linux, have them set up computer with a RAID 5 disk array, and then have that computer mount the disks on your regular desktop computer and use "rsync" and "cron" to automatically copy data from your desktop computer onto the RAID5 computer. Rsync is cool because it can copy only the differences in a file to update a file that has changed, so it cuts down on backup time and also network traffic on your LAN. Cron is the program that unixes use to schedule household chore type programs. Use samba to network to your other computers.

For the RAID 5 array, you need to have disks that are the same size, as it will set up the array to be based on the size of the smalled capacity disk in the array. Also, the size of the array is (N-1)*C where N= the number of disks in the array and C= capacity of smallest disk in array. So if you had five 250 gig disks in a RAID 5 array, your storage capacity would be (5-1)*250 = 1000 gigs.

It is probably the best way to set up an "on-the-fly" backup system that requires minimal attention. You still probably want to combine this with off site storage, and I would use hard disks for this as well. Get a couple of big drives, copy from the RAID onto them, take one off site, and use then swap those disks on a regular basis.

DVD and especially CDs are going to be a big waste of your time. And your time is valuable.

Since I was poor and didn't want to buy so many disks, I made a system from one of my older computers that used a scheme like this running with 2 large disks, not using an actual RAID card but using rsync to mirror data from one disk to another. It was totally transparent - I had it access my desktop computer every 4 hours and copy whatever changed. It ran for years and then I tried to upgrade from Red Hat 7 to Fedora 2 and something broke in samba and I haven't gotten around to fixing it.... I was actually planning on working on it this Sunday.