Managing Large Files

[Hugh Sakols]

My scans range in size from about 270 MB (6x6)to over 700 MB (6x9 with layers). I have two external hard drives and try to back up all worthwhile files. I'm curious how others manage their large files making sure that they have copies in case one hard drive dies. The big space eater is saving photoshop files that contain layers. Maybe I should just save the original tiff file and the photoshop layers files. If I loose the psd file, I can always go back to the tiff. I'm curious how others manage their humungous files?

[Bruce Watson]

I'm curious how others manage their humungous files?

I use DVDs. When I get done processing an image I dump all the associated files to DVD then delete them from my hard drive. This still gives me double backup - the DVD and the original negative, both stored following archival standards. With those two and a portfolio print I think I'm doing about as well as I can.

BTW, I don't use any file compression. Algorithms like LZ are relatively fragile IIRC. A one bit error can make an LZ compressed file unreadable. And don't write on CDs or DVDs with standard Sharpies - use markers designed for that use (water based, not solvent based inks)

[Henry Ambrose]

Multiple hard drives, well maintained with one drive/copy stored off-site pretty well takes care of anything that might happen. "Well maintained" means that you transfer your files off to new drives every so often, maybe every 2-3 years.

Imagine that you had an old SCSI drive with archives but no SCSI card on your current machine. (not that there's anything wrong with SCSI) This is just an illustration of difficulties that might arise if you don't keep up with the times. The other much worse scenario might be a hard dirve that been sitting un-powered for several years and fails when powered on or shortly thereafter. Hard drives are mechanical devices and who knows what might happen while it sits idle? Would you store a car for 3 years and then expect it to start right up when you called for it? (of course I have some old hard drives that are much much older than three years that still run fine but I would not depend on them in a critical situation)

I have a few CDs that are failing. I started using CDs for archiving around 1994 and used the very best media I could buy, but still a few failures. I found these last year when responding to a stock request. I have not even checked most of my archive because of the time factor. With something like 700-800 CDs it'd take a long time to check or transfer all that info. But then its mostly ephemeral commercial work that is long forgotten by me and the clients. If I have a few CDs fail I'll live through it.

I think optical media is fine but its still vital (maybe more so) to keep your files migrated. The estimated life span of any media shouldn't be taken as a guarantee. Not even close. And it can add up to more work and more expense than hard drives as cheap as HDs are per MB these days.

All that said, if you are scanning there is always the film to fall back on - you can make another scan at any time assuming the film is still with you. Digital captures are another story - you really don't have anything to fall back on if you lose the files or they are damaged.

I say its a balancing act between the real worth of your files and the possibility or the aggravation of re-scanning if something did get lost. If it really, really matters then carefully maintained multiple copies with offsite storage would be my pick.

[Jack Flesher]

I find that drives are cheap relative to the value of my files. I use a double mirroring, or software RAID 1-1, meaning I have a working drive with all of my images, a direct copy of that drive, and then a second redundant back-up of that drive which is stored offsite and is backed up monthly or after any significant shoot. This is by far the most fail-safe method of back-up if using mechanical drives. I buy 500G SATA drives in pairs when on sale, the last set were $189 each. Once a 500G drive is full, I relegate it to my historical image drive set and only keep that historical and a single offsite back-up. So only the active drive I am still adding images to is double-backed up and requires three drives; the historical images then require only 2 drives. In this fashion, I have an active 1 TB image array with 100% redundant back-up using a total of 5 drives for about $900.

[QT Luong]

I also use and recommend the method described by Jack, except that I synchronize manually instead of using RAID. DVDs are too time consuming to use and do not offer real price nor safety advantages.

[Ed Richards]

As lots of folks learned with Katrina, an off site copy is really important. On a more mundane level, house fires are not that rare, and a burgler will probably grab every external drive he can find. You can use a program like foldershare to automatically mirror to a remote computer. It can be an old cheap one, just so long as it can connect to the interent and can support external drives. That way you get an instant offsite backup as you work.

[Ted Harris]

I,too, use the multiple hard drive method. I make archival storage drives using 250 gb dries that I move into and out of an external case. When one is full I take it out and put it in storage. With these 'raw' drives now available for well under $100 they are the safest and cheapest way to go.

[Jack Flesher]

I should have clarified my software: I use is mirror-folder which is not a true raid, though I set it to mirror and synch the drives automatically so it pretty much works just like a RAID 1-1...

[Greg Miller]

I use 2 local external hard drives (1 is trhe master, the second is an automatically created backup). And 1 3rd drive offsite. The local backup and the offsite backup are rotated periodically in order to keep the offsite copy reasonably fresh.

I store everything as a psd file with layers. Hard drives are to cheap risk losing all the layer infromation.

I would not trust DVDs to be a primary or backup source for my files.

[MIke Sherck]

I'm an IT guy and have been for 25 years. The very *thought* of any digital technology meeting even the colloquial meaning of "archival" is intensely amusing. All digital storage technology is electro-mechanical in nature and, as will every machine sooner or later, will fail. MTBF (mean time between failure) statistics are, at best, averages: your hard drive/CD/DVD/Tape may exceed that statistic by some arbitrary amount -- or fail sooner. The only certainty is that it will fail.

It has been pointed out before that all technology becomes obsolescent sooner or later. Information stored on 8" floppy disks is essentially lost outside of specialized recovery services now, and it was once in wide use (assuming of course that any re-recordable magnetic medium will retain information for 25+ years -- a dubious prospect.) Long-ago "archived" data on optical disks may or may not degrade; it is certain that the drives to read them in are scarce and will become more scarce. All digital recording media face the same problem on relatively short time scales.

In other words, there is no such thing as "archival" backup for digital media. The very best you can do is to make multiple error-resistant copies and re-copy them to current media periodically. For today, I'd put important information on RAID-1 or RAID-5 disk arrays with (if possible) optical off-site storage and plan on replacing all media at least every five years. Just remember, though, that missing replacement cycles is playing with fire; miss two and you're running serious risks of not being able to recover the data outside of expensive data recovery specialists. Miss three and it's gone in all likelihood. That's a 15 year lifespan; more than most business records need and longer than most personal needs extend. Thus, it is unlikely to expand very much: the market doesn't need storage for a period longer than that outside of very specialized areas (scientific data, for example.)

The saving grace of all this is that digital media keeps expanding quickly; a 100 MB file used to be considered enormous and storage media could only accomidate a few such files. Now, it's quick and simple to copy that and a cheap DVD will hold many of them. The 100 GB storage problem of today will be less of a problem tomorrow and trivial in 5-10 years as media capacity expands and access gets faster. So even as new technologies create larger files, managing the relatively small older files become less of a time sink. Thus, I'd save the raw files: the manipulations can be re-created but once a bit is gone, it's gone for good.

Mike