PDA

View Full Version : Managing Large Files



Hugh Sakols
17-Nov-2006, 08:56
My scans range in size from about 270 MB (6x6)to over 700 MB (6x9 with layers). I have two external hard drives and try to back up all worthwhile files. I'm curious how others manage their large files making sure that they have copies in case one hard drive dies. The big space eater is saving photoshop files that contain layers. Maybe I should just save the original tiff file and the photoshop layers files. If I loose the psd file, I can always go back to the tiff. I'm curious how others manage their humungous files?

Bruce Watson
17-Nov-2006, 09:11
I'm curious how others manage their humungous files?

I use DVDs. When I get done processing an image I dump all the associated files to DVD then delete them from my hard drive. This still gives me double backup - the DVD and the original negative, both stored following archival standards. With those two and a portfolio print I think I'm doing about as well as I can.

BTW, I don't use any file compression. Algorithms like LZ are relatively fragile IIRC. A one bit error can make an LZ compressed file unreadable. And don't write on CDs or DVDs with standard Sharpies - use markers designed for that use (water based, not solvent based inks)

Henry Ambrose
17-Nov-2006, 09:57
Multiple hard drives, well maintained with one drive/copy stored off-site pretty well takes care of anything that might happen. "Well maintained" means that you transfer your files off to new drives every so often, maybe every 2-3 years.

Imagine that you had an old SCSI drive with archives but no SCSI card on your current machine. (not that there's anything wrong with SCSI) This is just an illustration of difficulties that might arise if you don't keep up with the times. The other much worse scenario might be a hard dirve that been sitting un-powered for several years and fails when powered on or shortly thereafter. Hard drives are mechanical devices and who knows what might happen while it sits idle? Would you store a car for 3 years and then expect it to start right up when you called for it? (of course I have some old hard drives that are much much older than three years that still run fine but I would not depend on them in a critical situation)

I have a few CDs that are failing. I started using CDs for archiving around 1994 and used the very best media I could buy, but still a few failures. I found these last year when responding to a stock request. I have not even checked most of my archive because of the time factor. With something like 700-800 CDs it'd take a long time to check or transfer all that info. But then its mostly ephemeral commercial work that is long forgotten by me and the clients. If I have a few CDs fail I'll live through it.

I think optical media is fine but its still vital (maybe more so) to keep your files migrated. The estimated life span of any media shouldn't be taken as a guarantee. Not even close. And it can add up to more work and more expense than hard drives as cheap as HDs are per MB these days.

All that said, if you are scanning there is always the film to fall back on - you can make another scan at any time assuming the film is still with you. Digital captures are another story - you really don't have anything to fall back on if you lose the files or they are damaged.

I say its a balancing act between the real worth of your files and the possibility or the aggravation of re-scanning if something did get lost. If it really, really matters then carefully maintained multiple copies with offsite storage would be my pick.

Jack Flesher
17-Nov-2006, 09:59
I find that drives are cheap relative to the value of my files. I use a double mirroring, or software RAID 1-1, meaning I have a working drive with all of my images, a direct copy of that drive, and then a second redundant back-up of that drive which is stored offsite and is backed up monthly or after any significant shoot. This is by far the most fail-safe method of back-up if using mechanical drives. I buy 500G SATA drives in pairs when on sale, the last set were $189 each. Once a 500G drive is full, I relegate it to my historical image drive set and only keep that historical and a single offsite back-up. So only the active drive I am still adding images to is double-backed up and requires three drives; the historical images then require only 2 drives. In this fashion, I have an active 1 TB image array with 100% redundant back-up using a total of 5 drives for about $900.

QT Luong
17-Nov-2006, 10:09
I also use and recommend the method described by Jack, except that I synchronize manually instead of using RAID. DVDs are too time consuming to use and do not offer real price nor safety advantages.

Ed Richards
17-Nov-2006, 10:11
As lots of folks learned with Katrina, an off site copy is really important. On a more mundane level, house fires are not that rare, and a burgler will probably grab every external drive he can find. You can use a program like foldershare to automatically mirror to a remote computer. It can be an old cheap one, just so long as it can connect to the interent and can support external drives. That way you get an instant offsite backup as you work.

Ted Harris
17-Nov-2006, 10:22
I,too, use the multiple hard drive method. I make archival storage drives using 250 gb dries that I move into and out of an external case. When one is full I take it out and put it in storage. With these 'raw' drives now available for well under $100 they are the safest and cheapest way to go.

Jack Flesher
17-Nov-2006, 10:57
I should have clarified my software: I use is mirror-folder which is not a true raid, though I set it to mirror and synch the drives automatically so it pretty much works just like a RAID 1-1...

Greg Miller
17-Nov-2006, 11:01
I use 2 local external hard drives (1 is trhe master, the second is an automatically created backup). And 1 3rd drive offsite. The local backup and the offsite backup are rotated periodically in order to keep the offsite copy reasonably fresh.

I store everything as a psd file with layers. Hard drives are to cheap risk losing all the layer infromation.

I would not trust DVDs to be a primary or backup source for my files.

MIke Sherck
17-Nov-2006, 12:28
I'm an IT guy and have been for 25 years. The very *thought* of any digital technology meeting even the colloquial meaning of "archival" is intensely amusing. All digital storage technology is electro-mechanical in nature and, as will every machine sooner or later, will fail. MTBF (mean time between failure) statistics are, at best, averages: your hard drive/CD/DVD/Tape may exceed that statistic by some arbitrary amount -- or fail sooner. The only certainty is that it will fail.

It has been pointed out before that all technology becomes obsolescent sooner or later. Information stored on 8" floppy disks is essentially lost outside of specialized recovery services now, and it was once in wide use (assuming of course that any re-recordable magnetic medium will retain information for 25+ years -- a dubious prospect.) Long-ago "archived" data on optical disks may or may not degrade; it is certain that the drives to read them in are scarce and will become more scarce. All digital recording media face the same problem on relatively short time scales.

In other words, there is no such thing as "archival" backup for digital media. The very best you can do is to make multiple error-resistant copies and re-copy them to current media periodically. For today, I'd put important information on RAID-1 or RAID-5 disk arrays with (if possible) optical off-site storage and plan on replacing all media at least every five years. Just remember, though, that missing replacement cycles is playing with fire; miss two and you're running serious risks of not being able to recover the data outside of expensive data recovery specialists. Miss three and it's gone in all likelihood. That's a 15 year lifespan; more than most business records need and longer than most personal needs extend. Thus, it is unlikely to expand very much: the market doesn't need storage for a period longer than that outside of very specialized areas (scientific data, for example.)

The saving grace of all this is that digital media keeps expanding quickly; a 100 MB file used to be considered enormous and storage media could only accomidate a few such files. Now, it's quick and simple to copy that and a cheap DVD will hold many of them. The 100 GB storage problem of today will be less of a problem tomorrow and trivial in 5-10 years as media capacity expands and access gets faster. So even as new technologies create larger files, managing the relatively small older files become less of a time sink. Thus, I'd save the raw files: the manipulations can be re-created but once a bit is gone, it's gone for good.

Mike

Gordon Moat
17-Nov-2006, 12:49
Often my large scans might come from a service bureau, or a pro lab; they seem to use a few of the gold CD-Rs for that purpose. However, I always make at least one copy of the scan onto an MO disk. My current MO drive is a Fujitsu. The cost per MB is not that great, but these have been very reliable for me. I even have one with extreme abuse that I still use (spilled soda, then washed with soapy water in sink, then let sit in the sun for about a year, which collected dust, then another wash, rinse, and sun drying, then I read the data which had been on it for a few years . . . that sort of abuse).

Regardless of type of media used, it would still be possible to have a software issue making a file unreadable. The back-up for a film scan is the original film, though that adds the cost of having another scan done, or time to do your own scan. As others indicate, don't take anything for granted.

Ciao!

Gordon Moat
A G Studio (http://www.allgstudio.com)

Kirk Gittings
21-Nov-2006, 11:07
I am generating almost 300 GB a year of image files so this is an important issue for me. As my third line of backup, I am burning duplicate DVD's. Any thoughts on the most likely archival disks, formats etc. for dvds that will give me the most long term mileage?

Gordon Moat
21-Nov-2006, 11:35
Hello Kirk,

It sounds to me that you might want to consider getting a dedicated server for all those files. Something with redundant drives. You could either go with a monthly service on something like that, or run your own in-house smaller system. The other benefit is easier access to previous files. I know a few pros now running servers, or having them run for them, after several years of various disk burners or external drives. In the long run the reliability and cost per GB are reasonable. Might be something for you to investigate.

Ciao!

Gordon Moat
A G Studio (http://www.allgstudio.com)

David_Senesac
24-Nov-2006, 21:21
I have a set of multiple Taiyo Yuden DVD's periodically archived for each of my marketed image's master .PSD file. At least one set is intended to be located off site. Then just one each for each TIF print file. If I lose the later I can always regenerate it from the .PSD without much work. If I lose the .PSD I would have to work from the CD/DVD media containing the raw drum scans that would certainly result in a different image than my printed masters thus requiring expensive reprinting of a master. If I lost all the above, I would have to have another drum scan and master print made plus considerable effort and time expense. In case of fire or theft, at least my offsite DVD's guarantee a significant recovery. I've also considered using backup drives that many are now using and might do that too when I upgrade my hardware and OS. Several of my deacde old backup CD's from the past have become flaky or unreadable plus given general computer obsolescence issues one ought to err on the conservative side when backing up valuable data. ...David

Graham Patterson
25-Nov-2006, 10:52
Fortunately I do not produce enough photographic output to scanned files to worry. My wife does, and at the moment she has a 1TB external system in mirror configuration as her primary backup for the 0.5 TB of drives in the G5. It still makes me nervous.

I am another person paid to maintain computer systems, and one of the key tenets about backup is to separate the data from the mechanics used to read it as far as possible. With hard drives used as storage, you could have data corruption and hardware failure involved. With tape and DVD you do have the option of replacing the drive if the media is sound. But big capacity tape systems are not cheap, though the incremental storage cost is reasonable.

With large drives at comparatively low prices (at least in the US), it is not especially hard to get a basic PC, slap in 4 500GB drives, run FreeNAS (software RAID network attached storage) from a USB device, and get 1.5TB of RAID 5 (3 data, 1 parity drive). That is still only one copy, though.

A decent way to duplicate the digital file back to film for archiving would be nice. It would weigh less and take up less space, too. 8-)

Daniel Geiger
25-Nov-2006, 22:33
I use a 1 TB NAS box in RAID5 (effective storage 750 GB), and have CD/DVDs off site (about 300). NAS is nice as you can use it cross platform with no problems; GB ethernet allows for pretty quick data transfer.

Tom Kershaw
8-Dec-2006, 04:55
Daniel,

What kind of Network Attached Storage are you using?

Tom