endless external HD shuffle – RAID 1, 5, or RAID unecessary?

[Chester McCheeserton]

I've been running two separate external drives and manually trying to keep everything important on both for many years which has worked fine.
When I'm working with big drum scans from large format negs, the layered PSBs frequently exceed 10 GB, I take steps like having 16 gb of ram, emptying off my desktop, and setting the scratch disk to dedicated empty 500 gb external ssd. I do all this on a 2012 macbook pro, with an internal SSD 250 gb that I try to leave at least 50-80 gigs free on.

Recently I've been shooting a lot with an A7r2 and gathering quite a but 4k video footage in addition to piling up more 5x7 film drum scans, and my 4 and 5 tb externals are both nearly full again.

My question is whether upgrading to a RAID with dual 8 or 10 gb drives is worth it? Has anyone ever had a drive fail where the RAID totally saved the day? Obviously I would still need a third drive to keep offsite in case of theft, fire, etc, but I can't decide if dropping the dough for a RAID set up is a smart decision. I guess I'd like to not have worry about upgrading for 3-5 years instead of doing it again in a year....for those that do think RAID is the way to go, any thoughts on 1 vs 5?

On the other hand, none of the folks I know who work on pictures seriously with computers bother with the whole RAID thing...it certainly would be cheaper to just buy two of the least expensive seagate 8 tb models and just keep doing what I've been doing...Any thoughts from those with experience would be much welcome...

Gracias,
Chester McCh

[jpheneger]

RAID has saved my bacon on three separate occasions. Having an enclosure with 4 drives in RAID 5 configuration makes sure I never lose data from a single drive failure. I don't have to swap drives around and it's all taken care of when a drive gets replaced. Highly recommended.

[williaty]

RAID is not a backup.

Repeat it with me:

RAID IS NOT A BACKUP


The purposes of RAID are 1) High availability even when a drive in the array fails, is replaced, and rebuilt (imagine being a business where your ecommerce DB lives on a RAID. If a drive fails, you don't go out of business while it gets replaced) and 2) Greater read or write speeds due to mirroring, striping, etc. The first purpose, frankly, is just about dead as drive sizes are reaching a point where rebuilds can take so long that the likelyhood of a 2nd drive failing while the 1st is getting rebuilt is starting to become uncomfortably high. For that matter, SSDs are starting to eat into the 2nd justification as well as their prices drop and capacities grow.

The reason RAID isn't a backup is that it doesn't provide data integrity against some of the most common problems. It provides a little redundancy against a single-drive hardware failure (good luck if your RAID card itself fails though!). It provides no protection against you mistakenly deleting something. It provides no protection against you getting a virus or becoming the victim of ransomware. It provides no protection against lightning. It provides no protection against a fire. It provides no protection against a flood, tornado, earthquake, or jealous ex. You get the picture. Well, if you're relying on RAID as a backup, you won't have the picture very long, I suppose.

For something as important as our photos are to us, the very minimum acceptable backup strategy is to have the copy on your workstation that you're working with, an on-network clone of your photo archive that you frequently sync, and a minimum of 2 clones available for off-site backup. I say two because you need to make a current clone, physically take it to your safe location (like a safe deposit box at the bank), store it there, retrieve the other clone you're using as an off site, drive it back home, and sync it. If you only have one off-site backup, you have 0 off-site backups while you've brought that one home to sync it.

Personally, I also make non-writable backups on a regular basis (read: burn a Blu-Ray disc). If the worst happens and I'm hit by particularly nasty ransomware that hides long enough to get my writable backups before it locks up and demands money, the only recovery I'll have is the stuff I burned to BRDs because, after all, the ransomware can't encrypt a disc it can't write to.

[jpheneger]

I am by no means suggesting that RAID is an alternative for backup. But RAID is a valuable part of my backup solution. My setup is this:

Copy from SD card to directly attached SSD (manual)
Copy from SSD to NAS (RAID 5) (automatic)
Copy from NAS to AWS S3 storage (automatic)
Reformat SD card when it goes back into camera (typically not next shoot).

With this arrangement I generally have 4 copies of the images with one being offsite. I have automated the process with the help of my NAS (insert Synology plug) and a few shell scripts.

I've lost images due to single drive failure in the past. Given how often I'm using my drives, I am most at risk for single drive failure - RAID is made for that. Yes there are problems with replicating a very large drive, but that's why i also backup to S3 - which has it's own issues. I'm not looking forward to the day I need to download 2.5TB from S3 to rebuild my entire NAS.

But if it's a question of no RAID vs. RAID I'll choose to use RAID every time.

[Chester McCheeserton]

Thanks guys. Did some research and looks like to get both the speed and redundancy benefit from a raid setup I'd need to go with at least 3 and ideally more drives, out of my budget for now. Just got 2 used 8tb drives and will keep them separate for now. Although I'll be dealing with this again in a year or 2.

Advice about raid not being back up noted.

[Corran]

Forget RAID. Go JBOD.

This is what I use, and it's a great, easy-to-use JBOD setup. The newest update is even easier. It's called unRAID.
https://lime-technology.com/download/

I have a 12TB server that has run almost nonstop for several years now. Highly recommended. I use a 4TB work drive and then store all RAW/TIFF files on the server once editing is finished.

[EdWorkman]

Folks with really important data back up off site
Fires and lightning are mostly avoided
But what is civilization going to do when the Big Solar Flare occurs.
I had saved some scans on CDs - very small capacity these days, and a pain in a pileful
I bought a pack of DVD disks ........

[Ted Baker]

I bought a pack of DVD disks ........

Creating multiple exact copies, can increase the reliability of this approach for very important stuff considerably. i.e. if the disk goes bad. You will need a solution to combine the two or more copies, however.

[wilderness]

Considering the large file sizes that are discussed here, discs for BU (even the newer Triple Layer Blu Rays (100gig) are not viable.
CD 650 meg
DVD Single Layer 4.7 gig
DVD Dual 8.5 gig
Blu Ray Single Layer 25 gig
Blu Ray Dual Layer 50 gig
Blu Ray Triple Layer 100 gig
I've progressed to Triple Layer Blu Ray and the discs are not cheap.
Incremental BU's are a must procedure for any use of discs, however incremental in most instances makes restoring a nightmare.
I've had multiple major failures with a name brand Dual Layer DVD's (think Ella Fitzgerald) after short periods of time. So much that I won't even buy that brand of disc any longer. Otherwise, my disc failures over a twenty year period have been minimal.
There's only a handful of companies that make discs and everybody else puts their name on them.
Been scanning A120 negatives for four months now at 2400DPI and my monthly BU's are running just over 25 gig. (TIF's may be downsized and restore to their original size, however who wants the extra work of resizing and saving two copies of a file.)

[Steven Ruttenberg]

I use 8 2 GB hard drives in 4 2 disco back ups. I have an anolog pair, digital pair, archive pair and a miscellaneous pair. Have not taken to of site storage yet maybe I will soon, but also looking at a fire proof safe stored in garage that I backup to over my network. My 4x5 images end up being upwards of 10-20gb in size for working files. Digital can get up there.

I need the fire proof safe anyway to store my negatives and prints as well so I should get started. Dvds, blue rays won't do it for me. I also don't use raid. I just have automatic copies made every night of each pair. I will soon need to go to a 6!TB drive pair for my 4x5 work.