endless external HD shuffle – RAID 1, 5, or RAID unecessary?

[Chester McCheeserton]

I've been running two separate external drives and manually trying to keep everything important on both for many years which has worked fine.
When I'm working with big drum scans from large format negs, the layered PSBs frequently exceed 10 GB, I take steps like having 16 gb of ram, emptying off my desktop, and setting the scratch disk to dedicated empty 500 gb external ssd. I do all this on a 2012 macbook pro, with an internal SSD 250 gb that I try to leave at least 50-80 gigs free on.

Recently I've been shooting a lot with an A7r2 and gathering quite a but 4k video footage in addition to piling up more 5x7 film drum scans, and my 4 and 5 tb externals are both nearly full again.

My question is whether upgrading to a RAID with dual 8 or 10 gb drives is worth it? Has anyone ever had a drive fail where the RAID totally saved the day? Obviously I would still need a third drive to keep offsite in case of theft, fire, etc, but I can't decide if dropping the dough for a RAID set up is a smart decision. I guess I'd like to not have worry about upgrading for 3-5 years instead of doing it again in a year....for those that do think RAID is the way to go, any thoughts on 1 vs 5?

On the other hand, none of the folks I know who work on pictures seriously with computers bother with the whole RAID thing...it certainly would be cheaper to just buy two of the least expensive seagate 8 tb models and just keep doing what I've been doing...Any thoughts from those with experience would be much welcome...

Gracias,
Chester McCh

[jpheneger]

RAID has saved my bacon on three separate occasions. Having an enclosure with 4 drives in RAID 5 configuration makes sure I never lose data from a single drive failure. I don't have to swap drives around and it's all taken care of when a drive gets replaced. Highly recommended.

[williaty]

RAID is not a backup.

Repeat it with me:

RAID IS NOT A BACKUP


The purposes of RAID are 1) High availability even when a drive in the array fails, is replaced, and rebuilt (imagine being a business where your ecommerce DB lives on a RAID. If a drive fails, you don't go out of business while it gets replaced) and 2) Greater read or write speeds due to mirroring, striping, etc. The first purpose, frankly, is just about dead as drive sizes are reaching a point where rebuilds can take so long that the likelyhood of a 2nd drive failing while the 1st is getting rebuilt is starting to become uncomfortably high. For that matter, SSDs are starting to eat into the 2nd justification as well as their prices drop and capacities grow.

The reason RAID isn't a backup is that it doesn't provide data integrity against some of the most common problems. It provides a little redundancy against a single-drive hardware failure (good luck if your RAID card itself fails though!). It provides no protection against you mistakenly deleting something. It provides no protection against you getting a virus or becoming the victim of ransomware. It provides no protection against lightning. It provides no protection against a fire. It provides no protection against a flood, tornado, earthquake, or jealous ex. You get the picture. Well, if you're relying on RAID as a backup, you won't have the picture very long, I suppose.

For something as important as our photos are to us, the very minimum acceptable backup strategy is to have the copy on your workstation that you're working with, an on-network clone of your photo archive that you frequently sync, and a minimum of 2 clones available for off-site backup. I say two because you need to make a current clone, physically take it to your safe location (like a safe deposit box at the bank), store it there, retrieve the other clone you're using as an off site, drive it back home, and sync it. If you only have one off-site backup, you have 0 off-site backups while you've brought that one home to sync it.

Personally, I also make non-writable backups on a regular basis (read: burn a Blu-Ray disc). If the worst happens and I'm hit by particularly nasty ransomware that hides long enough to get my writable backups before it locks up and demands money, the only recovery I'll have is the stuff I burned to BRDs because, after all, the ransomware can't encrypt a disc it can't write to.

[jpheneger]

I am by no means suggesting that RAID is an alternative for backup. But RAID is a valuable part of my backup solution. My setup is this:

Copy from SD card to directly attached SSD (manual)
Copy from SSD to NAS (RAID 5) (automatic)
Copy from NAS to AWS S3 storage (automatic)
Reformat SD card when it goes back into camera (typically not next shoot).

With this arrangement I generally have 4 copies of the images with one being offsite. I have automated the process with the help of my NAS (insert Synology plug) and a few shell scripts.

I've lost images due to single drive failure in the past. Given how often I'm using my drives, I am most at risk for single drive failure - RAID is made for that. Yes there are problems with replicating a very large drive, but that's why i also backup to S3 - which has it's own issues. I'm not looking forward to the day I need to download 2.5TB from S3 to rebuild my entire NAS.

But if it's a question of no RAID vs. RAID I'll choose to use RAID every time.

[Chester McCheeserton]

Thanks guys. Did some research and looks like to get both the speed and redundancy benefit from a raid setup I'd need to go with at least 3 and ideally more drives, out of my budget for now. Just got 2 used 8tb drives and will keep them separate for now. Although I'll be dealing with this again in a year or 2.

Advice about raid not being back up noted.

[Corran]

Forget RAID. Go JBOD.

This is what I use, and it's a great, easy-to-use JBOD setup. The newest update is even easier. It's called unRAID.
https://lime-technology.com/download/

I have a 12TB server that has run almost nonstop for several years now. Highly recommended. I use a 4TB work drive and then store all RAW/TIFF files on the server once editing is finished.

[EdWorkman]

Folks with really important data back up off site
Fires and lightning are mostly avoided
But what is civilization going to do when the Big Solar Flare occurs.
I had saved some scans on CDs - very small capacity these days, and a pain in a pileful
I bought a pack of DVD disks ........

[Ted Baker]

I bought a pack of DVD disks ........

Creating multiple exact copies, can increase the reliability of this approach for very important stuff considerably. i.e. if the disk goes bad. You will need a solution to combine the two or more copies, however.

[wilderness]

Considering the large file sizes that are discussed here, discs for BU (even the newer Triple Layer Blu Rays (100gig) are not viable.
CD 650 meg
DVD Single Layer 4.7 gig
DVD Dual 8.5 gig
Blu Ray Single Layer 25 gig
Blu Ray Dual Layer 50 gig
Blu Ray Triple Layer 100 gig
I've progressed to Triple Layer Blu Ray and the discs are not cheap.
Incremental BU's are a must procedure for any use of discs, however incremental in most instances makes restoring a nightmare.
I've had multiple major failures with a name brand Dual Layer DVD's (think Ella Fitzgerald) after short periods of time. So much that I won't even buy that brand of disc any longer. Otherwise, my disc failures over a twenty year period have been minimal.
There's only a handful of companies that make discs and everybody else puts their name on them.
Been scanning A120 negatives for four months now at 2400DPI and my monthly BU's are running just over 25 gig. (TIF's may be downsized and restore to their original size, however who wants the extra work of resizing and saving two copies of a file.)

[Steven Ruttenberg]

I use 8 2 GB hard drives in 4 2 disco back ups. I have an anolog pair, digital pair, archive pair and a miscellaneous pair. Have not taken to of site storage yet maybe I will soon, but also looking at a fire proof safe stored in garage that I backup to over my network. My 4x5 images end up being upwards of 10-20gb in size for working files. Digital can get up there.

I need the fire proof safe anyway to store my negatives and prints as well so I should get started. Dvds, blue rays won't do it for me. I also don't use raid. I just have automatic copies made every night of each pair. I will soon need to go to a 6!TB drive pair for my 4x5 work.

[Corran]

My 4x5 images end up being upwards of 10-20gb in size for working files.

What exactly are you doing that you need 10+GB working files?

I generally archive my *.tif scans with simple contrast/color editing (or dropping to grayscale for b&w images), which are around 1GB maximum, maybe 2GB if I really push the resolution up. Most images are archived in 8-bit because it's not that important to keep them at 16, which at 3000 DPI gives me about a 175 megabyte file. Even critical images with some Photoshop curves don't get anywhere near that kind of size. I just tried and a 4000DPI scan with some curves and saved as a Photoshop document only reached about 2.5GB.

[Steven Ruttenberg]

I scan my 4x5 at 3900 dpi and save as 16 bit files. Both color, bw and slide. There are many reasons for this, but I won't sacrifice information for the sake of space. My bw files are about 600mb and my color files are 1.6gb. Now, as I run them thru photoshop, I have a non-destructive work flow so I use layers, smart objects, etc. By the time I am done with things like luminosity masks, dodge/burn, frequency separation, sharpening, running thru camera raw, creating smart objects, etc it is not uncommon for my files to be 10-20gb (working files) that includes a bw file. From this, I can go back to beginning or anywhere in between the beginning and end product. This has a ton of advantages. I do this with my digital camera work flow as well. As I progress thru my work flow, I never destroy the original file information and can manipulate each and every step at any time. From this file, I create the files for printing and web.

As for 16 bit vs 8 bit, if I use 8 bit, I quickly can be left with posturization and banding in areas such as the sky and any smooth gradient like that. Plus, I get better color detail and reproduction, etc. My digital files from my DSLR will be around 2-10gb or smaller depending on what I need to do, but the giant 4x5 film (which at 3900 dpi has just slight less resolution than my Canon 5DMKIII) provide the 600mb to 1.6gb size files. Additionally, since I use a Canon ImagePrograf6400 printer, it likes files that have resolutions that are multiples of 300. Hence, 3900 dpi (although I may up it to 4200 to get same resolution as my Canon camera) so, 300, 600, 900, 1200, 1500, 1800,...3900, 4200. Epsons, I believe work best with multiples of 360.

Since I don't produce a lot of files, maybe 1 a week or so as I spend/take my time on each one, it is not that big of a deal, plus my Macpro is more than capable, although I am going to upgrade it next year I think to the latest Macpro. My current one is 6 years old and sometimes gets a bit schizophrenic when I use 3 monitors. But it is reliable workhorse and pretty damned fast. I can stitch a 43 frame pano from full frame dslr images in like 15 seconds. The giant sized analog files are just as easy to manipulate

Anyway, that is why my files are so big. And no, I won't compress my working file when I am finished as it would defeat the purpose of a non-destructive work flow.

[Corran]

Personally, I think that's a bit excessive.

[jp]

With my Epson scanner, I don't think 3000-4000dpi is that much better than somewhat lower resolutions. I'm not worried about the space, but time is important too and really big files take more time at the computer. I'm not retired and have to balance many competing time sinks without cutting corners.

[Steven Ruttenberg]

Not really excessive.

I did study with my Epson V850 and there was no measurable difference in scan times between 2400 and 3900. Additional there was a huge difference in quality between 2400 and 3900 dpi. The main reasons for my scan resolution is to take advantage of the size of the negative and its inherent better resolution for an equivalent focal length and field of view over 35mm. Another reason is that by starting with a high dpi scan, I can resize the picture down to any size I want for printing or web.

For example, my dslr has 24mp and at 300 dpi that is about a 19in x13in native print. For my 4x5, I would have a printable picture at 300 dpi of around 65in x 52in for 3900dpi. So from there I can size a print to almost anything that or below. Some have argued that the scanner is no good beyond say 2400dpi, which is a 32x40 native print however, I have tested the differences by comparing files at different scan resolutions and 3900-4200 is much better than even 3000dpi. I know, I know, plenty will disagree with and even state I am not right in the head, but I know what I know. Granted at some point a higher resolution scan dpi will not provide any increase in quality. In the end, it is what the artist prefers and what is right by them.

As for speed, I was working on these large files using a Surface Pro 4 and an external hard drive. While it got a little slow at times, it was workable and I would spend a few hours a night working on one photo for the week. I was traveling to another state to work for the week, so after work, I would go to my rental and work on a file. At home, my Mac Pro eats these files for lunch!

[Chester McCheeserton]

I'm sure we've all got our own workflows dialed in to what feels right. I scan my 5x7s on a drum at either 4000 or 3200, never done tests side by side to see if one really outdoes the other, I doubt it, at least on Portra, but sometimes 4000 makes me feel better.

I save the raw scan as an 8 bit tiff. But I do switch to 16 bit when I'm compositing together multiple negatives or multiple scans done at different apertures, (which I do quite frequently). Using the big soft paint brush to mask full size files together can make banding in skies if done in 8 bit.

and a handy trick someone showed me once is that you can get nearly all of the benefit of working in 16bit (with the exception of big soft brush moves just mentioned) just by working in 8bit, and only switching to 16 bit at the very end right before you flatten the layers, then switching back to 8. (although I'm sure lots of folks just never flatten) But I , like Steven, do frequently bump up over 10 gigs on the initial compositing in 16 bit.

Save em as PSB's, run the scratch to a giant empty external, rolling my mid 2012 macbook pro with 16 gigs of ram and a big internal SSD seems to work okay. Sometimes by the end I've got 15- 20 adjustment layers mostly burning and dodging curves and levels and hue/sat. But if you can do it with fewer layers and the print looks good, certainly that's all that counts.

[Steven Ruttenberg]

I'm sure we've all got our own workflows dialed in to what feels right. I scan my 5x7s on a drum at either 4000 or 3200, never done tests side by side to see if one really outdoes the other, I doubt it, at least on Portra, but sometimes 4000 makes me feel better.

I save the raw scan as an 8 bit tiff. But I do switch to 16 bit when I'm compositing together multiple negatives or multiple scans done at different apertures, (which I do quite frequently). Using the big soft paint brush to mask full size files together can make banding in skies if done in 8 bit.

and a handy trick someone showed me once is that you can get nearly all of the benefit of working in 16bit (with the exception of big soft brush moves just mentioned) just by working in 8bit, and only switching to 16 bit at the very end right before you flatten the layers, then switching back to 8. (although I'm sure lots of folks just never flatten) But I , like Steven, do frequently bump up over 10 gigs on the initial compositing in 16 bit.

Save em as PSB's, run the scratch to a giant empty external, rolling my mid 2012 macbook pro with 16 gigs of ram and a big internal SSD seems to work okay. Sometimes by the end I've got 15- 20 adjustment layers mostly burning and dodging curves and levels and hue/sat. But if you can do it with fewer layers and the print looks good, certainly that's all that counts.

Yep, all that counts in the end is the final product. I often say, what happens in the black box between beginning and end doesn't matter as long as the final product is what you want. I just finished with a 43 frame pano from my Canon 5DMKIII and the working file is 46GB, drops radically when I flatten for creating a print. This thing is like 135in x 98 in native format of 300dpi. Can you imagine a pano that large starting with 4x5?

[seezee]

I archive my images on a RAID5 NAS (4 disks) and keep my working images on a dual RAID0 SSD (directly attached via Thunderbolt 3 cable to the desktop). All of the files are backed up to the cloud. If I had more $$$ I'd be making physical backups to a large capacity portable drive & keeping it in a safe deposit box, swapping it out monthly with a 2nd portable drive.

RAID IS NOT A BACKUP.

[Steven Ruttenberg]

I archive my images on a RAID5 NAS (4 disks) and keep my working images on a dual RAID0 SSD (directly attached via Thunderbolt 3 cable to the desktop). All of the files are backed up to the cloud. If I had more $$$ I'd be making physical backups to a large capacity portable drive & keeping it in a safe deposit box, swapping it out monthly with a 2nd portable drive.

RAID IS NOT A BACKUP.

I tried an external cloud based service once, the estimated time for an initial backup was like a month of continuous data dump, so I said no. Eventually I will have a fire proof safe stored on my property away from house to protect all of my images.

[locutus]

if you are archiving .tiff's, why not bzip2 them? that would significantly reduce on storage requirements....

[seezee]

I tried an external cloud based service once, the estimated time for an initial backup was like a month of continuous data dump, so I said no. Eventually I will have a fire proof safe stored on my property away from house to protect all of my images.
Aye, it can take a deal of time for the initial backup. But after that it's incremental. You've nothing to lose by starting a cloud backup, and literally everything to lose if you don't.

[Jac@stafford.net]

Why do people imagine that 'the cloud' is more secure than ordinary local solutions? I suspect it is about abandoning personal responsibility.

[jp]

Why do people imagine that 'the cloud' is more secure than ordinary local solutions? I suspect it is about abandoning personal responsibility.

Realistically, cloud services are more reliable than the average tinkerer or amateur's or overloaded+underskilled businessperson declared to be in charge of IT DIY untested systems. It's a matter of being humble.. If you notice someone can do a task better and the price is right, it's wise to take advantage of it. Email is a good example.. I used to run mail servers for my ISP. Email is the original cloud service. I had to constantly maintain servers, anti-spam software, firewalls, blacklist subscriptions, software updates, even using opensource software. People running MS exchange had it even tougher. When I got out of that business, and into the next one, I let google handle that. They do email service better than I did. Now there are cloud options for about everything. Not all cloud options are good or are right for certain customers... For backup, carbonite is an ideal backup solutions for certain situations. It is not suitable for replacing the 6tb drive I used in my garage to backup photos. A couple years down the road, it might be. Honest evaluation of pros and cons and risks with each option is important. Most people doing computer things are wholly and willfully ignorant about risks.

[Jac@stafford.net]

Realistically, cloud services are more reliable than the average tinkerer or amateur's

We shall see. The cloud is reliable until it is not, then it's catastrophic.

[jp]

More reliable.. Not perfect. Try as we might, we can't outsource responsibility, just tasks.

[seezee]

Why do people imagine that 'the cloud' is more secure than ordinary local solutions? I suspect it is about abandoning personal responsibility.

The cloud is only one part of any decent backup plan. Its main virtue is that it is offsite. Copying files to physical drives and keeping them locally offsite is good too, but what happens when a tornado destroys your town (as has happened 10 miles north of me twice in the last decade; I know someone who lost their house both times)? Your house and your safe deposit box both gone and you're up sh*t creek without a hard drive.

[seezee]

Realistically, cloud services are more reliable than the average tinkerer or amateur's or overloaded+underskilled businessperson declared to be in charge of IT DIY untested systems. It's a matter of being humble.. If you notice someone can do a task better and the price is right, it's wise to take advantage of it. Email is a good example.. I used to run mail servers for my ISP. Email is the original cloud service. I had to constantly maintain servers, anti-spam software, firewalls, blacklist subscriptions, software updates, even using opensource software. People running MS exchange had it even tougher. When I got out of that business, and into the next one, I let google handle that. They do email service better than I did. Now there are cloud options for about everything. Not all cloud options are good or are right for certain customers... For backup, carbonite is an ideal backup solutions for certain situations. It is not suitable for replacing the 6tb drive I used in my garage to backup photos. A couple years down the road, it might be. Honest evaluation of pros and cons and risks with each option is important. Most people doing computer things are wholly and willfully ignorant about risks.

Exactly!

[angusparker]

My backup protocol is as follows:
Raid 1 on SSDs locally connected to computer
Cloud backup on Backblaze using Arq
Cloud backup on OneDrive using Arq
2 Blu-Ray M Disc archive on a yearly basis, stored in two locations

Cloud backups are not infalible as the company can screw up or go out of business but they are cheap and they offer protection against being an idiot! But good to have two cloud backups with different companies IMHO.


Sent from my iPad using Tapatalk Pro

[Steven Ruttenberg]

One reason I do not like the cloud is you are giving up your privacy to the host and opening yourself up to invasion of that privacy. Plus, there is always a bread crumb trail that leads back to you. Granted I would only put photos on there, but then again, I don't trust anyone with my personal privacy and freedoms. The cloud is no more safe than opening your face mask on your helmet in deep space. However, if you don't mind giving up some personal privacy knowing anyone one with half a brain can hack the system or the systems owners can take your info and do as they please, then I guess it is a good as part of a back up plane. I am not a conspiracy nut, but practical as I have had hacks on me before, and I even had face book label me with a political affiliation even though I do not post political anything on facebook.

Anyway, I plan for 3 failures like we did at NASA. We had the main system, and 3 redundant systems. Sounds like over kill, but in deep space you can't get a technician to make a house call. Right now, I am shy of the 3 redundancies. I have the main and 1 redundant drive. I need to set up the two additional drives in the near future, one in a fireproof safe that is connected to my in home network and then one somewhere on the property in an in ground sealed vault that is also connected to my system.

Maybe someday, I will trust the cloud and people will become trustworthy, but I ain't holding my breath. I might for giggles try carbonite again. I have a 1 gig connection to the outside world so it shouldn't take that long. I am also not throttled by my isp, but carbonite does throttle you unless you sign up and pay, the trial they give you sucks.

[angusparker]

One reason I do not like the cloud is you are giving up your privacy to the host and opening yourself up to invasion of that privacy. Plus, there is always a bread crumb trail that leads back to you. Granted I would only put photos on there, but then again, I don't trust anyone with my personal privacy and freedoms. The cloud is no more safe than opening your face mask on your helmet in deep space. However, if you don't mind giving up some personal privacy knowing anyone one with half a brain can hack the system or the systems owners can take your info and do as they please, then I guess it is a good as part of a back up plane. I am not a conspiracy nut, but practical as I have had hacks on me before, and I even had face book label me with a political affiliation even though I do not post political anything on facebook.

Anyway, I plan for 3 failures like we did at NASA. We had the main system, and 3 redundant systems. Sounds like over kill, but in deep space you can't get a technician to make a house call. Right now, I am shy of the 3 redundancies. I have the main and 1 redundant drive. I need to set up the two additional drives in the near future, one in a fireproof safe that is connected to my in home network and then one somewhere on the property in an in ground sealed vault that is also connected to my system.

Maybe someday, I will trust the cloud and people will become trustworthy, but I ain't holding my breath. I might for giggles try carbonite again. I have a 1 gig connection to the outside world so it shouldn't take that long. I am also not throttled by my isp, but carbonite does throttle you unless you sign up and pay, the trial they give you sucks.

If you use ArqBackUp on the client side the cloud service only sees encrypted rubbish and you are the only person with the key. No one except maybe the NSA with a very big computer and a lot of time (like months) is going to read you files. Personally I would love them wasting their time decrypting my large format photos ... :-)

[Steven Ruttenberg]

If you use ArqBackUp on the client side the cloud service only sees encrypted rubbish and you are the only person with the key. No one except maybe the NSA with a very big computer and a lot of time (like months) is going to read you files. Personally I would love them wasting their time decrypting my large format photos ... :-)

That is true. If it is encrypted like PGP, then the code is not breakable, at least in a 1000 lifetimes according to the military and the FBI and such who bitched a storm about it because of that, and that is the weak encryption algorithm. The only way they could get you is to intercept the key if it is emailed to you or to confiscate your computer or hack into it and get the key/s that way. I think what I would do is put the key into 1password and ditch the original key. The it is encrypted. I like that, an encryption of an encryption.

I may look up ArqBackUp. What is price? I have at least a couple of TB of images. Mostly crap and family, but also some keepers. :)

[angusparker]

That is true. If it is encrypted like PGP, then the code is not breakable, at least in a 1000 lifetimes according to the military and the FBI and such who bitched a storm about it because of that, and that is the weak encryption algorithm. The only way they could get you is to intercept the key if it is emailed to you or to confiscate your computer or hack into it and get the key/s that way. I think what I would do is put the key into 1password and ditch the original key. The it is encrypted. I like that, an encryption of an encryption.

I may look up ArqBackUp. What is price? I have at least a couple of TB of images. Mostly crap and family, but also some keepers. :)

https://www.arqbackup.com/ one time fee of $50. Great software and easy to use. You keep the encryption key so it's basically unbreakable. Got to protect our keepers!

[seezee]

I ditched ARQ because they claimed unlimited storage but then one day it stopped working. I wrote to customer service and got a weasely response, saying that I had exceeded the storage limits (when I payed for the service, they explicitly said there was no limit). Do not recommend.

I use Crashplan Pro instead. For a few bucks a month it's worth the peace of mind, and it has saved my bacon more than once. The UI could use some improvements, and restores are slow (but that's partially dependent on your ISP), but it's overall a good and reliable service.

[seezee]

Regarding privacy, your browsing habits give away more information than your backups are ever likely too. Hell, your posts on this forum give away more. That's like worrying about serum cholesterol from egg consumption but being an alcoholic. It's not the eggs gonna kill you.

[angusparker]

I ditched ARQ because they claimed unlimited storage but then one day it stopped working. I wrote to customer service and got a weasely response, saying that I had exceeded the storage limits (when I payed for the service, they explicitly said there was no limit). Do not recommend.

I use Crashplan Pro instead. For a few bucks a month it's worth the peace of mind, and it has saved my bacon more than once. The UI could use some improvements, and restores are slow (but that's partially dependent on your ISP), but it's overall a good and reliable service.

Arqbackup is not a cloud service but software that automates and encrypts your content to any of about twenty cloud service. I use Backblaze and OneDrive as my cloud services. It works with both.


Sent from my iPad using Tapatalk Pro

[bieber]

I use a RAID array for my main work station, and sync my photo library between my desktop, laptop, and a couple of external hard drives (one of which is at my inlaws' house on the other side of the country and gets updated a couple times a year when we're over that way. I figure if a natural disaster takes out Florida and California at the same time, I've likely gone with it one way or another). For online backup I use Amazon Glacier. It's dirt cheap to upload and store, but if you ever need to retrieve your data it'll take a few hours and cost you a pretty penny. Given that it's my backup of last resort, I'm pretty happy with that gamble. With any luck I'll spend the rest of my life paying on the order of a dollar or two a month to Amazon for storage and never end up having to pay the couple hundred dollars it would cost me to retrieve it all.

wrt the security and privacy of these services, as someone who's spent their entire professional career so far as a software engineer I would say that I'm not particularly worried about it. As far as security goes, they all have top notch security teams, to the extent that your data is drastically more likely to be compromised by malware on your local machine than it is on one of Google or Amazon's servers. I'd be a little bit less comfortable with a smaller service provider, but even some random second-tier service is still going to have much better security practices than the average computer user. And for privacy, well, yes, there are probably employees who could access your data if they wanted to. Those mechanisms are also going to be very tightly monitored, and anyone who does access your data is probably going to have to explain themselves to an auditor in short order. At every company I've worked for so far, inappropriate access to user data is the one thing that will absolutely get you fired on the spot without warning. And even if there is some engineer running around peeping into peoples' AWS data over at Amazon, they have tens of thousands (millions?) of customers with an unimaginably huge amount of data stored. The odds that some random interloper would stumble upon any of your files are astronomically low, unless you happen to have an Amazon engineer with a personal grudge against you. And of course if you're really, really paranoid you can always just encrypt your files on your local machine before sending them to a cloud storage service, and at that point you're storing random bytes for all they know.

Another thing I'd like to suggest, for anyone who's somewhat comfortable with the command line, is using git annex to version and manage large files. It's a little bit of a learning curve (and, I think, probably easier to get working on a Mac or Linux box than Windows), but it's an absolute godsend for backups. With one command I can fully back up all or any subset of my photos to another hard drive or computer, or Glacier (plus a bunch of other cloud services if you're so inclined), with a careful record kept of which files are where and only transferring the ones that actually need to be transferred. It can also do neat things like letting you set a minimum number of locations for any given file to be present and stopping you if you try to delete a copy that would bring you below the minimum, but my library isn't really big enough at this point to worry about that (I will admit I'm compressing the crap out of my scans, my plates really aren't good enough to justify multi-hundred-meg tif files yet anyways).

[seezee]

Arqbackup is not a cloud service but software that automates and encrypts your content to any of about twenty cloud service. I use Backblaze and OneDrive as my cloud services. It works with both.


Sent from my iPad using Tapatalk ProRight. It's been a while & my recollection is fuzzy. It might have been the cloud service I was backing up to that limited me. In any case I need TB of storage, not GB — and none of the free cloud storage tiers are that large. CrashPlan offers unlimited storage for only a few bucks a month. Encryption is available, but obviously that slows down backups and restores.

[jpheneger]

That is true. If it is encrypted like PGP, then the code is not breakable, at least in a 1000 lifetimes according to the military and the FBI and such who bitched a storm about it because of that, and that is the weak encryption algorithm. The only way they could get you is to intercept the key if it is emailed to you or to confiscate your computer or hack into it and get the key/s that way. I think what I would do is put the key into 1password and ditch the original key. The it is encrypted. I like that, an encryption of an encryption.

I may look up ArqBackUp. What is price? I have at least a couple of TB of images. Mostly crap and family, but also some keepers. :)I do something similar to this but I create and store my keys on my IronKey thumb drive. Its encrypted and requires a password to unlock. Enter the password too many times incorrectly and it fries itself. Just be sure not to forget the password...

[seezee]

I do something similar to this but I create and store my keys on my IronKey thumb drive. Its encrypted and requires a password to unlock. Enter the password too many times incorrectly and it fries itself. Just be sure not to forget the password...I hope you're backing up your IronKey to multiple locations. Right now you have a single point of failure if you ever need to retrieve files.

SSDs (including USB keys) will discharge themselves over time, losing all data. Toss a key in a drawer for 8 years and never use it? Guess what you'll find when you need it. NOTHING.

[jpheneger]

I hope you're backing up your IronKey to multiple locations. Right now you have a single point of failure if you ever need to retrieve files.

SSDs (including USB keys) will discharge themselves over time, losing all data. Toss a key in a drawer for 8 years and never use it? Guess what you'll find when you need it. NOTHING.I have the keys printed and stored in my safe deposit box, just in case I lose the USB drive. Can't be too safe!

[Steven Ruttenberg]

I am trying out arqbackup and wasabi for cloud, .0049 cents per mb which is like 5 bcks a month fora TB. I have 2 TB now and will more later. Wasabi also does not charge for downloading or restoring your files. You can also do versioning. It also encrypts data.

Sunday Night have a little cal encrypted data set done licall and only I have key generated locally. When I upload, the data is then encrypted by Wasabi. Key is shown on screen to copy it, downlad it.

So far Wasabi is as good as it gets.

[angusparker]

I am trying out arqbackup and wasabi for cloud, .0049 cents per mb which is like 5 bcks a month fora TB. I have 2 TB now and will more later. Wasabi also does not charge for downloading or restoring your files. You can also do versioning. It also encrypts data.

Sunday Night have a little cal encrypted data set done licall and only I have key generated locally. When I upload, the data is then encrypted by Wasabi. Key is shown on screen to copy it, downlad it.

So far Wasabi is as good as it gets.

And if you want a second cloud, backup, recommended, try Backblaze. Price and terms are similar. They can be uploaded one after other using Arq.


Sent from my iPhone using Tapatalk Pro

[Steven Ruttenberg]

Hmmm, I think 2 copies of everything at home plus encrypted back up on cloud is enough. I got 20TB. At home.