endless external HD shuffle – RAID 1, 5, or RAID unecessary?

[Corran]

My 4x5 images end up being upwards of 10-20gb in size for working files.

What exactly are you doing that you need 10+GB working files?

I generally archive my *.tif scans with simple contrast/color editing (or dropping to grayscale for b&w images), which are around 1GB maximum, maybe 2GB if I really push the resolution up. Most images are archived in 8-bit because it's not that important to keep them at 16, which at 3000 DPI gives me about a 175 megabyte file. Even critical images with some Photoshop curves don't get anywhere near that kind of size. I just tried and a 4000DPI scan with some curves and saved as a Photoshop document only reached about 2.5GB.

[Steven Ruttenberg]

I scan my 4x5 at 3900 dpi and save as 16 bit files. Both color, bw and slide. There are many reasons for this, but I won't sacrifice information for the sake of space. My bw files are about 600mb and my color files are 1.6gb. Now, as I run them thru photoshop, I have a non-destructive work flow so I use layers, smart objects, etc. By the time I am done with things like luminosity masks, dodge/burn, frequency separation, sharpening, running thru camera raw, creating smart objects, etc it is not uncommon for my files to be 10-20gb (working files) that includes a bw file. From this, I can go back to beginning or anywhere in between the beginning and end product. This has a ton of advantages. I do this with my digital camera work flow as well. As I progress thru my work flow, I never destroy the original file information and can manipulate each and every step at any time. From this file, I create the files for printing and web.

As for 16 bit vs 8 bit, if I use 8 bit, I quickly can be left with posturization and banding in areas such as the sky and any smooth gradient like that. Plus, I get better color detail and reproduction, etc. My digital files from my DSLR will be around 2-10gb or smaller depending on what I need to do, but the giant 4x5 film (which at 3900 dpi has just slight less resolution than my Canon 5DMKIII) provide the 600mb to 1.6gb size files. Additionally, since I use a Canon ImagePrograf6400 printer, it likes files that have resolutions that are multiples of 300. Hence, 3900 dpi (although I may up it to 4200 to get same resolution as my Canon camera) so, 300, 600, 900, 1200, 1500, 1800,...3900, 4200. Epsons, I believe work best with multiples of 360.

Since I don't produce a lot of files, maybe 1 a week or so as I spend/take my time on each one, it is not that big of a deal, plus my Macpro is more than capable, although I am going to upgrade it next year I think to the latest Macpro. My current one is 6 years old and sometimes gets a bit schizophrenic when I use 3 monitors. But it is reliable workhorse and pretty damned fast. I can stitch a 43 frame pano from full frame dslr images in like 15 seconds. The giant sized analog files are just as easy to manipulate

Anyway, that is why my files are so big. And no, I won't compress my working file when I am finished as it would defeat the purpose of a non-destructive work flow.

[Corran]

Personally, I think that's a bit excessive.

[jp]

With my Epson scanner, I don't think 3000-4000dpi is that much better than somewhat lower resolutions. I'm not worried about the space, but time is important too and really big files take more time at the computer. I'm not retired and have to balance many competing time sinks without cutting corners.

[Steven Ruttenberg]

Not really excessive.

I did study with my Epson V850 and there was no measurable difference in scan times between 2400 and 3900. Additional there was a huge difference in quality between 2400 and 3900 dpi. The main reasons for my scan resolution is to take advantage of the size of the negative and its inherent better resolution for an equivalent focal length and field of view over 35mm. Another reason is that by starting with a high dpi scan, I can resize the picture down to any size I want for printing or web.

For example, my dslr has 24mp and at 300 dpi that is about a 19in x13in native print. For my 4x5, I would have a printable picture at 300 dpi of around 65in x 52in for 3900dpi. So from there I can size a print to almost anything that or below. Some have argued that the scanner is no good beyond say 2400dpi, which is a 32x40 native print however, I have tested the differences by comparing files at different scan resolutions and 3900-4200 is much better than even 3000dpi. I know, I know, plenty will disagree with and even state I am not right in the head, but I know what I know. Granted at some point a higher resolution scan dpi will not provide any increase in quality. In the end, it is what the artist prefers and what is right by them.

As for speed, I was working on these large files using a Surface Pro 4 and an external hard drive. While it got a little slow at times, it was workable and I would spend a few hours a night working on one photo for the week. I was traveling to another state to work for the week, so after work, I would go to my rental and work on a file. At home, my Mac Pro eats these files for lunch!

[Chester McCheeserton]

I'm sure we've all got our own workflows dialed in to what feels right. I scan my 5x7s on a drum at either 4000 or 3200, never done tests side by side to see if one really outdoes the other, I doubt it, at least on Portra, but sometimes 4000 makes me feel better.

I save the raw scan as an 8 bit tiff. But I do switch to 16 bit when I'm compositing together multiple negatives or multiple scans done at different apertures, (which I do quite frequently). Using the big soft paint brush to mask full size files together can make banding in skies if done in 8 bit.

and a handy trick someone showed me once is that you can get nearly all of the benefit of working in 16bit (with the exception of big soft brush moves just mentioned) just by working in 8bit, and only switching to 16 bit at the very end right before you flatten the layers, then switching back to 8. (although I'm sure lots of folks just never flatten) But I , like Steven, do frequently bump up over 10 gigs on the initial compositing in 16 bit.

Save em as PSB's, run the scratch to a giant empty external, rolling my mid 2012 macbook pro with 16 gigs of ram and a big internal SSD seems to work okay. Sometimes by the end I've got 15- 20 adjustment layers mostly burning and dodging curves and levels and hue/sat. But if you can do it with fewer layers and the print looks good, certainly that's all that counts.

[Steven Ruttenberg]

I'm sure we've all got our own workflows dialed in to what feels right. I scan my 5x7s on a drum at either 4000 or 3200, never done tests side by side to see if one really outdoes the other, I doubt it, at least on Portra, but sometimes 4000 makes me feel better.

I save the raw scan as an 8 bit tiff. But I do switch to 16 bit when I'm compositing together multiple negatives or multiple scans done at different apertures, (which I do quite frequently). Using the big soft paint brush to mask full size files together can make banding in skies if done in 8 bit.

and a handy trick someone showed me once is that you can get nearly all of the benefit of working in 16bit (with the exception of big soft brush moves just mentioned) just by working in 8bit, and only switching to 16 bit at the very end right before you flatten the layers, then switching back to 8. (although I'm sure lots of folks just never flatten) But I , like Steven, do frequently bump up over 10 gigs on the initial compositing in 16 bit.

Save em as PSB's, run the scratch to a giant empty external, rolling my mid 2012 macbook pro with 16 gigs of ram and a big internal SSD seems to work okay. Sometimes by the end I've got 15- 20 adjustment layers mostly burning and dodging curves and levels and hue/sat. But if you can do it with fewer layers and the print looks good, certainly that's all that counts.

Yep, all that counts in the end is the final product. I often say, what happens in the black box between beginning and end doesn't matter as long as the final product is what you want. I just finished with a 43 frame pano from my Canon 5DMKIII and the working file is 46GB, drops radically when I flatten for creating a print. This thing is like 135in x 98 in native format of 300dpi. Can you imagine a pano that large starting with 4x5?

[seezee]

I archive my images on a RAID5 NAS (4 disks) and keep my working images on a dual RAID0 SSD (directly attached via Thunderbolt 3 cable to the desktop). All of the files are backed up to the cloud. If I had more $$$ I'd be making physical backups to a large capacity portable drive & keeping it in a safe deposit box, swapping it out monthly with a 2nd portable drive.

RAID IS NOT A BACKUP.

[Steven Ruttenberg]

I archive my images on a RAID5 NAS (4 disks) and keep my working images on a dual RAID0 SSD (directly attached via Thunderbolt 3 cable to the desktop). All of the files are backed up to the cloud. If I had more $$$ I'd be making physical backups to a large capacity portable drive & keeping it in a safe deposit box, swapping it out monthly with a 2nd portable drive.

RAID IS NOT A BACKUP.

I tried an external cloud based service once, the estimated time for an initial backup was like a month of continuous data dump, so I said no. Eventually I will have a fire proof safe stored on my property away from house to protect all of my images.

[locutus]

if you are archiving .tiff's, why not bzip2 them? that would significantly reduce on storage requirements....