PDA

View Full Version : So. What do you do with 1megabyte files?



Bruce B
30-Oct-2014, 19:20
I've always loved large-format images, and making same. Just using a real camera is a treat, and seeing the detail and sharp transitions in resulting images is more than satisfying. Then I had some recent 4x5s scanned (my scanner will only do 6cm.) Wow. I may need to use my 6x9 film back more often. I need some advice about today's digital darkroom techniques.

For a start, what resolution should be used for scans?

Are these usually kept on DVD's, with reduced images stored on hard drives (both internal and external)?

My computer has an i7processor, 24gb of RAM, and my OS is 64bit Windows 7. When I open Photoshop the monitor jumps up in the air 1/4 inch :). So I can handle large files. I'm just wondering what the usual practice is surrounding digital scanning, storage, and processing.

Thanks in advance,
BAB

Jmarmck
30-Oct-2014, 19:55
1 megabyte is not much of a file.

With that amount of ram you should not have any trouble processing just about anything. There will be several opinions on what resolution to scan. Some say scan once at the highest resolution so you only can once. That is a valid approach but I will start out around 1200 dpi and scan to an uncompressed TIFF. On a 4x5 that is somewhere between 40 and 50 mb or 5500x4400 pixels (8 bit). This suitable for internet after some reductions and post processing. If the image is suitable I will scan at higher resolutions depending upon the final use. Scanning at 6400 will produce a 1.5 gb file with roughly 30kx24k pixels. Overkill for the internet but for suitable for printing, if the image will hold up.

gregmo
30-Oct-2014, 20:07
For a start, what resolution should be used for scans?
BAB

You are going to get a wide range of answers. Only you can make that call..what will you be using them for/ how large of a print do you need. You may find this article interesting if you haven't seen it: https://www.onlandscape.co.uk/2011/12/big-camera-comparison/

I have images scanned by Lenny Eiger (member here). 4x5 & 617 at 4000-8000 (1.3-6gb file) and 5x7 at 4000 (3gb file).

Bruce B
30-Oct-2014, 20:23
I meant 1000mb or a gigabyte. Sorry. (Or duh.) The most dense scan I have is at 3800ppi, with a file a bit over 1gig. I'm wondering what resolution is really necessary for, say, a 16x20 print.

Thanks for your response.

Bruce B
30-Oct-2014, 20:24
Thanks, so how do you store these? I'm assuming on DVD's ?

paulr
30-Oct-2014, 20:47
You don't need anywhere near that resolution for a 16x20 print. People will argue this, but you can make a print with less than half the linear resolution which, if done properly, will be indistinguishable from the full resolution version. By this I mean that in a double-blind test no one will be able to tell the difference.

This isn't to say you should down-res your scans. I believe some experimentation is in order to find out what resolution, with your current scanning technology, extracts all the image information possible. I'm guessing with large format it will be under 3880, but it's quite possible that you will have some fine detail near the center of some images that takes advantage of this resolution. This, in turn, would be visible in a larger print.

So, if you want to store the best scan possible, it may make sense to keep doing what you're doing. If you save the files with LZW or Zip compression you can cut the file size in half (more or less). This is lossless; the only disadvantage is greatly increased open and save times (especially if you add layers).

If you're ok with saving files just for the best possible 16x20, you can downres to 1500ppi or so. This gives a file that's 1/4 the size you're seeing. Even smaller when compressed.

I'm not comfortable archiving files on optical media. Its permanence is questionable, because of the fugitive dyes. I keep everything on a big hard drive, with another big hard drive in the machine for automatic hourly backups (time machine), and other drives in a fireproof box or offsite for periodic manual backups ( carbon copy cloner).

Keeping this much data around is a drag, but at least we're not dealing with video ...

Peter De Smidt
30-Oct-2014, 21:45
I agree with Paul. Dvd's are not a good medium. Not only are they of questionable longevity, they are slow. (If you do use them, check out the reviews as to longevity. Some are much better than others.) As Paul suggests, getting a couple of big hard drives is a good solution, as long as you're diligent in backing up to all of the drives. Off-site storage for one is a good idea.

Bruce B
31-Oct-2014, 05:42
Thanks again, as this article addresses most of my basic questions. BAB

Bruce B
31-Oct-2014, 06:01
Thank, Paul,
That makes sense, and as I think about it gives me two options.

First I can have the scans done at around 3800 and save that scan on a second hard drive; then downsize to 1600 and save those in my regular manner so as to have these quickly available. My 'regular manner' involves storing on the 1tb drive in my computer as LZW tiffs and also on 3tb desk hard drive as well. I also have two 1tb portable hard drives that I periodically rotate into a safe-deposit box to protect against fire or theft. Unfortunately these have less than 200gb of space remaining, so I may just buy another portable drive for saving film scans. It's relatively easy to increase the storage capability in the computer itself with another slave drive and I could opt, I suppose, for a couple portable 2tb hard drives for the rotation aspect.

The second option is scan initially at 1600 for working with the images, then save the negatives in that same safe deposit box. If larger scan is needed I can do that later. It's good to know that others are skeptical of DVDs.

BAB

Bruce B
31-Oct-2014, 06:05
Thanks, Peter. I do, indeed, store off site. Currently I have enough space on computer, desk top drive, but I'm going to be squeezed soon on the off-site portables. I'll just have to address that soon. I appreciate the advise.
BAB

Christopher Barrett
31-Oct-2014, 06:13
The golden resolution for inkjet printing has long been 300dpi, though many will argue for 360dpi for Epson printers. Now keep in mind that that is the resolution of your print size, so obviously the film will have to be scanned at a higher resolution to be enlarged to 16x20. For a 16x20 print, you want a file that is 4800 pixels by 6000 pixels. For 4x5 film, that means scanning at 1200 dpi. *You'd actually want a little more since the image area is not the whole 4x5 inches.

Personally, I scan as high as the scanner allows, so 4000 dpi on my Howtek Drum Scanner. Hard drives are cheap and computers keep getting faster. I'd hate to put all the time and effort into perfecting a file and then wish I'd scanned at a higher res some time down the road.

I've never had a DVD or CD go bad, but all the experts say that these are NOT stable media. Hard drives are a little scary too (I have had some of those go bad) so redundancy is imperative. Most commercial operations have double redundancy, so 2 backups of the original. My next investment is going to be an LTO tape drive for archiving off site.

HTH,
CB

Jmarmck
31-Oct-2014, 07:01
I have found tape backups to be cumbersome. They are by no means stable. Nothing is stable, (with tongue in cheek) unless you put it on the internet.
I use a series of USB hard drives and duplicate when necessary. I also backup my music in the same manner.

Jac@stafford.net
31-Oct-2014, 07:24
[... snip excellent information...]I'm not comfortable archiving files on optical media. Its permanence is questionable, because of the fugitive dyes. I keep everything on a big hard drive, with another big hard drive in the machine for automatic hourly backups (time machine), and other drives in a fireproof box or offsite for periodic manual backups ( carbon copy cloner).

I am also uncomfortable with optical media, but perhaps for no good reason today with JVC Archival DVD-R media. My reasons come from the burden of over thirty years of backup and recovery efforts for a university which I must admit has come up with a good system. (Their third tier (of four) storage was carried to safes in my former lab.) I do not do any significant digital storage of personal images. Period. The originals are on film, the darkroom is up-to-date and the films are ready to be scanned just-in-time, however I doubt there is any value worth the effort. That's just me.

Tape is far, far from archival. I would never use it. Perhaps my experience is dated, but I've experienced tape failures throughout the years, and have even coded some recovery methods, including physically exercising the tape to defeat some of its tendency to shrink or stretch - not so important with good methods.

A facetious closing - if you want an image to last, it must be worthless in every respect. Just put it on the open net with your name and it will live forever to haunt you. Happy Halloween.
.

Randy Moe
31-Oct-2014, 08:53
+1 Jac!

Lenny Eiger
31-Oct-2014, 10:58
You have lots of RAM, plenty of processor speed. Get some storage hard drives and be done. There is no reason to downsize anything. Use adjustment layers so you can go back, adjust anything up or down at will...

Lenny

LF_rookie_to_be
31-Oct-2014, 11:38
Several Hitachi 1TB hard drives in external enclosures w/ Firewire connection (Mac-only user) store 'em all. Incidentally, my drum scans of 4x5s are exactly 1GB. I keep every raw scan, as well as all selected and PSed for inkjet there. A Macbook on 10.6 w/ 8GB RAM and 50-80 GB free space chews 1GB scans just fine.

Bruce B
31-Oct-2014, 14:17
I have a reluctance to using DVDs, even the JVC 100 year variety. At 3.7 gb, that's only three scans per disc at 3800ppi. For $100 or a bit more I can buy a portable hard drive that will store almost 2000 such scans. And retrieval is quicker as well. Besides which, silver doesn't degrade as quickly as discs or hard drives. I'm currently sorting b/w and transparency 4x5s from 25 and 30 years ago, keeping the ones that are keepers for storage in the aforementioned safe deposit box. That and a hard drive I carry with me (as I do currently) would satisfy my archival considerations.

My biggest problem currently is I have no scanner that handles 4x5, which curtails my use of that format. It's much easier to get 6x9 color developed here in Maine, as there's no longer anyone that does 4x5 E6. A scanner may be in my future, but right now I have no problem sending them out to be scanned except for the turn around time.

Thanks for the suggestions.....much appreciated.

Darin Boville
31-Oct-2014, 15:19
Just a quick note on back ups of these large files. I use a lot of large files, too--huge things, though I don't print them big. I just like to torture myself.

Anyway, I used to carry around a set of hard drives in Pelican cases in the car--my back up set. That way if an earthquake destroyed my house I'd still likely have the car files, and it was nice to have them near (for comfort) on long trips.

Recently, however, I converted those card hard drives over to cloud storage. I make flat tiffs of all the finished images plus copy in my entire Aperture, Final Cut libraries, and my desktop (and I forget what else)--and back it all up to the cloud. Terrabytes of stuff. Takes FOREVER for the first back up--literally weeks--but then runs quietly in the background afterwards.

I pay something like $45 a month, I think. Well worth the piece of mind and low level of hassle. No more forgetting to make the car back-ups. No more wondering if a quake or fire will destroy my entire life's work (including my non-digital negatives).

It's good. Do it....

--Darin

Bruce B
1-Nov-2014, 05:35
Great idea, except I'm retired and don't want the expense. Also, I once started to buy a Carbonite subscription until I noticed they'd dramatically increased the price. Too, although this is a bit obtuse, the cloud is hackable, the hard drives in my safe deposit box are not. Thanks, however, for the suggestion. A lot of photographers seem happy with cloud storage. BAB

Light Guru
1-Nov-2014, 06:14
Thanks, so how do you store these? I'm assuming on DVD's ?

Nope just use hard drives and have a backup. I can't even remember the last time I put a DVD in my computer. Optical media like CDs and DVDs are more dead then film in my opinion. Many computers now days don't even come with them. And they certainly are no longer needed to install programs or even new operating systems.


So, if you want to store the best scan possible, it may make sense to keep doing what you're doing. If you save the files with LZW or Zip compression you can cut the file size in half (more or less). This is lossless; the only disadvantage is greatly increased open and save times (especially if you add layers).

No seed to bother compressing files to try and save space. Hard drives nowadays are BIG and cheep.

Jmarmck
1-Nov-2014, 07:08
Cloud is not for me. One, I do not like my works just floating around where anyone with a bit of knowledge can grab it......not that anyone would. Two, my internet is probably one of the worse services out there. The switch box down the block does not have the capacity to handle the neighborhood. So there are days when I am constantly getting kicked off. This is also the reason I do not want the Adobe subscription service. I am happy with hard drives. They have space and can be moved around very easily.

paulr
1-Nov-2014, 07:32
No seed to bother compressing files to try and save space. Hard drives nowadays are BIG and cheep.

I don't know how you can make that generalization for everyone. A hard drive that's twice as big costs at least twice as much. Whether the cost-benefit difference makes sense for you depends on your circumstances. Personally, I've got many terabytes of storage and still choose to compress my files. Doing so has allowed me to postpone an expensive upgrade for over a year.

paulr
1-Nov-2014, 07:34
I'm sure the cloud is the future of all this. At the moment, unless you have South Korean bandwidth or Darin's patience, it's probably not a reasonable option for most people working with media files.

Jmarmck
1-Nov-2014, 07:45
I am waiting for the day when the personal computer is the size of a thumb drive and connects to peripheral devices via wireless. Lossless compressions will take 1 gig to less than a kilobyte. Monitors are flexible transparent/opaque mylar sheets that can be rolled up. Yet another break through on memory equal to the last one where like capacity devices are 1/100 the size of current technology. Your entire life would be on that one device that would connect to any monitor/sound system/printer/etc without drivers or wires.

We are nearly there with the smart phone.

bob carnie
1-Nov-2014, 09:00
I am waiting for the day the enterprise can beam me up.

Light Guru
1-Nov-2014, 09:19
I don't know how you can make that generalization for everyone. A hard drive that's twice as big costs at least twice as much. Whether the cost-benefit difference makes sense for you depends on your circumstances. Personally, I've got many terabytes of storage and still choose to compress my files. Doing so has allowed me to postpone an expensive upgrade for over a year.

Hard drives that are twice as big don't cost twice as much. You can get a 1tb drive doe $55, a 2tb drive for $85 a 3tb drive for $120 and 4tb drive for $150. Those prices are not doubling each time you double the size of the drive (prices found on newegg)

Jac@stafford.net
1-Nov-2014, 16:54
Hard drives that are twice as big don't cost twice as much. You can get a 1tb drive doe $55, a 2tb drive for $85 a 3tb drive for $120 and 4tb drive for $150. Those prices are not doubling each time you double the size of the drive (prices found on newegg)

If you back-up in serial mode, how do you cope with failures?

Light Guru
1-Nov-2014, 17:07
If you back-up in serial mode, how do you cope with failures?

The whole point of backups is so that you can cope with failure. You have the data on multiple drives so that if one failed you can replace it and but the data back on from the back up if the system drive fails to re back up if it was the backup drive that failed.

Bruce B
2-Nov-2014, 06:33
I'm not so sure about the cloud being everyone's future. Adobe's very own super safe cloud was severely hacked not long ago. And on another forum I cautioned individuals not to store their financial information on Carbonite. One irate member countered, "Carbonite is approved by HIPAA!" Among health care professionals such as myself that statement would have been met with a sharp peel of laughter. HIPAA can't take care of its own paper work, let alone someone else's. "The cloud" is just an air-conditioned warehouse of very large servers, and is subject to fire, vandalism, and hacking. I'm not being paranoid and I understand there are normal risks in every means of stashing data. However, to store one's financial information on the cloud is inviting disaster in my opinion. That said, I do use the internet to make transactions at our local bank :). Too, there's the question of access. I live in a small town in Maine and we're lucky to have moderately fast web connections. Many in Maine do not.

BAB

Jac@stafford.net
2-Nov-2014, 07:55
The whole point of backups is so that you can cope with failure. You have the data on multiple drives so that if one failed you can replace it and but the data back on from the back up if the system drive fails to re back up if it was the backup drive that failed.

I'm sorry I posted what I did. The subject is OT and came up amidst a concern for a rare cascading failure I worked on.

Back to photography!
Jac

SergeiR
3-Nov-2014, 07:11
so.. 1G files, i hope. First of all, unless you got drum scan - you most likely will be more than fine with 400m-800m tiff files for b&w. Once you hit 2G you can't often open bloody thing in Photoshop (bug in Vuescan on writing out Tiffs? i keep forgetting to check tech specs of format), which kinda limits printing options quite a bit.

Now. On "what to do" - scan, check. If you like it enough to scan, you might want to keep them around as "digital negative". So, extra couple of external hard drives to back work up, may be external network storage (they are reasonably cheap). I mean you can get few terabytes drives around. Just don't store them all in same place ,thats all. And unless you shoot like man possessed - 2T drive will last you for a while.

Corran
3-Nov-2014, 09:36
Of interest might be the topic I started back in July about Network Attached Storage:

http://www.largeformatphotography.info/forum/showthread.php?105200-Anybody-using-Network-Attached-Storage-RAID-Arrays-for-their-files

The last post is my update with specs and information on my system. I've got 10TB of storage, so my 4x5 scans which are usually at most about 1GB don't even make a dent.

Off-site storage is a different topic and one has many options. Potentially just the negatives stored off-site.

Randy Moe
3-Nov-2014, 09:54
Of interest might be the topic I started back in July about Network Attached Storage:

http://www.largeformatphotography.info/forum/showthread.php?105200-Anybody-using-Network-Attached-Storage-RAID-Arrays-for-their-files

The last post is my update with specs and information on my system. I've got 10TB of storage, so my 4x5 scans which are usually at most about 1GB don't even make a dent.

Off-site storage is a different topic and one has many options. Potentially just the negatives stored off-site.

Bryan, your unraid plan looks pretty good and actually affordable NOW, as even this digital half life will need it soon.

Great idea and thread I missed.

Greg Miller
3-Nov-2014, 10:00
I'm not so sure about the cloud being everyone's future. Adobe's very own super safe cloud was severely hacked not long ago. And on another forum I cautioned individuals not to store their financial information on Carbonite. One irate member countered, "Carbonite is approved by HIPAA!" Among health care professionals such as myself that statement would have been met with a sharp peel of laughter. HIPAA can't take care of its own paper work, let alone someone else's. "The cloud" is just an air-conditioned warehouse of very large servers, and is subject to fire, vandalism, and hacking. I'm not being paranoid and I understand there are normal risks in every means of stashing data. However, to store one's financial information on the cloud is inviting disaster in my opinion. That said, I do use the internet to make transactions at our local bank :). Too, there's the question of access. I live in a small town in Maine and we're lucky to have moderately fast web connections. Many in Maine do not.

BAB

While cloud storage is still impractical for most people that have TBs of data to move (because of the speed of the internet; and the cost per GB for the services that offer an initial data seed by sending a USB drive)). But using a cloud backup service where the data gets encrypted prior to transfer and storage is certainly extremely safe; even for financial data. And using a cloud backup service with multiple redundant data centers is also extremely safe. If multiple data centers go dwon, we have much bigger problems than the status of our image files.

Corran
3-Nov-2014, 10:15
Bryan, your unraid plan looks pretty good and actually affordable NOW, as even this digital half life will need it soon.

Great idea and thread I missed.

Good, glad I linked it then.

Large 2TB+ drives are so cheap now, and since I had extra computer parts around anyway it was trivial to build. I'm glad I did it.

I still remember buying a 500 megabyte drive for $300 and a "huge" 64 megabyte stick of RAM, back in the day for $200. How time flies...

Randy Moe
3-Nov-2014, 10:26
While cloud storage is still impractical for most people that have TBs of data to move (because of the speed of the internet; and the cost per GB for the services that offer an initial data seed by sending a USB drive)). But using a cloud backup service where the data gets encrypted prior to transfer and storage is certainly extremely safe; even for financial data. And using a cloud backup service with multiple redundant data centers is also extremely safe. If multiple data centers go dwon, we have much bigger problems than the status of our image files.

Don't be negative! :)

But I agree, if the cloud disperses we all have big problems.

Jac@stafford.net
3-Nov-2014, 16:36
First of all, unless you got drum scan - you most likely will be more than fine with 400m-800m tiff files for b&w. Once you hit 2G you can't often open bloody thing in Photoshop (bug in Vuescan on writing out Tiffs?).

Gotta be a bug in something other than Photoshop.

polyglot
4-Nov-2014, 00:44
Hard drives off-site are IMHO the best+easiest option. Cheaper and more-reliable than DVDs, without the bandwidth and cost issues of the cloud. Sure, the cloud is not really secure, but then neither are the unencrypted hard drives I keep in my desk drawer at work (photography is not my day job). If you want to use the cloud securely, you can encrypt the data yourself and no one except the NSA will break it and they'll probably just use a rubber hose on you to get the key.

Some quick and dirty points to ponder:
- hard drives fail. all the damn time. using a RAID-like technology will reduce your pain level when (not if) this occurs because physically removing and replacing a chunk of metal and waiting a day for the resync to finish is far easier then restoring from a backup
- RAID is not a replacement for backups; you need both
- on-site backups are not backups. they have to be off-site to be fire- and theft-proof
- automatic backups (e.g. online) are easy and more likely to make sure you have the latest of everything in your backup
- automatic backups are far more likely to propagate user errors (accidental file deletion or corruption) from your primary system to your backup media
- if you cloud, google up "stackable cryptographic filesystems" like eCryptfs and encfs; they will transparently encrypt your data on its way to the cloud and on your machine while it's powered off.
- do periodic tech-refresh:
- a disc in a cupboard that you last looked at 5 years ago is not an archive or a backup, it's a useless hunk of metal or plastic
- copying all your data to new media every few years is the best way to make sure it lives on; when your old media fills up you buy a bigger one and move to that
- got any Amiga discs? Got any 3.5" discs that you wish you had pulled the data off 15 years ago when you still could? Got anything on a Zip drive? Copy it to HDD right now!
- optical media suck (no really. 100 years is just BS) and are really expensive+bulky compared to hard drives
- if you MUST use optical media (e.g. to post something), use par2 to make it more resilient
- if you use par2, try to make the par2 set span a set of optical discs so that if the root directory of one disc is unreadable, it can be resurrected from the other discs (this is computationally expensive but absolutely worth it if the content needs to survive any more than a couple of years on consumer-grade media)
- DO NOT USE PROPRIETARY DATA FORMATS because the vendors will not be in business in 20 years. use free (libre not gratis) and open-source software and file formats that OSS can process
- got any old Lotus-123 files? Wordstar? That wasn't long ago

Learn to use Linux or one of its brethren; it's not hard and you get high quality data-management tools. Hell, just learn to use rsync for a start! Google up zfs. Google up btrfs and have a play but don't rely on that for real data just yet.

If you use linux and its included raid-like tools, those tools work great with all block devices including USB drives. Your backup drives can happily be a RAID-5 or RAID-Z set in a stack of cheap USB cases, just plug them all in, start your RAID driver and mount the external volume.

koraks
5-Nov-2014, 02:27
I use a Synology Diskstation with two drives in RAID one for local storage, which is really convenient as I can access my data (including photos) from anywhere on my local network - and the internet. It also meshes well with mobile devices and as it automatically thumbnails images, quick browsing of images on e.g. a phone or tablet makes everything even a little better accessible. Moreover, I can share images with friends and family (or optionally anyone, depending on who I give access) without the use of a web hosting service, although I do use the latter for web 'publication'.

Of course, as pointed out, this is a good way of providing online storage, but it's not a proper backup. So the next step will be to plant one of these at a family member's house and set it up so that all data are automatically and constantly replicated between these two Diskstations. Synology offers excellent software and support for all sorts of applications, this being one of them. Moreover, since it essentially runs Linux, it's possible to create custom scripts e.g. using rsync.

While a Synology solution is a little more expensive than entry-level NAS systems (but still not insanely so), it's really the best solution I am currently aware of. Paired with some sort of off-site backup, it seems like the most convenient and easy to use solution there is for my needs.