Photoshop and Computing Power

[Ted Harris]

As my scanner capabilities grow sodo the sixe of the files I am creating and hence the the need for more and more computing power. I am now regularly scanning 4x5 and 5x7 chromes @ 2400 - 3200 ppi resulting in files from 500 Mb to 1 Gb. My current Dual Processor 867 Mhz G4 with 1.75 Gb of RAM is gasping. Opening a file can take 4-6 minutes and most Photoshopoerpations take as long or nearly as long, this even with dedicting 75% of available memory to Photoshop and plenty of storage space for scratch disks. So the question.

Working with such large files requires a new configuration but what is the best expenditure? I am leaning toward a G5 dual processor 2 Ghz machine (to which I will add 2 Gb RAM immediately to the installed 512) because of the RAM capabilities and configuration available, OTOH srue is a lot of $$. Anybody else been in this fix recently and solved it and wht sort of speed increases do you see when working with your large image files on the new machine?

Thanks ....

[Frank Petronio]

I think the biggest bang for your buck is the lower processor speed 1.8 mhz dual G5, and spend the savings to get 3-4 gb or RAM rather than just 2 gb,(which is too little IMHO). RAM makes the biggest difference in actual production, even back in the day when 64 mb cost me $2000!

I'd also get the biggest hard drive and a second matching drive for backups, or if you are geeky, a RAID.

Next, I'd get the biggest, best monitor I could afford to put on my desk. Studies have shown that graphic artists are more productive with 21-inch monitors on slow machines compared to 17-inch monitors on fast machines. Scrolling and zooming not only take time, but they slow you down because you have to reorientate yourself each time you move.

Only after than would I spend money on a faster processor (and the video card upgrade would be last of all...)

I used to run employees and buy on this basis, and did a lot of comparisions. Not much has changed - RAM is still the bottleneck.

It also helps to buy in the early to middle of the product cycle, after the early bugs are squashed but there is no replacement model on the horizon for six months or so. The big Mac intros are quarterly, so you can sort of gauge the next cycle.

Smalldog.com sells a lot of refurbs - I've had excellent luck, although I always reformat and resinstall everything whenever I get a new Mac - never just use them out of the box (unless you are my Mom.)

Of course, I would love a dual 30-inch LCD system with 8 gb of RAM!

[Bruce Watson]

I've got a similar problem on a PC. The basis of the problem is, Adobe developes for micro$oft first, and windoze is going to remain 32 bit for the foreseable future. Thus, so will photoshop. It can only handle 2GB internally (not the 4GB you'd think), so with files the size you and I use, it's going to be paging to disk a lot.

I suspect that the biggest bang for your buck is going to be SCSI disks. Still the fastest things around, and if you are working on a 1GB file, you are going to be disk limited because the paging is unavoidable. That's why when you make an adjustment and everything freezes, you hear the disk drive and the light stays solidly lit.

So, I'm sort of with Frank. Slower processors, max out memory, and multiple fast SCSI drives.

[Ted Harris]

Frank,

One additional point ..... there is a logic board and RAM configuration difference between the 1.8 and 2.0 machines and that is why I have not been considering the 1.8 machine. The 1.8 will only accept 4 GB of RAM and it comes with 2 128 MB chips installed in two slots leaving only two slots available (the RAM must be installed in pairs. So you either install 2 GB or 4 GB throwing away the installed 256. The 2 Ghz machine is capable of handling 8 GB of RAM and comes with 512 installed and six slots still available for expansion.

BTW I totally agree on Small Dog, a great company but unfortunately they don;t have any refurbs that seem to fill the bill for me right now. I have purchased several machines and a lot of other stuff there in the past. You can also see our dogs there ... look for Ted and Amy ...

Thanks

[Frank Petronio]

You're right. I haven't been in the market and keeping up on the specs, but the faster bus of the 2.0 over the 1.8 might make a worthwhile difference (but not sure where the payoff is...)

Even with the 2gb limit on 32-bit, can't you use more RAM for the OS and other apps?

One thing I do is partion the disc to leave a couple gigs as a scratch disc. Always empty, never fragmented. Seems to work well. I also keep my apps in one partition and my work in another - keeps things tidy and I think it hunts less, in theory if not by proven evidence.

I only scan to 120mb files, but they pile up quickly with all the layers and whatnot. I'd hate to think of lots of layers on a 500mb base... but you could do what we used to do back in the 68000 days... do a small file, work out your corrections, save the adjustments and notes, and then apply them to the larger file (and grab a cup of coffee, listen to Depeche Mode, and watch Sienfeld (get into the era...)).

[Armin Seeholzer]

Hi Ted

I will also go for the 2x 2 Giga Mac in the next 2-3 months or if mainwhile the 2.5 comes much cheaper then I take it.
But I also prefer the 2 Giga with filled fully up with Ram then the 2.5 with not so much Ram.
Good luck.

[Ken Lee]

Photoshop allows you to specify where the swap file(s) are located; which drive(s). If you have a second or third physical drive on your machine, then you can perform read/write operations in parallel. Power users know that the swap disk should be completely empty. PS uses it for temporary storage only, so it never really gets "written to". So in addition to RAM, a second or several physical drives, separate from the drive on which your file is located, will speed things up. I'm not talking about logical drives, but physical drives, each with their own disk and head.

[Matthew Cromer]

You could create a 4GB ram disk for swap if you had an 8GB machine. I bet that would be fast!

[Frank Petronio]

There is this guru guy - Jeff Schewe - who makes more money from consulting than from being a photographer anymore - he writes and talks all over - and he always has the top of the line, ultimate system with beta copies of the next version of Photoshop. He had the quad processor 604 Daystar with accelerators, RAIDs, and gigs of Ram - with two 21-inch Pressviews - back in 1996 or so. He did 300mb Live Picture files in real time "back in the day" when most of us were still using floppies.

Granted, he still did corny stuff like little people standing next to giant pigs, but he did it really well.

If you track him down online (or that Andrew Rodney guy, I think they are partners in a consulting business), he would be a good benchmark for just how hopped up you can get. But after awhile, it does kind of seem like bragging rights, doesn't it?

[Keith S. Walklet]

I'll bump the devoted scratch drive concept and add the Layer Tranfer Technique for increased efficiency. I'm only running a 400 mhz G-4 with 1.5 gbs of RAM, though like yourself, I am starting to feel the ache to move up to a new system since I've been scanning files that end up 500mb to 1GB.

I watched my friend work on his G5 with full-size 16 bit images as quickly as my machine operates with 8 bit. I simply wouldn't be able to work with these monster files if I didn't use the Layer Transfer Technique. Since most of the heavy lifting Photoshop does is with all the calculations with layers, etc. this techique advocates downsampling the original scan to a smaller discreet working file (base 50 mb) in which you add all your layers, and when things look right, upsampling the layers file, pasting the high res data into the file. Photoshop correspondingly upsamples the all the layers with the resize (which are mostly generalizations anyway). It is only in the final steps, upsampling, copying and pasting the data, that things slow down.

If you're not familiar, this is my variation of the process as passed along by Rich Sieling of West Coast Imaging. They use the process in their workflow:

A. Clean up your scan:
1. Spot, crop and save your scan give it a name something like "ABCD_spotted_scan.psd". (Rich spots and crops after he does the Layer Transfer process, but I do it first, preferring not to have a dirty image to look at, and figuring I will at least crop to working dimensions that eliminate extraneous data).

B. Prepare the LT File:

1. Open the spotted file, downsample to 8x10 at 360 dpi (check constrain proportions and change dpi to 360, and the long side to 10 and let the computer calculate the short side)

2. Save as a discreet file called something like "ABCD_8x10_LT_UF.psd"

3. Start adding your layers and adjustments to that file. Once you've got the file looking the way you like it, SAVE IT! in unflatted form.

C. Prepping for the "Layer Transfer":

1. Open "ABCD_spotted_scan" and resize it roughly to your target output size --again with proportions constrained--set the dpi to the intended output resolution. (I use 360 since I run an Epson 9600) and the long side to your intended dimensions (say 20 inches) and let the computer calculate the short side. For the sake of this example, for film with an aspect ratio of 4x5 you'd end up with dimensions of 7200 pixels (20 inches) on the long side by something in the neighborhood of 5850 pixels (16 inches) on the short side.

2. Make note of what the exact size of the second dimension is in pixels (write it down because you'll need it in a moment).

3. Select all and EDIT>COPY

D. Upsample the "ABCD_8x10_LT_UF.psd" file:

1. Activate the "ABCD_8x10_LT_UF.psd" file and under IMAGE>IMAGE SIZE--with proportions constrained--enter the same size as you made the spotted file, matching the dpi and long side. Let the computer calculate the short side. If you're lucky, it will end up calculating the same number as step C-1 example (5850). Sometimes it is a couple of pixels off due to rounding errors.

2. If the dimensions match exactly, proceed to step 4. If they don't , go to step 3.

3. Uncheck constrain proportions, physically enter the proper short side dimension from step C-1 and press enter to finalize the resize.

4. Select the background layer and EDIT>PASTE the copied information from the spotted file into your upsampled LT file.

5. Zoom in to actual pixel level and either click the eyeball on and off (or adust the opacity up and down) for this new layer to inspect and make sure the data from the high res spotted file lines up exactly with the low res data from the LT file.

6. If everything has worked properly, the data lines up and you can discard the background layer and you have an unflattened high resolution file that you can either save as "ABCD 16x20 MASTER UF", or flatten and sharpen for output, depending on your preferences.

Prior to output, the data gets converted to 8 bit, but the advantage of this apporach is that you've been able to work quickly but retain the integrity of the original high-resolution file throughout the process. Think about how much faster it would be with a new machine!

[John Berry ( Roadkill )]

I second Frank Petronio on Jeff schewe. I went to both his seminars here in Seattle and he is the bomb. I learned more in those two sessions than I would have learned on my own in years. He has lectures in pdf format at his site jeffschewe.com check it out

[Kirk Gittings]

Keith,
Some of you guys scare me! I remember when"state of the art" technology was a Zone VI and Tri-X in HC-110. I'm not sure my 55 year old brain is ready for this.

[QT Luong]

Keith, thanks for sharing with us this useful workflow.

It is stated on Adobe's web site that PS currently can use only 2GB of memory, so I assume unless you like to leave tons of other applications open, going above 2.5G wouldn't help too much. Study barefeats.com for reviews and comparative speed tests for macs. The author seems to be of the opinion that nowadays the marginal performance boost of SCSI doesn't really justify the huge difference in cost.

Myself, in general, I work only with 300MB files max, and my G5/2.0 has been quite fast. Check also the Apple store for good deals on refurbs. I got mine there.

[Lars Åke Vinberg]

When your file sizes are large enough - and they probably will be - the limiting factor will be disk read/write bandwidth. 2GB of RAM might be enough, but PS writes to scratch disk long before the memory is fully used.

More specifically, the bandwidth to the scratch disk is much more important than the bandwidth to the file storage disk, as PS writes en obscene amount of data to the scratch disk. For example, I stitched two part scans of a 6x17 image - about 600MB each, 16-bit depth. During this session my scratch file grew to about 18 GB. The operations were simple - open both files, copy one, close, resize canvas on other, paste, position, smooth overlap. 18GB temp file.

Additonally, and this has been said before - it is very important for good performance that the scratch disk is on a separate disk drive that is not used for anything else while PS is running. While travelling I use a laptop with a Serial ATA card and an external 3.5" SATA disk for scratch disk that gives 60MB/s transfer speed. PS performance is good. If I use a Firewire connection to the same disk then PS performance drops by 50%, and using USB2 performance is way down.

For absolutely optimal performance, a RAID or partition striping configuration should be used for the volume used for scratch disk.

[Keith S. Walklet]

> I'm not sure my 55 year old brain is ready for this.

LOL! It is actually much easier to do than it is to describe, especially without providing illustrations of each step. It might help if you print out the instructions and use a colored highlighter pen to differentiate between the discreet file names, so you can see at a glance which one each instruction is aimed at. They are so similar when just presented in 12pt Times that it just makes everything more confusing.

BTW, I've heard rumors that the version of Photoshop under development is supposed to blow the lid off of max usuable RAM, so hopefully by next year, one will be able to take full advantage of the 6gbs that a fully outfitted G5 can handle. In the mean time, the devoted scratch disk is the way to go for hardware and in PS preferences, one can designate a hierarchy of drives to use in the event that the scratch ceiling is exceeded. Just make sure to avoid using the same drive you are reading/writing to as your scratch drive or things slooooowwww way down.

[Martin Courtenay-Blake]

Once you are happy with your processor and RAM options...the more the merrier of both..then optimising disc strorage is the next key area. The biggest mistake most people make is using internal RAID controllers or, even worse, on board RAID chips. These simply eat into your main system performance.

There are three main options for high performance external storage. The fastest would be to use a fiber - fiber system. This uses a fiber HBA (controller card) in the computer and connects to an external RAID controller attached to a number of FC-AL drives. 10Gb/s FC-AL is readily available and is very very fast but expensive.

The next option would be to use U320 scsi HBA's and RAID systems. This is a little less expensive and will offer very high levels of performance.

The most cost effective way of getting high disk transfer rates and access speeds is to use U320 HBA's with a SCSI - SATA RAID system. This uses the easily obtainable and relatively cheap S-ATA discs. Again the key is in the external RAID controller.

The best performance is obtained using disc stripping. Using RAID 0 data is written across all the discs in the system which increases speed. This does, however, leave you open to data loss should a disc fail. The safest option is RAID 5 where one disc of the set is written as a parity disc enabling data sets to be fully recovered in the event of a disc failure.

Finally, both Fiber and SCSI systems (including some that use S-ATA discs) can be "Dual Hosted". This means that the computer is connected to the external storage system with two cables from two host channels...simply doubling the performance.

In case you are wondering I have spent many years designing and building high performance computing and storage systems.

Martin

[Keith S. Walklet]

Hah, Martin! Now my head hurts! But it actually makes sense...

As an aside, I forgot to mention why I deal with files over 300mb. In the beginning (1996) I was working with base files between 100mb (35mm) and 300mb (4x5) scanned on a TANGO drum scanner, which more than adequate for my imaging needs.

Still, being the control freak that I am, I sought to have more and more control of the imaging process from start to finish and the only equipment that remained priced out of my range was my own personal drum scanner. As more and more "prosumer" scanners were introduced and went through the typical electronic product cycle and dramatic price drops, it became impossible to ignore the temptation to try them out.

While very, very good, the Nikon 8000 and Epson 4870 aren't equivalent to the resolution of a TANGO scan, so I do everything I can to squeeze the last bit of data out of them. That means scanning at their optical resolution and greatest bit depth (14 for the Nikon and 16 for the Espon). The resulting files from these scans are typically 100mb (35mm at 4000 dpi and 14 bit on the Nikon), 585mb (4000 dpi and 14 bit on the Nikon), and 1 gb for 4x5 (3200 dpi and 16 bit on the Epson). Then I use the Layer Transfer method to shrink them to workable size, but reinstate the full data in the final stages of that process.

[Greg Miller]

The 2 best bangs for the buck are:

1) second hard drive dedicated to PD scratch disk

or

2) Add more than 2GB of RAM. PS will use 2 GB, the OS can use some of the > 2GB, and aloocate the remainder to PS scratch file. Havinf the scratch file on a RAM disk will be significantly fatser than having the scratch file on a hard drive. Go this route if you can afford the extra RAM.

I consistently work on 1.5 GB files when stitching 13+ images at a time. No need for layer transfers or other gimmicks.

www.GregMillerPhotography.com

[QT Luong]

Greg, how do you allocate RAM as PS scratch space ? It sounds a sure (although expensive) way to increase performance.

[Keith S. Walklet]

Good info Greg. Could you clarify whether your base files (scan only, no adjustment layers) are 1.5 gigs or are your layered and stitched files 1.5 gigs. Even with an 8x10 at 300 dpi, I am running into multiple gigs on a layered file (which my machine handles well enough), and when I res up in preparation of the Layer Transfer, I've hit 20 gigs before flattening for output.

[Greg Miller]

QT - first you need software to to set up a RAM disk (i.e. AR RAM Disk (freeware)). The RAM disk will have a drive letter (for example "R:") just like a hard drive. So "R:" will look like a hard drive but will actually be a specific amount of RAM. Then just point PS scratch to be "R:", THe only expense is for any extra RAM that you purchase.

Keith - my base files tend to range from 500mb to 1 GB (total; individual images range from 50mb to 500mb depending on format (35mm or 4x5)) and grow from there to up to 1.99 GB due to layers. Note that PS will not save a file larger than 2GB.

[Michael Chmilar]

Unfortunately, the next version of Photoshop (CS 2.0, or PS 9.0) will not break the 2GB barrier. According to thinksecret.com:



Photoshop CS 2.0 will not feature 64-bit support, but lays the groundwork for that support, which will arrive with Photoshop 10 (presumably CS 3.0). Photoshop users will also have to wait until Version 10 for the software to use more than 2GB of RAM.



Personally, Adobe will not get my upgrade money for 9.0, since breaking 2GB is the most important upgrade for me. I will wait for 10.0.