Photoshop CS3 for Windows and Mac is a 32-bit application. This means that Photoshop, your plugins, and the memory for your image and work must all fit into 4GB. Memory requirements beyond that make Photoshop use the scratch disk to swap data in and out of memory, which slows you down.
Even if you are running a 64-bit operating system, the above still applies. A 32-bit OS also has to carve out a chunk of the 4GB for its own use, so 64-bit theoretically could give a little more of the 4GB to you.
Adobe has stated (Apr 2, 2008) that CS4 will be 64-bit for Windows, but not for Mac. When Photoshop is 64-bit, it will break the 4GB barrier, and make use of far more RAM.
Adobe has not announced a release date for CS4, so it will not be for a while. The issues that are stopping them from going to 64-bit on the Mac may be resolved by then...or not.
In the meantime, the best choice for performance is to have at least 4GB of RAM and a fast scratch disk. Another GB or two of RAM will give some extra space for the operating system to do some of its other tasks, and should help a little.