I use a 2009 Mini as a back-up. It got upgraded to 8Gb RAM and a 250Gb SSD, works like a charm.
There are later Minis that will accept more RAM. For instance, the 2012 i7 models would be a great fit for your purposes and are under $500.
Maybe the latest Minis aren't necessarily the best option.
Or consider the all-in-one iMac, very powerful and a 2012 model is easily upgraded if you want.

I use PS and LR quite a lot with large scanned images, and I'm running a 2011 iMac as my main computer.