Photo Shop and working with a Drobo storage.

[bob carnie]

I bought a Drobo unit a year or so ago, and started bringing alias folders to my desktop from work stored on the Drobo .

I have noticed that a year later the Drobo is filling up with data and this seems to translate to slower timelines such as saving files, some adjustments, and moving files around.

I have considered that the Mac Pro I am using is getting slower with age, or am I opening up myself to slower times by working on files directly from the Drobo than now is getting full.
Would loading daily working files to a smaller clean hardrive and working from that speed things Up?

or do you think that it may be time for a new Mac. I try to keep my desktop very clean and my Macs main drive does not have any files stored on it.

Any help would be appreciated... I should know this stuff but frankly I am in new territory with the number of images I have on my server.

[Jac@stafford.net]

I have noticed that a year later the Drobo is filling up with data and this seems to translate to slower timelines such as saving files, some adjustments, and moving files around.

I have considered that the Mac Pro I am using is getting slower with age, or am I opening up myself to slower times by working on files directly from the Drobo than now is getting full.
Would loading daily working files to a smaller clean hardrive and working from that speed things Up?

or do you think that it may be time for a new Mac. I try to keep my desktop very clean and my Macs main drive does not have any files stored on it.

I have never known of a Mac that gets slow because of age. Poor performance usually comes from disc issues, or running out of RAM. If you haven't changed to having more programs open, then RAM is probably not your issue.

How full is the Drobo? How are Photoshop's scratch discs set up? Is one or more on the Drobo? Can you move your aliases to a folder below the desktop?

Have you purged your Mac's logs and caches?

[Jmarmck]

As a general rule I have a work drive. This is normally where the operating system is located as well. The temp space/page file are also located on that disk. The idea is to not transfer large amounts of data between drives and workspaces while in edit. As Jac said a system slowing down normally is due to dwindling resources. In the case of Windows it is also a matter of stability caused by added drivers and background software. Any drive reaching capacity will slow the OS. I find that when the OS drive reaches about 20 gb it starts to slow down.

I recently bought a half terabyte solid state drive as the primary OS and workspace drive. It seems to have cleared up many issues, one of which was drive noise being picked up by an open mic. Read and write times are quicker and it is quieter. Now, the reinstallation of a clean OS is probably responsible for much of that.

I have separate drives for storage of photos and music. But I do not work off these drives. When a project is finished it get transferred to its respective spot..........off the main OS drive. If it is really finished, it is burned to something else and stored.

[Greg Miller]

Your best performance will come from using an internal or eSATA drive. You can use that while working on your image until editing is complete, then move it to the Drobo.

Modern SATA drives transfer data at a theoretic speed of 6 Gbps.

A Drobo connected via ethernet would be limited by ethernet @ 1 GBPS, but more likely the processor of the Drobo. Also, if the Drobo is more than 80% full, you will see degradation in write times.

[Greg Miller]

As a general rule I have a work drive. This is normally where the operating system is located as well. The temp space/page file are also located on that disk. The idea is to not transfer large amounts of data between drives and workspaces while in edit. As Jac said a system slowing down normally is due to dwindling resources. In the case of Windows it is also a matter of stability caused by added drivers and background software. Any drive reaching capacity will slow the OS. I find that when the OS drive reaches about 20 gb it starts to slow down.

I recently bought a half terabyte solid state drive as the primary OS and workspace drive. It seems to have cleared up many issues, one of which was drive noise being picked up by an open mic. Read and write times are quicker and it is quieter. Now, the reinstallation of a clean OS is probably responsible for much of that.

I have separate drives for storage of photos and music. But I do not work off these drives. When a project is finished it get transferred to its respective spot..........off the main OS drive. If it is really finished, it is burned to something else and stored.


I prefer keeping my Photoshop scratch drive on a separate drive than the OS. The OS has it's own swap file and you don't want Photoshop and the OS competing for resources while doing swaps. It may not matter much with an SSD. But with traditional drives I always add an extra internal 80GB hard drive that is dedicated to the Photoshop scratch file. Of course none of this matters if you have enough RAM where Photoshop does not need to swap. Another thing that happens is sometimes you get an orphan Photoshop swap file that eats up a ton of hard drive space (and thereby slowing everything down if the drive is filling up). These are easy to find and delete if the swap file in on a dedicated drive.

The slowness of saving files would be due to the Drobo, as well as moving files around if that means moving files between the Drobo and other drives.
The slowness of adjustments is more likely a matter of filling up the RAM and Photoshop having to swap. Pointing the Photoshop scratch file to a drive other than the OS drive (but not the Drobo) might very well help that.

[bob carnie]

I should add , I am not seeing slowness with adjustments in PS. I am seeing most of the slowness in moving files around.


I have been moving huge amounts of files around on my Drobo to organize my life lately . The Drobo is a 14 tb mirrored drive , and yes I have a lot on it right now .

I have 2300 large size files of current personal work average 200mb. this took awhile to scan , actually a few years and I am trying to organize it for film making and printing.
It took me quite a bit of time over the weekend to move things around.

[Greg Miller]

If your Drobo is more than 80% full, it will slow down - it will have to fragment files it saves into available sectors scattered all over the RAID array. When you exceed 90% it will begin to slow down even more and even more quickly.

[bob carnie]

I am beginning to think this is the cause.

Now a new server is in order... any thing bigger than the Drobo I am using you can suggest Greg? I actually am getting panicked about this as what the hell would I do if the current Drobo goes kaput?


If your Drobo is more than 80% full, it will slow down - it will have to fragment files it save it in available sectors scattered all over the RAID array. When you exceed 90% it will begin to slow down even more and even more quickly.

[Greg Miller]

I am beginning to think this is the cause.

Now a new server is in order... any thing bigger than the Drobo I am using you can suggest Greg? I actually am getting panicked about this as what the hell would I do if the current Drobo goes kaput?

I'm a big fan of Synology NAS products. You can get a 24 Bay NAS: https://www.synology.com/en-us/products/overview/DS2413+ (they also have them with fewer bays).
That would last you for a while.

I like to create several 2 or 3 TB partitions within the RAID array. Synology has a built in backup process so you can attach 2TB or 3TB USB drives and backup data to them automatically and take some off-site.

[bob carnie]

Ok I will look into this and I may have a few more questions for you... thanks a bunch


I'm a big fan of Synology NAS products. You can get a 24 Bay NAS: https://www.synology.com/en-us/products/overview/DS2413+ (they also have them with fewer bays).
That would last you for a while.

I like to create several 2 or 3 TB partitions within the RAID array. Synology has a built in backup process so you can attach 2TB or 3TB USB drives and backup data to them automatically and take some off-site.

[paulr]

Any drive that's more than 50% full will be dramatically slower than when it was new. It's not just fragmenting; your data will start occupying physical drive regions that are closer to the spindle, where linear speeds are slowest. It just takes more time to read and write data. I keep my data drives until half full, and then re-purpose them as backup drives (where performance doesn't matter ... I'll fill a backup drive all the way).

As far as scratch drives for PS, it's questionable economics to try to speed things up with fast, dedicated drives. This used to be the only way, back when PS was limited to a couple of gigabytes of RAM. Now it's generally better just bump up the ram so you use scratch as little as possible. Especially true if you don't have an earlier mac pro (like mine) that uses ridiculously expensive error-correcting DIMMs.

It's helpful to turn on the "efficiency" info option on the bottom left of the main PS window. This will tell you the degree to which you're using the scratch disk. 100% efficiency means not at all. Once you get to 70%, you'll probably notice the slowdown. There are many things besides file size effect scratch use. Probably the biggest one is the number of history states you ask PS to remember. If you can give up a little history, you'll use less scratch. Also the nature of the operations you do in PS. Anything that effects the whole image is going to require more scratch than something that just effects a single image tile. And of course, anything that effects multiple layers will require more than something that effects just one. Photoshop actually writes the minimum number of tiles to scratch that it can get away with.

If things get sluggish and you've tried everything else, you can purge the history states, and a lot of other cached stuff (edit menu). This gives a fresh start.

[Greg Miller]

As far as scratch drives for PS, it's questionable economics to try to speed things up with fast, dedicated drives. This used to be the only way, back when PS was limited to a couple of gigabytes of RAM. Now it's generally better just bump up the ram so you use scratch as little as possible. Especially true if you don't have an earlier mac pro (like mine) that uses ridiculously expensive error-correcting DIMMs.


It's still the case. The logic has always been maximize RAM above all else. Once the RAM is consumed then Photoshop starts paging to a swap file. At that point having a dedicated fats drive will keep performance as optimal as possible. And since that tends to happen a lot more frequently than saving files, I would do that before replacing a drive that is 50% full. I have been managing and dealing with high performance databases for many many years (even being on the bleeding edge of implementing very very large databases), and I have never heard anyone suggest replacing drives when they were 50% full. Even if performance degrades significantly at 51% full, I can't imagine it making a meaningful difference as seldom as most people save a file. IF it adds 2 seconds to saving a gigabyte file, who is really going to notice,

[Preston]

Bob,

I was looking for a backup solution and did quite a bit of research on the various 'small business' NAS units. I agree with Greg regarding the units from Synology. They are definitely worth a serious look. Go to Anandtech (http://www.anandtech.com/) and look at their in-depth reviews.

One of the best things I did was to install an SSD (Intel) as my OS drive. It's a 240 GB and contains my OS (Windoze), programs and my 'working' files. The performance difference over the standard HD's is pretty impressive. I have a pretty decent box and CS6 will open a 'working' 1.2 GB PSD in about 7 seconds--not too shabby. If you have a large enough SSD to hold your working files, and you have a goodly amount of RAM, PS likely won't be going to the scratch disk. Keep in mind that SSD's don't like to be more that 75-90% full. They use the unused space for TRIM, so don't overfill them. Also SSD's should not be defragmented, doing so will reduce their life.

You don't mention which version of PS you're using, but here's a couple of articles from Adobe regarding fine tuning of Photo Shop. They helped me.

CS5: http://blogs.adobe.com/crawlspace/2011/05/how-to-tune-photoshop-cs5-for-peak-performance.html

CS6:
http://blogs.adobe.com/crawlspace/2012/10/how-to-tune-photoshop-cs6-for-peak-performance.html (http://blogs.adobe.com/crawlspace/2012/10/how-to-tune-photoshop-cs6-for-peak-performance.html)

--P

[Light Guru]

The Drobo is a 14 tb mirrored drive , and yes I have a lot on it right now

Mirrored to what? The redundancy djrob os do is NOT mirroring, so if you are mirroring then what are you mirroring to and how is that mirroring set up. If you really are mirroring in some weird setup this could slow things down.

To best answer your question about why it is being slow you need to provide more info.

How full is the drive? - total space used vs actual storage capacity. Note that when setting up the Drobo if you choose a larger total potential capacity then the capacity of the drives in the Drobo the mac finder will show the potential capacity. For example I set my Drobo up with a potential max size of 8TB. I currently have 4 1TB drives in it. The the Mac OS finder says I have a 8TB drive but I don't really have 8TB right now. I could replace drives with larger ones to get there though.

How is the Drobo connected to your computer? Not all computer connections are created equal when it comes to speed?

[bob carnie]

Lots of good advice and comments...

To be very frank... when it comes to computers I am a complete luddite, I can operate photoshop to my needs , I understand the numbers and how they relate to the materials I work with, but I think I missed the boat on technical computer knowledge. I use CS3 at work which is linked to the system I am discussing right now, at home I use CS6.
With my new move I am considering going to CS on Cloud package for multiple stations..
I think I need to address this storage situation now before it becomes a nightmare. I have so much critical data on the Drobo right now and if something were to happen to that information I will need a surgeon to take out the pen lodged in my skull.


I am going to have someone who is a geek come in see this thread and see if I can answer some of the questions and observations you folks are making. He can look completely at this
system and give me the answers some are asking..

Light Guru.. you scare me with your question... because I cannot answer your question Mirrored to what ??


In my lab setup it must be noted that I am using different generation equipment... Lambda , creo and imacan scanners 1995-99 Mac Pro- Epson 7800 - 2006-2008 - PS3 and PS6 now going to Cloud.
Also driving the the scanners are and assortment of older macs and in some cases IBM computers with software that is all different.

Unfortunately for me Or fortunately my world is one of various platforms and I wish that I could just use one computer, one scanner, one printer- this would allow me to understand and learn faster, but I am trying to understand 20 years of technology and still run the business. Also like many of you I am over 60 and we all know what that means.


In a perfect world - I would have HALY- she would be my main storage where everything is sent to for archival storage and retrieval when needed.
-Then I would have a monster fast Computer drive that would allow me to operate my day to day editing, downloads , uploads.
-Then moving forward I could figure a way to drive all the older equipment with state of art computing that is easily replacable ( try finding software for a Fuji Frontier scanner and have it installed} not fun.
-All my software would come from above and I would pay a reasonable fee for my collective usage of PS , Illustrator ect.

I may be screwing up my own thread with the nonsense above but if I could find this perfect world where I do not have to worry about my workflow slowing down it would be priceless.

[Greg Miller]

Light Guru.. you scare me with your question... because I cannot answer your question Mirrored to what?

No need to be scared. Light Guru was using "mirrored" in a literal sense. You were using it in a generic sense. Drobo has their own proprietary system (patterned after RAID) that provides data redundancy across multiple drive - a single drive failure will not cause you to lose data. Mirroring, in a technical sense, is a specific type of redundancy where 2 (or more) drives have an exact same copy of the data. That isn't how RAID or Drobo's system work. But they are providing data redundancy by having the same data exist on multiple drives, which is what I believe you were try to convey. You do not technically have mirroring on a Drobo, but you do have data redundancy to protect against a drive failure.

[Light Guru]

No need to be scared. Light Guru was using "mirrored" in a literal sense. You were using it in a generic sense. Drobo has their own proprietary system (patterned after RAID) that provides data redundancy across multiple drive - a single drive failure will not cause you to lose data. Mirroring, in a technical sense, is a specific type of redundancy where 2 (or more) drives have an exact same copy of the data. That isn't how RAID or Drobo's system work. But they are providing data redundancy by having the same data exist on multiple drives, which is what I believe you were try to convey. You do not technically have mirroring on a Drobo, but you do have data redundancy to protect against a drive failure.

I'm betting you are right and the OP is just using the wrong terminology.

The OP also seems to have ignored the other questions that I and others have asked.

How full is the drive? I and at least one other person have asked this and the OP has commented several times on the thread and NOT answered it. We ask the question because drive performance can be linked to hoe full it is (especially when using devices like the Drobo)

Open the Drobo dashboard, select your Drobo and tell us what it says for capacity and for space used.

Also how is the Drobo connected to your computer. I ask because Some connection types are slower then others.

Also what model Drobo do you have?

We cannot help you unless you answer the questions we ask.

[polyglot]

If this data is valuable, you must also consider (off-site) backups, even if they're refreshed only once every month or so. If the drobo breaks or eats its data or you accidentally delete a file or the box is stolen, what next? Having a stack of HDDs sitting in a cupboard a couple of suburbs away is something everyone should seriously think about. Doubly so if your income depends on that data!

Mirroring and redundancy within a device is absolutely no replacement for real backups that are physically separated from anything that might go wrong with your primary copy.

[bob carnie]

Light Guru

I did answer this , I am having someone look into this and report back to answer some of the questions asked.
One of the drives has failed and we are replacing that right now.

I will try to answer some of your questions. It is a five drive unit Drobo . I believe a Drobo 5D
It is connected through a Netgear Pro Safe 24 port global switch
Each computer is cabled to this switch
As I was working the bottom line told me I had 13.5tb available
I believe (need geek) that the Drobo has two main folders... Archive .... Production....
I have been lately dragging Alias from Drobo and working on files and saving back into Drobo
Before that I was working off a separate Lacie Drive via Alias and then saving into the Archive on Drobo files finished.

Bob




I'm betting you are right and the OP is just using the wrong terminology.

The OP also seems to have ignored the other questions that I and others have asked.

How full is the drive? I and at least one other person have asked this and the OP has commented several times on the thread and NOT answered it. We ask the question because drive performance can be linked to hoe full it is (especially when using devices like the Drobo)

Open the Drobo dashboard, select your Drobo and tell us what it says for capacity and for space used.

Also how is the Drobo connected to your computer. I ask because Some connection types are slower then others.

Also what model Drobo do you have?

We cannot help you unless you answer the questions we ask.

[Greg Miller]

Light Guru

I did answer this , I am having someone look into this and report back to answer some of the questions asked.
One of the drives has failed and we are replacing that right now.

I will try to answer some of your questions. It is a five drive unit Drobo . I believe a Drobo 5D
It is connected through a Netgear Pro Safe 24 port global switch
Each computer is cabled to this switch
As I was working the bottom line told me I had 13.5tb available
I believe (need geek) that the Drobo has two main folders... Archive .... Production....
I have been lately dragging Alias from Drobo and working on files and saving back into Drobo
Before that I was working off a separate Lacie Drive via Alias and then saving into the Archive on Drobo files finished.

Bob

Just looking at your numbers and making some assumptions, my guess is that you have five 3 TB drives. That mean no redundancy. You should definitely get that checked. You really want raid, or drobo flavor of raid if the data is important at all. Also as i mentioned before, it is handy to partition the array into 2tb or 3tb chunks so that it is easy to backup each partition to a 2tb or 3tb USB drive (taking some off site to protect against fire, theft, water,...).

[Amedeus]

There's a lot of good stuff in this thread.

My work flow.

All my raw files are stored on an 8 bay, 3Tb/bay Drobo. 24T. I'm 80% full and I'm not noticing a "slow down"

Drobo is only used for storage, not working on files.

I have a 250Gb scratchdisk for CS6

All my retouching and editing work ends up on my main drive (1T, MacPro) and when finished goes to Drobo and one other extermal 4Tb RAID which stores CS6 files only. Once full, that drive gets replaced. My main drive gets cleaned off when I reach about 800Gb.

Once a quarter all my files, PSD and original digital or scanned film files are backed up on drives that are stored in my office in a sizable suitcase, 7 miles from my studio. This is a process that can take a few evenings of my time ;-) I acutally wish I could afford a second Drobo or similar at this time for easier maintenance rather than lugging around the suitcase but that's another story.

Backing up on a single device, RAID or not, is not considered a viable back-up strategy in my mind. Hard drives do fail and we believe we are safe with RAIDs. Sometimes we are. Power supplies fail and are capable of wiping out multiple hard drives in a RAID. Control boards fail and restripe/wipe parts or all from your RAID (Had this happen on a Buffalo RAID ... operating system bug, all data lost on the RAID ... I did have a back-up at the office ... )

YMMV

[Kirk Gittings]

Hard drives do fail

I keep an original HD and a backup for each year. Used to be 1GB each but recently gone over to 2TB each. I assumed I was safe. A few years ago I needed to pull some stock for a magazine request from storage 5-6 years pervious. The original failed so I walked over to get the backup. On the way back I bumped my elbow on a shelf and dropped the back up. Neither was recoverable-a whole years worth of work. I had to contact all my clients and get the files I had delivered back-never have completely. Now in addition I have a web backup.

[Light Guru]

It is connected through a Netgear Pro Safe 24 port global switch Each computer is cabled to this switch

Each computer? If you are using the same drive from multiple computers at the same time this will definitely show down performance.


As I was working the bottom line told me I had 13.5tb available

By bottom line are you referring the the bottom line in the OSX finder?

You CANNOT judge free space on a Drobo via the Finder. Like I said in a previous comment when setting up a Drobo you select what size you want the drive to max out at, this is the size that the Drobo tells the computer for how big the drive is.

For example when I set up my Drobo I set it up to so that it could be up to 8TB in size, however I only have 4 1TB drives in the Drobo. Because of this the Finder acutely reports the drive to have more free space then total storage space.

119595
119597

This is why I said to get the size and free space from the Drobo Dashboard.

It is vary posible your Drobo is almost full (thus causing performance issues) even though the finder says it has lots of free space.

[bob carnie]

Thanks
Learning a lot thanks to all of you.

I am finding out that what you are saying is exactly right. I am using up a lot of space with redundant files so I am deleting copies this weekend.


What about Cloud storage for the most important files ( I have 2300 files of my personal work - file size averages 200 mb, - I cannot lose this stuff.
not at least until I print the edition so you can see this will take me years, took me years to scan.

I am being advised google, amazon cloud storage and just pay for it .

any thoughts ??




Each computer? If you are using the same drive from multiple computers at the same time this will definitely show down performance.



By bottom line are you referring the the bottom line in the OSX finder?

You CANNOT judge free space on a Drobo via the Finder. Like I said in a previous comment when setting up a Drobo you select what size you want the drive to max out at, this is the size that the Drobo tells the computer for how big the drive is.

For example when I set up my Drobo I set it up to so that it could be up to 8TB in size, however I only have 4 1TB drives in the Drobo. Because of this the Finder acutely reports the drive to have more free space then total storage space.

119595
119597

This is why I said to get the size and free space from the Drobo Dashboard.

It is vary posible your Drobo is almost full (thus causing performance issues) even though the finder says it has lots of free space.

[Greg Miller]

Thanks
Learning a lot thanks to all of you.

I am finding out that what you are saying is exactly right. I am using up a lot of space with redundant files so I am deleting copies this weekend.


What about Cloud storage for the most important files ( I have 2300 files of my personal work - file size averages 200 mb, - I cannot lose this stuff.
not at least until I print the edition so you can see this will take me years, took me years to scan.

I am being advised google, amazon cloud storage and just pay for it .

any thoughts ??

Cloud storage is great. The challenge is getting large amounts to the cloud at Internet speeds. Unless you have very fast upload speeds, it will take months and months to get the data to Google or amazon..you can look at cloud storage where you can seed the cloud by shipping a USB drive. But those plans tend to cost significantly more than amazon.

[bob carnie]

I have pretty much standard upload speed for small business, nothing beyond that, I will look into how to do this using USB drives.



Cloud storage is great. The challenge is getting large amounts to the cloud at Internet speeds. Unless you have very fast upload speeds, it will take months and months to get the data to Google or amazon..you can look at cloud storage where you can seed the cloud by shipping a USB drive. But those plans tend to cost significantly more than amazon.