Need Help with Statistics Formula in Shutter Test Spreadsheet

[Larry Huppert]

I'm trying to improve the tests I perform on my shutters on a periodic basis. P reviously, I used 3 samples for a particular shutter speed to determine an avera ge shutter speed and f-stop difference between reality and the listed speed.

Are three samples enough?

I'd also like to calculate a goodness factor for the consistency between measure d samples. If I'm remembering way far back in my memory, I believe standard dev iation is what I'm looking for. Is this correct?

Assuming this is correct, how would you convert a calculated std dev based on ti me samples into a std dev based on f-stops?

[Pete Andrews]

I'm sorry Larry. I thought that I could be as pedantic as the best, but exactly how will all this help you get better pictures?

[Larry Huppert]

Pete,

re: "...exactly how will all this help you get better pictures?"

A simple answer - more accurate exposures, and knowing if your shutter needs service. If I remember the little owners manual that came with a Copal Press shutter, they consider normal operation of a shutter to be ~ +/- 1/3 of a stop from the listed speed. Hum, when shooting chromes, 1/3 of a stop is significant. In actual test, a brand new out of the box shutter does show this type of variation. By knowing this, I can compensate. The point of the testing is to just know where you stand.

[N Dhananjay]

The average is a decent measure, (although the modal value might be better). Standard deviation is the measure of spread that you are looking for. If you have a normal distribution (i.e., bell shaped curve), about 95% of the time, your speed will be no more than 2 standard deviations away from the average. Hope this helps. DJ

[neil poulsen]

A sample of 3 is not enough. A sample of eight will give you an average which, with a 5% chance for error (this is reasonably small), will be within 1.5 standard deviations of the true value. A sample of sixteen will give you an average that (to the same 5% chance of error) will be within one standard deviation of the true value. Think of a standard deviation as about one sixth of the width of the distribution. (Crude estimate.)

I'm not quite sure what you mean by your last question.

The standard deviation is the measure of consistency between measured samples of which you're thinking. With a good estimate of this value, you can determine how close your average is to the true value using the two sample sizes provided above. Any scientific calculator can calculate the standard deviation. Use the "n-1" versus the "n" calculation, if you're given the choice.

[Sal Santamaura]

I own a Calumet electronic shutter tester, and have never noticed enough variation (with modern Copal shutters) to consider such a statistical approach. Typically everything from 1/125 on down is within 1/12 stop between trials; 1/250 and 1/400 or 1/500 might deviate 1/3 or 1/2 stop respectively. I don't use those high speeds on a view camera, and don't worry about inconsistencies in them. A calibration card for 1 second through 1/125 gets taped to the lensboard. Periodic rechecking confirms the card is still correct, and no such shutter's performance has changed noticably over a dozen years.

I believe you should probably have any shutter that shows significant variation in its lower speeds serviced rather than expend effort on a statistical exercise.

[neil poulsen]

I would want to invest at least five data points, if one is interested in a worthwhile estimate of the average and standard deviation. The data is easy to collect. It's true that, the smaller the variation, the less sample one needs, but to a point.

The "statistical exercise" to which the previous post refers is collecting valid data. Evaluating data in this fashion is done both formally and informally in many disciplines. There's no reason why this kind of evaluation can't extend to photography.

[Gene Crumpler]

I would submit that each shutter speed is a different distribution. What would be more informative is the variation of a single shutter speed over time. Therefore, to get a useful statistic, one would need to test each speed several times and then look at the variability at each speed setting. One can not assume that an error at one setting is also valid for settings near it, i.e. the error at 1/60 second is not necesarily the same at 1/30 or 1/125.

[david clark]

Hi Larry, you could calculate the standard deviation of your shutter speeds; the formula is avalible in any old statistics book. You could probably even do the calculations to see if one speed is significantly different from another. I don't know what criterion a shutter is compared to? as far as standard deviations. Practically, I think you'd just test each speed, one at a time; fire the shutter 15 times or so writing down the times as you go, and it ought to be apparent what the average speed is, how much the shutter varries on each firing, and if the shutter is handing up sometimes. If your shutter is reliable within your accepted tolerances, 1/3 to 1/2 stops, on each indicated speed, just average out the true speeds, and then write them on somethiing that's going to be handy when you need them. I've written them on time tape and placed that on the lens board or lens cap. That's basically all you need I guess. I don't think the standard deviation route is particularly helpful unless you just want to do it to do the math. Cheers, David

[Larry Huppert]

Gene,

Yes, I am testing each shutter speed N number of times so I can see if there is a problem with a particular speed. As an example, on a particular shutter, I see variations from -2/10 stop to +3/10 stop across different shutter speeds. Copal would say this shutter is within spec, but knowing the characterizaton of this 1/2 stop variation is important.

Neil,

My last statement regarding conversion of std dev to f-stops means the following: In a laymans sense, I thought std dev tells you how close together the various samples are to each other relative to the average value. The smaller the std dev the better off you are in this case. I didn't think that std dev actually related to the "true value". For example, at 1/4sec, lets say the ave measured speed is +1/3 stops relative to the "true value" of .25sec. If the measured values are pretty consistent in being off the same amount that is better than multiple samples of testing that speed being inconsistent, but still averaging out to +1/3 stops. Doesn't std dev give you this "goodness" factor? If I do a std dev calculation on a sample of measured shutter times, won't the resulting std dev relate to those numbers in the time domain? What I think I would like to know is that the std dev of the measured values fall within X f-stops of the average. Wouldn't this tend to normalize the result so you don't have to relate the std dev to actual shutter times anymore. I've probably really bastardized mathematical concepts here.

[Sal Santamaura]

Neil, my point was that, with scatter only in the neighborhood of 1/12 stop for shutter speeds I actually use, I've not found a reason to evaluate such data in a statistical exercise. Performing such evaluations for photographic gear is fine, and if Larry either has older, more inconsistent shutters, or just feels like running numbers for his modern shutters, that's his prerogative. I was simply trying to guide him down what seems a more profitable path if his *modern* equipment doesn't fire with typical consistency.

Larry, doesn't the Copal specification you quoted to Pete refer to a deviation from marked speed, rather than variation between individual firings? If so, it is consistent with my data. *Scatter* of that data for Copal shutters is usually no more than 1/12 stop, around a measured value typically within 1/3 stop of marked speeds, 1/250 and 1/400-500 excepted.

[Larry Huppert]

Sal,

I don't have the Copal spec handy. If memory serves me, it said something like "accuracy: +/- 30%". I don't remember seeing any quoted spec regarding variation between individual firings of the shutter at the same speed. Obviously, no deviation would be ideal.

Some replies have suggested it isn't worth the work to check the std dev on modern shutters. Since I'm loading this info into a speadsheet anyway, it didn't seem like that much more work to do the std dev part as well. By calculating a goodness factor, I can treat this whole thing as a truely mind numbing exercise, and just end-up with the data.

[david clark]

Larry, standard deviations only come in 1s, 2s, or 3s. If the variation of true recorded firings when your sample size is about 30 firings falls about/within 66% of your mean value, then your shutter speeds are probably within 1 standard dev. (I think???). But I'm not so sure the assumptions and logic of probabilities apply to this situation as I can't see shutter speeds falling on the normal curve. I think all you need to figure is if basically your shutter fires reliably at some speed which means a grouping within about a 1/3 stop or 1/2 stop. Best, David

[neil poulsen]

Larry,

That's correct; the standard deviation relates each value to the average, not to the "true value" that you describe. Actually, the "true value" that you describe is better called the nominal value. The term "true value" is usually reserved for the population mean, or the "true" shutter speed average at the particular shutter setting that you selected. It's this second type of "true value" to which I was referring. I think we were using the same terminology for two different things.

As to f-stops, I think I see what you mean. Let's call 0.25 secs in your example the nominal value. Let's say that you fire the shutter, and the measured speed is "T". If I understand correctly, the f-stop equivalent to which you refer is the difference between "T" and nominal (0.25secs) in terms of f-stops, and this would be LOG(T/0.25)/LOG(2). (The base of the logarithm doesn't matter.) So, if T=0.5 secs, you would overexpose by one stop. If this is what you mean, then you can collect your data, transform each value to the difference in f-stops, and plot the data in a histogram (or dotplot) to see how the data are distributed in the f-stop domain. To estimate the center of this distribution, and depending on the shape of the distribution in the time domain or in the "f-stop" domain, I would probably calculate the average in the time domain, and then transform this average (using the above formula) to estimate the center of the distribution in the f-stop domain. Again, it depends on the distributions. I would expect that the distribution in the time-domain would be more normal, whereas the distribution in the f-stop domain would be more heavily skewed to the right. But, I don't have the means to take the number of measurements that I would need to check this assertion.

In representing the variation in the f-stop domain, I would probably calculate a 95% confidence interval in the time domain, and then transform these limits to the f-stop domain. In this way, you can be 95% confident that the actual average (the true average) lies between these two values.

If you wouldn't mind sending me your data, it would give me a chance to check this out. Send me 10 values for each of the shutter speeds in which you're interested, and 30 values for your favorite shutter speed. With that data, I could respond with a pretty clear picture of how your shutter is behaving. If you can find them, also send me the specs on your shutter.

As to Sal's point, I would agree that, if a shutter is showing obvious signs of large inconsistencies, then there's no need to test further. But, once the shutter's returned from the repairman, I would find it worthwhile to collect the kind of data to which Larry refers. (In fact, I did!)

neil

[Struan Gray]

The standard deviation and average of your measured shutter speeds is only a useful concept if the shutter is nicely behaved. I agree with Neil that the first thing to do is to plot the shutter speeds as a histogram and check that they lie on a reasonably smooth curve, clustered around a meaningful average value. A really erratic shutter will show up unambiguously at this stage.

The 'best' way to do the measurements is with a photocell, a storage oscilloscope operating in single-shot mode and a computer into which you can download the individual shutter traces for analysis. If you trigger off the flash contact, or have an oscilloscope with a programmable trigger and time delay, you can capture each firing of the shutter automatically. Even with non-press shutters you can build up hundreds of readings with no great effort, but be aware that some shutters behave differently when run at a high duty cycle - I have an old Compur which is fine if I fire it every five minutes, but slows down if I pop it over and over again in quick sucession.

The ultimate analysis involves plotting a two-dimensional histogram of the light intensity behind the shutter versus time. This is a useful diagnostic for bouncing shutter blades or curtains, uneven accelleration, stickyness and a host of other shutter ailments. By integrating the average curve you can get true effective speeds for those high settings on leaf shutters where the blades don't stop moving.

And that last point hints at something which is actually photographically useful: measuring the high speeds of leaf shutters at varying aperture stops. A speed marked '500' on a leaf shutter is only correct for one aperture, since it takes into account the finite time needed to open and close the shutter. At much smaller apertures the shutter blades reache the edges of the aperture much sooner, and the shutter is effectively open for longer. In the worst case, the difference can be up to a stop, far larger than the variability of even antique shutters.

[Larry Huppert]

Struan,

Your last point is well taken regarding faster shutter speeds. When testing the 1/500, 1/400 & 1/250 speeds on Copal (regular) shutters at different apertures, I was amazed how significant the travel time of the shutter is compared to the listed speed. I rarely have any need for anything this fast, and it made me realize that giving up the fast speeds (1/250 and faster) on Copal Press shutters was even less significant than it initially appeared.

Thanks to all for your statistical expertise. I'm going to increase my sample set, and plot some graphs.

[Pete Andrews]

I don't know where this idea that standard deviation can only be an integer value comes from; or that you can have standard deviations (plural). Standard deviation is the RMS (root mean square) value of the deviation from the mean of the samples, of all the individual sample points. It can be any value, fractional or integer.The confusion might be that the samples have to be normalised to a value greater than 1 for standard deviation to be meaningful.

Anyway, I still don't see how knowing this is of any help in getting more accurate exposure.You seem to think that knowing the value of their standard deviation will somehow make your shutters behave less erratically; it won't.A more meaningful analysis would be to know the maximum deviation from the mean of any particular speed, and the frequency, or likelihood, of that deviation occurring. This gives you the odds of getting an exposure that's acceptably accurate.Say you took 10 readings at 1/60th second, and 8 of them were within 1ms of 17ms, but 2 of them were off by +5ms (still only 1/3rd stop). The standard deviation in this case is quite small (~2.36ms), but the odds are still 5:1 that your exposure'll be out by 1/3rd of a stop.If your shutter is truly random in giving you this variation, then you'll have this chance every time you take a shot. You might take 100 shots in a row, and get a bad exposure on every one, then take another 400 where the exposure is bang on.If you don't like those odds, then you'll have to have the shutter serviced or replaced.What I'm trying to illustrate is the difference between statistical analysis, and real probability. No amount of analysis will alter the odds of an event happening.

By all means test your shutter(s) regularly, but don't expect collating data about shutter performance to give you more accurate exposures in reality. If the shutter is erratic, the exposure will still be down to dumb luck at the moment you take the picture.

As for translating exposure time to an f-stop equivalent:Take the actual exposure time and divide it by the ideal exposure time; then take the logarithm of this value and divide it by the log of 2. [log(N+x/N)/log2]. That gives you the deviation in stops.

[neil poulsen]

If one can assume that the distribution is Normal (bell-shaped Gaussian), then the average and standard deviation offers the advantage of being able to obtain the information to which Pete refers in two numbers (the average and S.D.), versus many numbers. So, computing the average and standard deviation is a method of summarization. For example, under the assumption of Normality, 68% of the data will fall in the interval [A-SD,A+SD], 95% of the data will fall in the interval of [A-2SD,A+2SD], and a little more than 99% of the data will fall in the interval [A-3SD,A+3SD].

But, as Pete suggests, there's an advantage of being able to look at multiple representations of the raw data. (e.g. the standard deviation won't necessarily tell someone if a shutter is behaving erratically.)

[Struan Gray]

I think it was Lord Rutherford who said: "If you want to make an accurate measurement, make it once."

>>No amount of analysis will alter the odds of an event happening.

Pete, I think the good Rev. Bayes is buried somewhere near you: that strange whirring sound you can hear is him spinning in his grave. In your example, the odds are 5:1 that you'll be out by 1/3 of a stop *given* that you only bothered to test ten times. Test more often, and you'll have a better idea of how bad the shutter is.

I agree that any outlier is cause to wonder if a CLA is in order, but the price of a CLA is more than I've paid for any of my LF lenses and I see no reason to spend $100 on a repair when I can use $4000 of equipment proving to myself that I can muddle along for another year or so. Since two of my shutters are polariod-press models scavanged from dumpsters I also like the idea that popping the shutter 10 times on 1/20 is more accurate than firing it once on 1/2. Naturally, the cat thinks I'm mad.

[Pete Andrews]

Let him spin Struan, let him spin (I'll bet he stops on double zero - the house wins).

Statisticians seem to think that a significantly 'large' sample is 30 or greater. Now I reckon that if you regularly test a 10 speed shutter 30 times at each speed, setting the speed both in an upward and downward direction, then you'll probably have a statistically greater chance of the shutter failing altogether on an important shot, than if you'd left the darned thing alone and taken the speeds at face value. What do you think to this radical theory?

In practise mechanical shutters rarely give a speed significantly faster than their marked speed, any fault, dirt or wear tends to slow the shutter. A normal, or Gaussian distribution won't result from such conditions, so all bets are off as far as standard deviation is concerned.

[Larry Huppert]

The directions which come with the Calumet tester are interesting in one regard. They suggest you gather 3 samples per shutter speed for each shutter tested. However, they do not suggest you bang the shutter three times in a row. They say to start at 1 sec and march towards the fastest speed, and then from the fastest speed back to 1 sec, ... only taking one measurement each time at each speed.

While testing a particular shutter while not using this methodology (I popped the shutter 8 times in a row at each speed), I found an interesting anomaly which very much supports some of the views expressed here. The shutter was almost dead-on at 1/15 sec, except when it was off in an eratic way - significantly. The average came to be 1/3 stop off the listed speed, however this was meaningless.

If I then tried the Calumet methodology through 8 readings, it measured dead-on. It appeared that a heavy duty cycle on this Copal Press shutter was a problem at this particular speed. The Calumet method is closer to the duty cycle of actual shooting, except for one situation. The serial same speed test was pretty close to how you use a shutter when multipoping a strobe. When balancing ambient vs. strobe light, you still need your shutter to be accurate.

[Struan Gray]

What do you think to this radical theory?

It sounds like the sort of shutter I would own. A properly designed mechanical shutter should cope with a mere 30-pops-per-speed test, but that begs the question as to whether the shutter is in fact properly designed, which is of course one of the things you are trying to find out.



Practical photography is far more about being consistent than it is about being accurate, so I still think that if you are going to test a shutter at all it makes sense to try and find out what the distribution of errors look like. I agree entirely that the shape of the histogram is what matters, not whether you can find a nice analytical expression for the histogram's width.

I've only ever seriously tested on shutter: an electronic model specifically designed for high duty-cycle applications (see www.uniblitz.com). But I suspect that a modern shutter right out of the box will, if tested at normal rates of fire, cluster pretty tightly around the average, even if that average is some tens of percent removed from the marked speed. Both those facts are useful for the photographer to know.

[ross_1450]

Larry

do you have what you need? I am a statistician-md and if you tell me how many measurements you can stand taking, what chance you will take on making and error, and how sure you want to be of the chance of rejecting your estimated value correctly, I will give you some feedback. Ross