By Ctein

A few weeks back, a reader wrote to Mike comparing the sharpness of prints made with his 6-MP and 12-MP digital cameras. He put in some effort to compare apples and apples. Both cameras had the same format, and he used the same high quality prime lens on both cameras at the same ISO and aperture. He worked on a tripod, photographing the same target at the same distance under the same conditions, and he printed out the photos using the same workflow in several different sizes. He saw little difference in sharpness between the two cameras. Apparently, up to fairly large print sizes, it didn’t matter whether he used a 6- or a 12-megapixel camera.

The reader had made an error. He’d stopped down the lens all the way to ƒ/11. That meant his likely blur circle was larger than the pixel size on either camera. I guesstimated that the most resolution difference between the two cameras he could hope to see was 10–15%, so small as to be nearly undetectable in real photographs.

A good equipment test is really a well-designed scientific experiment that measures the variables on which you want to collect high-quality data. Few people have the knowledge or skills to design such experiments. There’s a reason people go to expensive schools for many years to learn how to do this. The majority of tests that I read about online are done incorrectly. When they produce unexpected results, as they did in the case of this reader, the first thing to investigate is the testing procedure.

I’ve written many columns and comments about print sharpness and what people can and can’t see. There have been many comments from readers who couldn’t verify my assertions, making comparison prints on their inkjet printers. The readers neglected to determine whether their printers were even capable of reproducing such fine detail before making the comparison prints; they weren’t. Like Mike’s correspondent, their experiments were too crude to measure what they were looking for.

That’s why I was so happy with this column back in January: “How Sharp Is Your Printer; How Much Can You See?” It’s not easy to come up with a robust test that a non-expert user could run that tells them something useful and meaningful about what kind of fine detail they’re getting and seeing.

There’s another way tests go wrong. Mike has described to me how tests of total harmonic distortion failed to tell us how audio equipment would sound, because it really matters to the human ear which harmonics get distorted. Once you know that, you can design an appropriate test, but the most competently-executed THD test won’t tell you what you want to know.

I just ran into this with my Olympus Pen E-P1 (which I am very a very happy with, don’t get me wrong.) Published tests report a pleasantly low level of noise with this camera. DxOMark is still my favorite for quantitative information, and they have comparison data for my old Fuji S100fs and the Olympus. My field tests pretty much agree—with a glaring exception. No one’s noise tests distinguish between visually-random noise and noise that appears as a regular pattern. By the time electronic noise gets massaged into a photograph random electronic variations can turn into a very non-random artifacts.

I photographed a moonbow on New Year’s Eve, handheld with the EP-1 (God, but I love an ƒ/1.7 lens and image stabilization!) The top image was taken at ISO 800, the lower one at ISO 1600. In these comparison illustrations I’ve cranked the chroma all the way up to “11″ to make the noise readily visible; it’s nowhere as obvious in the original photographs.

-

ISO 800

ISO 1600 (click on either image to see it larger)

The increase in truly random noise between ISO 800 and 1600 wouldn’t make the higher-speed photograph unusable, especially if I ran it through good noise reduction software. The color banding that you can see in the ISO 1600 shot does. That’s a manifestation of noise that isn’t reported in the published tests, and it’s a real killer. At ISO 800, the nonrandom noise isn’t a showstopper. At ISO 1600, it is unacceptably distracting to the viewer in the correctly-printed photograph, and it’s not something I can filter with software.

Now, keep in mind that this is the kind of subject which is most likely to bring out such noise—broad uniform areas of low saturation in the dark part of the tonal scale. A more variegated scene would disguise this noise. What it means, though, is that I can’t really count on this camera for quality results above ISO 800. That’s about a half a stop faster than quality results out of the Fuji S100. The thing is, based on the simple noise data, I should get nearly two stops more usable speed out of the Olympics. Clearly, not gonna happen. That’s a disappointment, and it’s a failure of the testing protocols. The data being reported is technically, but not visually, accurate.

Tests. They don’t always tell you what you need to know.

Ctein’s regular weekly column appears on TOP every Thursday morning.

Send this post to a friend

Note: Links in this post may be to our affiliates; sales through affiliate links may benefit this site. More…