Testing Cameras and Lenses

The photo on the right shows a simple but highly effective test rig I use to test the detail and sharpness of all my cameras and lenses. In a bit, we'll get to the nitty gritty of building and using this test rig, but to start off, It's important to understand why all the extra effort of target testing is so beneficial.

Benefits of Target Testing

Here's a partial list of some of the benefits of target testing your cameras and lenses.

• Gain confidence in your equipment.
• Buy top performing cameras and lenses.
• Save thousands of dollars on equipment purchases.
• Pioneer new techniques.
• Pioneer new types of photography.
• Learn what not to do, and what not to buy.
• Make larger gallery-quality prints.
• Analyze and confirm results from the field.
• Accurately predict upcoming camera and lens developments.
• Avoid myths, marketing hype, and sales pressure.
• In the field, efficiently select and configure equipment. This allows near 100% concentration on expertly executing photographic fundamentals.

A Tiny Bit of Science

The whole point of target testing is to use science to help us improve our photography. To this end, understanding a tiny bit of optical and digital imaging science helps immeasurably.

In a nutshell, photographic image quality is controlled by just two primary factors, magnification and system resolution. In turn, magnification is set by focal length and focus distance, and system resolution is primarily set by camera resolution and lens resolution.

Camera resolution is primarily set by physical pixel size. Larger pixels translate into lower pixel density, which ultimately translates into lower camera resolution.

In outline form it looks like this:

• Magnification
  • Longer Lens Focal Length = More Magnification
  • Longer Focus Distance = Less Magnification
• System Resolution
  • Camera Resolution
    • Bigger Pixels = Lower Camera Resolution
  • Lens Resolution

To test lens resolution, magnification and camera resolution must be kept equal.

To test camera resolution, magnification and lens resolution must be kept equal.

To test lenses of differing focal lengths, magnification can be equalized by changing focus distance.

If a zoom lens is involved, magnification can be equalized by keeping focus distance constant and zooming lens focal length.

That's it. Hopefully not too confusing.

Practical Applications

Once you've set up your test rig and gotten the hang of using it, it's amazing the practical things you can do with it. Here's a partial list of some of the practical ways I've used my test rig.

Photographic Fundamentals. The reason I put so much effort into testing all my cameras and lenses is to learn their capabilities and optimal configurations like the back of my hand. That way, in the field, I don't have to worry about any of this. I can put near 100% concentration into expertly executing photographic fundamentals. Ultimately, this is what creates great photographs, not extra detail and sharpness.

Emotion Control. I have a lens that at one time I was absolutely convinced was defective. Testing with an identical borrowed unit proved this was all in my head. After doing a lot of testing, you'll be flabbergasted how much and how often your emotions about equipment fly in the face of reality. And you'll be flabbergasted how much money, time, and angst these emotions end up costing you.

Defective units. Sometimes you actually will have a defective unit.

Digiscoping. Back in 1999, I tried digiscoping with a Nikon Coolpix 950 and a cheap, fixed-eyepiece spotting scope. It was not pretty. Consequently, for many years I harbored the notion digiscopes were poor man's monster lenses. Today, my test rig has shown me monster lenses are dumb man's digiscopes.

Monster Lenses. For many nature photographers, the monster telephoto lens has a way of growing to monstrous proportions in the psyche. It truly is the ultimate nature photography status symbol. If you've fallen under the spell of monster lens envy, I strongly recommend renting the lens of your dreams and a Sigma 50-500mm lens. Thoroughly try-out both these lenses in the field and on your test rig. Create a comparison checklist that includes not only detail and sharpness, but also weight, tripod support, handholding, ergonomics, comfort, logistics, travel, and productivity. Does the extra magnification of your monster lens really provide that much more detail and sharpness? If necessary, repeat the rent and test cycle until you have a thorough understanding of what you're getting, and what you're getting into. $4,000-$9,000 is a lot of money to spend on a single lens. Make sure you're getting your money's worth.

Looking Good Photographing vs. Good Looking Photographs. In the field, I often observe camera equipment being worn as jewelry. I must admit, in the face of all this bling, I sometimes feel twinges of equipment embarrassment for my funky little mongrel digiscopes and unglamourous third-party lenses, and perhaps a bit of regret for giving up endorsements and publicity by using all this incongruous equipment. With testing, however, these feelings quickly pass. Test results give me absolute confidence in my motley outfit. Every bit of equipment is deployed based on results, period. I would much rather wear a bag over my head in the field, than when I show my photographs.

Magnification and Camera Resolution. Most photographers attribute high digital image quality to large numbers of megapixels and high lens resolution. Brand name, high price, and big sensors are also commonly associated with high digital image quality. However, magnification and camera resolution (pixel density) are almost universally ignored. Testing shows this is a huge oversight. Two of the most powerful and inexpensive ways to increase digital image quality are increasing magnification, and using a dSLR camera with higher pixel density.

RAW File Mode. Since March 2000, I've made numerous serious (and seriously expensive) attempts at establishing a high-volume RAW workflow. In all cases, benefits just could not rise far enough above the costs of using RAW in a high-volume operation (especially the costs in time). Recent testing, however, has exposed a new lensmaker gambit of designing professional, lightweight, low-resolution, RAW-only superzoom lenses. For me, such lenses have many great uses, and as a result, RAW processing has gained a solid toehold in my high-volume workflow.

Hoof-in-Mouth Prevention. When commenting about digital image quality, testing often prevents me from making a jackass of myself, and putting a hoof in my mouth. Whenever you comment about the image quality of a camera or lens without first testing your assumptions, chances are good you'll be making an embarassingly incorrect statement.

In-camera Features. Simple on-off tests help establish the efficacy of various in-camera features. For example, target tests show, to my tastes, Nikon in-camera sharpening does more harm than good. Consequently, disabling this feature is part of my LF JPEG exposure baseline.

Etcetera. This is just a starter list of the many practical applications of your camera and lens test rig.

Building and Operating a Test Rig

The test rig consists of a used dollar bill taped on all four sides to the geometric center of a sheet of standard 11x8.5" white copy paper. The copy paper is then centered and taped on four sides to the top edge of an old wooden crate.

Lens height is set by using a sturdy tripod to raise the center axis of the lens to the height of Washington's nose.

Camera positioning, focus distance, and lens alignment are set using several different methods. For example, when the camera needs to be positioned at a 21' focus distance, two strings attached 14" to the left and right of center are employed. Both these strings have markers at 21' 3/8". When these markers are joined and the strings pulled taut, a position is formed for the center axis of the lens at the film plane of the camera.

Another positioning method is to move the tripod back and forth until the vertical sides of the white background paper touch the vertical sides of the camera frame within the viewfinder. A tape measure running from the back of the crate and along the center crossbar is then used to align the center axis of the lens.

Agreed, this isn't the most precise laboratory instrument in the world, but we're not trying to put a man on Mars here. We're just trying to prevent those pesky little $500-$50,000 purchasing blunders.

A Real World Lens Test

The photographs below show a real world lens test between a Nikon 18-200mm lens at 200mm and a Sigma 50-500mm lens also at 200mm. Both images are unaltered 500x166-pixel crops from the original images straight out of the camera. The fact that both images cover about the same surface area of the dollar bill is good confirmation of virtually identical magnification. By using the same Nikon D2X body for both lenses, camera resolution is kept exactly equal.

Thus, with magnification and camera resolution equal, we're assured the differences in detail and sharpness are due strictly to differences in lens resolution.

By closely examining the micro-detail of these two photographs, especially in the lower left corner, one can positively conclude, at 200mm, the Sigma 50-500mm is a significantly sharper lens than the Nikon 18-200mm. This conclusion is supported by extensive results from the field.

The two photographs below show how focus distance and focal length were used to equalize magnification. In the case of the Nikon 18-200mm, focus distance was set by moving the tripod until the width of the white background filled the camera frame. In the case of the Sigma 50-500mm, the same focus distance was used, but the lens was zoomed until the white background filled the width of the camera frame. This procedure ensured virtually equal magnification.

Controlling Incidental Variables

For these tests to be valid, incidental variables such as camera shake, camera positioning, lighting, wind, and focusing accuracy must be carefully controlled.

Here are some of the techniques used to control these incidental variables.

• Lenses cleaned, front and back.
• Camera body securely mounted on a sturdy, locked-down carbon fiber tripod.
• Lens positioned to the height of the center of the dollar bill.
• Lens axis positioned at a 90-degree angle to the center of the dollar bill.
• Test photos taken in strong sunlight at the same time of day.
• Both test rig and and tripod set on a level concrete base.
• Exposure settings kept exactly equal.
• Exposure settings set for fast shutter speed (e.g. 1/2000 second).
• Remote shutter release and mirror lockup used.
• In-camera sharpening turned OFF.
• Continuous, Single Area AF mode used with the center focus area selected.
• A minimum of four frames per exposure, focal length, and focus distance setting taken with refocusing between each frame.
• Again for emphasis, at least four frames, refocusing between each frame.
• Frames carefully examined on the computer to select the sharpest exemplar from each set.

Repeatability

The best way to check and defend the validity of your testing is to repeat the tests. When comparing test results, check for the following problems.

• Position. Equipment materially changes resolution positions.
• Magnitude. The resolution gap between equipment varies widely.
• Field Results. Results from the field fail to substantially match results from target testing.
• Entropy. Increased volume of target testing exacerbates all the above problems.
• Degrees of Freedom. Have you changed cameras or lenses between tests? Was one test shot with cloud cover and the other with sunlight? The only material variable change between tests should be time.

If you chronically suffer from any of these problems, double check your procedures for errors.

Recommendations

Here's a brief list of recommendations that could significantly improve the detail and sharpness of your photographs, and end up saving you more than $10,000. I'm highly confident in these recommendations because I have the test results to back them up. If you have legitimate, repeatable test results that dispute any of my recommendations, please, please email me. If you simply disagree with my recommendations, but have no test results, please, please do not email me.

• Current 35mm full-frame sensors offer substantially less resolution than 10+ megapixel APS-C sensors. Thus, when magnification is increased 1.5X to take advantage of their larger size, 35mm full-frame sensors generate detail and sharpness about equal to a 10+ megapixel APS-C sensor. When magnification is absolutely equal, 10+ megapixel APS-C sensors generate substantially more detail and sharpness. Bottom line, current 35mm full-frame sensors are inefficient and overpriced. Save your money.
• Monster 600mm and 800mm lenses costing $6,000-$9,000 generate little more detail and sharpness than a compact 500mm lens costing $1,000. Again, save your money.
• It's a myth that good-quality teleconverters substantially degrade optical resolution. With Barlow-style lenses, telescope makers have been proving this for decades. Some of my sharpest, most detailed photographs come from stacking Sigma 1.4X and 2X teleconverters on a Sigma 300-800mm telephoto lens.
• A top-performing 10-megapixel digiscope costing $2,700 generates far more detail and sharpness than a conventional dSLR/monster telephoto lens combination costing $15,000. Said digiscope is also far easier to use in the field, and therefore far more productive. Again, save your money.
• Through the magic of RAW-processing, inexpensive superzooms costing a few hundred dollars can generate detail and sharpness equivalent to bigger, premium, professional film lenses costing thousands of dollars. Save the weight, and your money.

Note how all these findings run just about 180-degrees counter to what most magazines, marketers, salespeople, Internet pundits, colleagues, and friends will tell you. For example, most "experts" will tell you the Canon 1Ds Mark II with its high $7,200 price tag, big 35mm full-frame sensor, and huge 16,700,000 pixel count delivers substantially better detail and sharpness than 10-12-megapixel APS-C dSLR cameras.

Science and testing show the opposite is true.

At stake here are thousands and thousand of dollars of your money. Also at stake are weeks, months and even years of your time, and the baseline image quality of your photographs. High stakes, indeed.

Fortunately, testing for detail and sharpness is relatively quick, easy, and inexpensive, and the results are accurate, objective, and repeatable. Considering the stakes, undertaking your own camera and lens testing program will undoubtedly be worth the extra time and effort.

—MTH

Copyright © 1999-2007 NaturePhotoAdvisor.com