Before Mario Queiroz, Google’s VP of Product Management, stepped on stage to announce the Pixel 2 earlier this month, a video presented attendees with the question: “What if smartphones got smarter and simpler?” As he talked about the company’s flagship phone, it became clear the question didn't refer to the phone’s blazing fast processor, its crisp, high-resolution display, or the water-resistant design—all good smartphones today have all that. So instead, Queiroz and other Google execs spent much of their stage time talking about how they improved the Pixel 2’s camera.
While phones like the iPhone 8 Plus, iPhone X, and Galaxy Note 8 pack dual-lens setups to do all sorts of funky stuff like depth-sensing and optical zoom, Google’s new phone uses machine learning to make pictures look great. To prove that the Pixel 2's camera was the best on the market, Google turned to DxOMark, the industry standard for camera ratings, which gave the phone a 98—the highest of any smartphone to date.
That’s not just a win for Google. Those bragging rights made clear that DxOMark had grown from a small review website into a major influencer. Now that cameras have become the key differentiators among phones, DxOMark has grown into a power broker for the smartphone industry.
On the Mark
DxOMark launched in 2009 to assess cameras like the Nikon D3X, Panasonic Lumix G1, and the Hasselblad H3DII-39. DxOMark's parent company, DxOMark Image Labs, creates photo processing software, so these kind of camera reviews fed back into its core business. At the time, the company targeted professionals and photography enthusiasts, aiming to inform their purchasing decisions and make sure they grabbed the best camera to suit their needs.
Within a few years, though, camera phones were ubiquitous and everyone was a photographer. Nicolas Touchard, DxOMark’s vice president of marketing, says the company shifted their focus to reach anyone looking for a good smartphone camera who wanted to understand how those cameras work. This started in 2012, when the site published its first smartphone review of the then-two-year-old iPhone 4. It scored a 50.
This September, the site made another shift when it adapted its review process to better suit today’s smartphone cameras. In 2012, most smartphones didn't have dual-lens setups or optical zoom; some were designed for use in low-light, but "not was much as they are today," says Touchard.
It wasn’t just the new features smartphone cameras were shipping. It was the new ways people were using them. They became a way to capture family gatherings, a child's first steps, vacation adventures, and even shoot feature-length movies. DxOMark wanted to give readers as much information as possible to buy the best smartphone camera they could—not just for professional photographers, but for everyone with a high-quality camera in their pocket.
So, how does DxOMark create its ratings? The company conducts a series of tests in a controlled studio environment with pro-level lighting rigs, sensors, and some other specialized gear. They also take the phone out in the real world to test how the cameras fair in the places you’ll be using them the most, and to see how they perform in different lighting.
For traditional cameras like DSLRs, DxOMark shoots images in raw and analyzes them solely based on the that camera produces—no tweaking allowed. On smartphones, though, this isn’t really possible. Smartphone camera sensors are small, so they can’t capture as much light as something like a mirrorless camera or a DSLR. So most smartphones use software to compensate for the shortcomings of their smaller lenses, automatically adjusting the exposure, saturation, and other settings and make the final picture look better. (This has the added benefit of improving phone cameras over time with software updates, without having to upgrade the hardware.) DxOMark factors this into their smartphone reviews by assessing the final image, not just what the sensor sees.
Once it's fiddled around with the camera, DxOMark dives deep into its assessment of the camera’s strengths, weaknesses, and overall performance. For phone scores, it considers exposure and contrast, dynamic range, color (hue, saturation, white balance), texture and noise, autofocus, artifacts (aliasing, distortion, flare), flash, zoom, and bokeh. These categories also get a score of their own, so even if a phone has the best camera overall, you can see if it falls behind in flash or white balance.
DxOMark posts sample images, write-ups of their findings, and fancy charts after completing the testing process. In some cases, the site even adds shots from competing devices for comparison.
“I think we really bring something that didn't exist before in terms of scoring and scaling products,” Touchard says. “It's 1,500 pictures taken, with several hours of video and processing.”
At the end of it all, DxOMark gives the camera a final score, ranging from the low 50s up to 100. Touchard says the scoring system was built to be open, and the limits will change as cameras improve. Rather than look at the numbers as a goal to achieve, he says it should be viewed as a map to see where cameras in a particular realm sit compared to the competition.
Still, a single score can’t capture everything about a camera’s abilities. Austin Mann, a travel photographer known for his yearly iPhone camera reviews, says there's a big gap between a camera's spec sheet and the actual experience of using it. "You can't really appreciate that gap if you're just looking at technical performance," he says. "If it crashes, it doesn't matter if it got 1,000 on DxOMark, it didn't get the shot."
That’s why, Touchard says, the full review and subscores are there. DxOMark provides hard data, as well as in-depth reviews of using each phone's camera.
“I would love to have ways to ensure people go into details, but we really can't control that—that's the way it is on the internet,” Touchard says. “I think our scoring, with its architecture, is a pretty good compromise, and it has the merit of existing.”
