Tuesday, July 30, 2013

Samsung Caught Fudging Benchmarks with International Versions of Galaxy S4 (Updated)

This one is quickly making the tech rounds; Samsung has allegedly been caught by the tech community (through the work of AnandTech) of manipulating benchmark scores within the international version of their Galaxy S4 phones. The situation is not dissimilar to how the graphic card companies ATi and nVidia used to program their cards to perform better for the most popularly used synthetic benchmarks published in gaming magazines; the designers add extra code to detect when a benchmark program is being run, and then the processor speed is boosted in turn.

While not unsurprising, it certainly is disappointing. One gets the feeling that Samsung's (dis)virtue in this case is that they are the first of the smartphone makers to get caught; it would be unreasonable to expect that they were the only ones. The question now is how widely used is this practice; does it extend to their other phones and tablets as well?

If there is one issue that isn't in jeopardy, it's that no-one could accuse the S4 of being an underpowered phone; even if you aren't getting the stated amount of performance, it's still more performance than what people can practically use in a smartphone. While multimedia demands are increasing year-by-year, you don't need all of the S4's power to run the most popular imaging apps like Instagram or Vine. All modern phones play and capture HD video seamlessly; still photo-wise, the excess of CPU horsepower is even more evident. Aside from the obvious ethical implications of rigging benchmark tests, the bigger problem is that the smartphone market is well on its way towards commoditization; as the products and user experience become increasingly similar, the companies are going to struggle to differentiate their phones from the competitors, let alone last year's models.

As an aside, if you think this sort of thing doesn't happen in the camera world, it sort of does and doesn't. ISO ratings have been notoriously inconsistent throughout the history of the digital camera; if you go back through the years, you'll see on the various tests that (as an example) Canon's and Nikon's often gave different exposure values for the same selected ISO. If you look at some of the high-end small sensor compacts of the past three years, you'll find that their ISO sensitivity ratings seem overstated compared to what you would find with DSLR's. The reason for all of this is because, for technical reasons, the definition of ISO sensitivity in digital terms has a bit of leeway; and while most cameras fall within the norm, some have pushed the definition in order to publish a more favourable spec number.


Samsung has issued a response here. In their words, the CPU and GPU speeds run at the maximum when heat is not an issue, but are limited for gaming apps which would overheat the phone with extended use. This is actually a reasonable explanation, however, as many have already pointed out, it does not explain why the S4 is hardcoded to perform at the higher speeds for a specific set of synthetic benchmarks.

As the tech community digs around, we'll no doubt find more examples of this, but Samsung has already been stung by being the first to be out-ed. The problem remains, how to differentiate the phone in an age of increasing iteration. Because the multiple cores of the Exynos 5 Octa processor was a big talking point at the launch of the S4, it would have been a disappointment (especially for the marketing department) if the benchmark numbers weren't appreciably higher.

No comments:

Post a Comment