AnTuTu AI Benchmark Tool Now Available For Android

Artificial intelligence, next to 5G and cameras, has become one of the pillars of modern smartphones. Unlike those, however, AI hasn't exactly been easy to measure given the many variables, tools, and implementations. Of course, that could almost be said of smartphone benchmarks today anyway, so it's not too surprising that AnTuTu, one of the household names in mobile benchmarks, has finally come up with a tool to do exactly that. At least for Android phones, that is.

Considering how new AI is on smartphones, it's not really surprising that each OEM and chip maker have gone off and done their own thing. Some use dedicated and separate Neural Processing Units (NPUs) while others like Qualcomm just repurpose the DSP for AI use. As there is no standard for mobile AI, AnTuTu was left to come up with its own tool for reference.

AnTuTu's AI benchmark tests for two things, speed, and accuracy. Of the two, however, accuracy is given more weight to the point that favoring speed actually knocks off a few points from the overall score. Two tests are used to benchmark a phone's AI prowess, a 200-image classification test and a 600-frame video object recognition test. These tests use widely available and platform-agnostic neural networks (Inception v3 and MobileNet SSD, respectively) before translating to the neural network provided by the chip maker's SDK.

There are some caveats to the way the tool works, however. For one, not all chip makers actually provide an AI SDK, with Samsung being the biggest culprit. For those instances, AnTuTu falls back to Google's TensorFlow Lite, which it noted to be unsatisfactory and unreliable. Different Android versions can also affect the results since Google has been refining the platform's AI support over the years.

AI benchmarking it even trickier than regular mobile benchmarking because of the many moving parts and variables involved. And while AnTuTu may be the first few to make such a tool available to end users, others may soon follow with their own standards and rules. It might be a while before we can start citing AI benchmarks, and even then, like any mobile benchmark, they'll be useful only as a jumping board for discussion.