Ke Mao and colleagues at University College London write about their research on “Robotic Testing of Mobile Apps” . As they say, testing apps on mobile devices is a very difficult challenge. Contemporary handhelds have so many possible inputs, they are used by everyone in all kinds of situations, and apps run in a complex environment potentially interacting with other apps, networks, and services. Throw in security, privacy, and operator safety. And so on.
It’s almost as if those hardware guys were trying to create a system that can’t possibly be programmed! 🙂
Ono of the key problems is that testing software on a handheld device requires some way to test, well, holding in hands. Much testing must be (literally) manual, and that’s difficult, slow, and expensive. Testers augment manual testing by capturing human tests and replaying them, possibly with variations. These are simulated interactions, which do not necessarily use the device realistically.
Mao et al. argue that the way to go is to develop robots that simulate human hands and motions. This may not be perfect, but, as they say, “the robotic gestures will at least be physical gestures”. Ideally, robots may work faster, more consistently, and cheaper than human testers, allowing more coverage.
“a robotic-test generator for mobile apps. Here, we compare our approach with simulation-based test automation, describe scenarios in which robotic testing is beneficial (or even essential)”
The robot also has the advantage that it can be directly linked to automated test generation, pushing test scenarios into the robot’s code to be executed. The researchers use manual tests and also have a model to extrapolate new test cases that are “realistic”.
Their current implementation is actually limited to finger taps. That’s probably useful, but surely we want two whole hands, to hold, turn, shake, and drop the device.
I imagine that this idea will be widely used for both testing and implementing simulated users (i.e., ‘bots’ in the software sense, run by actual physical robots!).
However, there is still a need for as much human testing as possible, particularly field testing. Personal devices must operate in the real world, which is full of unexpected and unforeseeable surprises, not least of them being those darn users, who do the daftest things.
Robot hands are a great addition to the testing arsenal, but scarcely replace existing methods. More testing is always better than less testing. Better testing is always better than poorer testing. But test software as much as possible in as many ways as possible.
- Ke Mao., Mark Harman, and Yue Jia, Robotic Testing of Mobile Apps for Truly Black-Box Automation. IEEE Software, 34 (2):11-16, 2017. http://ieeexplore.ieee.org/document/7888396/media