  Automated graphical user interface (GUI) tests can reduce manual testing
activities and increase test frequency. This motivates the conversion of manual
test cases into automated GUI tests. However, it is not clear whether such
automation is cost-effective given that GUI automation scripts add to the code
base and demand maintenance as a system evolves. In this paper, we introduce a
method for estimating maintenance cost and Return on Investment (ROI) for
Automated GUI Testing (AGT). The method utilizes the existing source code
change history and can be used for evaluation also of other testing or quality
assurance automation technologies. We evaluate the method for a real-world,
industrial software system and compare two fundamentally different AGT tools,
namely Selenium and EyeAutomate, to estimate and compare their ROI. We also
report on their defect-finding capabilities and usability. The quantitative
data is complemented by interviews with employees at the case company. The
method was successfully applied and estimated maintenance cost and ROI for both
tools are reported. Overall, the study supports earlier results showing that
implementation time is the leading cost for introducing AGT. The findings
further suggest that while EyeAutomate tests are significantly faster to
implement, Selenium tests require more of a programming background but less
maintenance.
