  Test automation is important in software industry but self-assessment
instruments for assessing its maturity are not sufficient. The two objectives
of this study are to synthesize what an organization should focus to assess its
test automation; develop a self-assessment instrument (a survey) for assessing
test automation maturity and scientifically evaluate it. We carried out the
study in four stages. First, a literature review of 25 sources was conducted.
Second, the initial instrument was developed. Third, seven experts from five
companies evaluated the initial instrument. Content Validity Index and
Cognitive Interview methods were used. Fourth, we revised the developed
instrument. Our contributions are as follows: (a) we collected practices mapped
into 15 KAs that indicate where an organization should focus to assess its test
automation; (b) we developed and evaluated a self-assessment instrument for
assessing test automation maturity; (c) we discuss important topics such as
response bias that threatens self-assessment instruments. Our results help
companies and researchers to understand and improve test automation practices
and processes.
