  Software testing is sometimes plagued with intermittently failing tests and
finding the root causes of such failing tests is often difficult. This problem
has been widely studied at the unit testing level for open source software, but
there has been far less investigation at the system test level, particularly
the testing of industrial embedded systems. This paper describes our
investigation of the root causes of intermittently failing tests in the
embedded systems domain, with the goal of better understanding, explaining and
categorizing the underlying faults. The subject of our investigation is a
currently-running industrial embedded system, along with the system level
testing that was performed. We devised and used a novel metric for classifying
test cases as intermittent. From more than a half million test verdicts, we
identified intermittently and consistently failing tests, and identified their
root causes using multiple sources. We found that about 1-3% of all test cases
were intermittently failing. From analysis of the case study results and
related work, we identified nine factors associated with test case
intermittence. We found that a fix for a consistently failing test typically
removed a larger number of failures detected by other tests than a fix for an
intermittent test. We also found that more effort was usually needed to
identify fixes for intermittent tests than for consistent tests. An overlap
between root causes leading to intermittent and consistent tests was
identified. Many root causes of intermittence are the same in industrial
embedded systems and open source software. However, when comparing unit testing
to system level testing, especially for embedded systems, we observed that the
test environment itself is often the cause of intermittence.
