  Recurrent neural networks (RNNs) have been applied to a broad range of
applications, including natural language processing, drug discovery, and video
recognition. Their vulnerability to input perturbation is also known. Aligning
with a view from software defect detection, this paper aims to develop a
coverage guided testing approach to systematically exploit the internal
behaviour of RNNs, with the expectation that such testing can detect defects
with high possibility. Technically, the long short term memory network (LSTM),
a major class of RNNs, is thoroughly studied. A family of three test metrics
are designed to quantify not only the values but also the temporal relations
(including both step-wise and bounded-length) exhibited when LSTM processing
inputs. A genetic algorithm is applied to efficiently generate test cases. The
test metrics and test case generation algorithm are implemented into a tool
TestRNN, which is then evaluated on a set of LSTM benchmarks. Experiments
confirm that TestRNN has advantages over the state-of-art tool DeepStellar and
attack-based defect detection methods, owing to its working with finer temporal
semantics and the consideration of the naturalness of input perturbation.
Furthermore, TestRNN enables meaningful information to be collected and
exhibited for users to understand the testing results, which is an important
step towards interpretable neural network testing.
