  Testing Deep Learning (DL) based systems inherently requires large and
representative test sets to evaluate whether DL systems generalise beyond their
training datasets. Diverse Test Input Generators (TIGs) have been proposed to
produce artificial inputs that expose issues of the DL systems by triggering
misbehaviours. Unfortunately, such generated inputs may be invalid, i.e., not
recognisable as part of the input domain, thus providing an unreliable quality
assessment. Automated validators can ease the burden of manually checking the
validity of inputs for human testers, although input validity is a concept
difficult to formalise and, thus, automate.
  In this paper, we investigate to what extent TIGs can generate valid inputs,
according to both automated and human validators. We conduct a large empirical
study, involving 2 different automated validators, 220 human assessors, 5
different TIGs and 3 classification tasks. Our results show that 84%
artificially generated inputs are valid, according to automated validators, but
their expected label is not always preserved. Automated validators reach a good
consensus with humans (78% accuracy), but still have limitations when dealing
with feature-rich datasets.
