  Background: Academic search engines (i.e., digital libraries and indexers)
play an increasingly important role in systematic reviews however these engines
do not seem to effectively support such reviews, e.g., researchers confront
usability issues with the engines when conducting their searches. Aims: To
investigate whether the usability issues are bugs (i.e., faults in the search
engines) or constraints, and to provide recommendations to search-engine
providers and researchers on how to tackle these issues. Method: Using
snowball-sampling from tertiary studies, we identify a set of 621 secondary
studies in software engineering. By physically re-attempting the searches for
all of these 621 studies, we effectively conduct regression testing for 42
search engines. Results: We identify 13 bugs for eight engines, and also
identify other constraints. We provide recommendations for tackling these
issues. Conclusions: There is still a considerable gap between the search-needs
of researchers and the usability of academic search engines. It is not clear
whether search-engine developers are aware of this gap. Also, the evaluation,
by academics, of academic search engines has not kept pace with the
development, by search-engine providers, of those search engines. Thus, the gap
between evaluation and development makes it harder to properly understand the
gap between the search-needs of researchers and search-features of the search
engines.
