  The COVID-19 pandemic has resulted in a tremendous need for access to the
latest scientific information, primarily through the use of text mining and
search tools. This has led to both corpora for biomedical articles related to
COVID-19 (such as the CORD-19 corpus (Wang et al., 2020)) as well as search
engines to query such data. While most research in search engines is performed
in the academic field of information retrieval (IR), most academic search
engines$\unicode{x2013}$though rigorously evaluated$\unicode{x2013}$are
sparsely utilized, while major commercial web search engines (e.g., Google,
Bing) dominate. This relates to COVID-19 because it can be expected that
commercial search engines deployed for the pandemic will gain much higher
traction than those produced in academic labs, and thus leads to questions
about the empirical performance of these search tools. This paper seeks to
empirically evaluate two such commercial search engines for COVID-19, produced
by Google and Amazon, in comparison to the more academic prototypes evaluated
in the context of the TREC-COVID track (Roberts et al., 2020). We performed
several steps to reduce bias in the available manual judgments in order to
ensure a fair comparison of the two systems with those submitted to TREC-COVID.
We find that the top-performing system from TREC-COVID on bpref metric
performed the best among the different systems evaluated in this study on all
the metrics. This has implications for developing biomedical retrieval systems
for future health crises as well as trust in popular health search engines.
