  Documents in the health domain are often annotated with semantic concepts
(i.e., terms) from controlled vocabularies. As the volume of these documents
gets large, the annotation work is increasingly done by algorithms. Compared to
humans, automatic indexing algorithms are imperfect and may assign wrong terms
to documents, which affect subsequent search tasks where queries contain these
terms. In this work, we aim to understand the performance impact of using
imperfectly assigned terms in Boolean semantic searches. We used MeSH terms and
biomedical literature search as a case study. We implemented multiple automatic
indexing algorithms on real-world Boolean queries that consist of MeSH terms,
and found that (1) probabilistic logic can handle inaccurately assigned terms
better than traditional Boolean logic, (2) query-level performance is mostly
limited by lowest-performing terms in a query, and (3) mixing a small amount of
human indexing with automatic indexing can regain excellent query-level
performance. These findings provide important implications for future work on
automatic indexing.
