  Epidemiologic and medical studies often rely on evaluators to obtain
measurements of exposures or outcomes for study participants, and valid
estimates of associations depends on the quality of data. Even though
statistical methods have been proposed to adjust for measurement errors, they
often rely on unverifiable assumptions and could lead to biased estimates if
those assumptions are violated. Therefore, methods for detecting potential
`outlier' evaluators are needed to improve data quality during data collection
stage. In this paper, we propose a two-stage algorithm to detect `outlier'
evaluators whose evaluation results tend to be higher or lower than their
counterparts. In the first stage, evaluators' effects are obtained by fitting a
regression model. In the second stage, hypothesis tests are performed to detect
`outlier' evaluators, where we consider both the power of each hypothesis test
and the false discovery rate (FDR) among all tests. We conduct an extensive
simulation study to evaluate the proposed method, and illustrate the method by
detecting potential `outlier' audiologists in the data collection stage for the
Audiology Assessment Arm of the Conservation of Hearing Study, an epidemiologic
study for examining risk factors of hearing loss in the Nurses' Health Study
II. Our simulation study shows that our method not only can detect true
`outlier' evaluators, but also is less likely to falsely reject true `normal'
evaluators. Our two-stage `outlier' detection algorithm is a flexible approach
that can effectively detect `outlier' evaluators, and thus data quality can be
improved during data collection stage.
