  Measurement of the interrater agreement (IRA) is critical in various
disciplines. To correct for potential confounding chance agreement in IRA,
Cohen's kappa and many other methods have been proposed. However, owing to the
varied strategies and assumptions across these methods, there is a lack of
practical guidelines on how these methods should be preferred even for the
common two-rater dichotomous rating. To fill the gaps in the literature, we
systematically review nine IRA methods and propose a generalized framework that
can simulate the correlated decision processes behind the two raters to compare
those reviewed methods under comprehensive practical scenarios. Based on the
new framework, an estimand of "true" chance-corrected IRA is defined by
accounting for the "probabilistic certainty" and serves as the comparison
benchmark. We carry out extensive simulations to evaluate the performance of
the reviewed IRA measures, and an agglomerative hierarchical clustering
analysis is conducted to assess the inter-relationships among the included
methods and the benchmark metric. Recommendations for selecting appropriate IRA
statistics in different practical conditions are provided and the needs for
further advancements in IRA estimation methodologies are emphasized.
