  Inter-rater reliability (IRR), which is a prerequisite of high-quality
ratings and assessments, may be affected by contextual variables such as the
rater's or ratee's gender, major, or experience. Identification of such
heterogeneity sources in IRR is important for implementation of policies with
the potential to decrease measurement error and to increase IRR by focusing on
the most relevant subgroups. In this study, we propose a flexible approach for
assessing IRR in cases of heterogeneity due to covariates by directly modeling
differences in variance components. We use Bayes factors to select the best
performing model, and we suggest using Bayesian model-averaging as an
alternative approach for obtaining IRR and variance component estimates,
allowing us to account for model uncertainty. We use inclusion Bayes factors
considering the whole model space to provide evidence for or against
differences in variance components due to covariates. The proposed method is
compared with other Bayesian and frequentist approaches in a simulation study,
and we demonstrate its superiority in some situations. Finally, we provide real
data examples from grant proposal peer-review, demonstrating the usefulness of
this method and its flexibility in the generalization of more complex designs.
