  Multiple raters are often needed to be used interchangeably in practice for
measurement or evaluation. Assessing agreement among these multiple raters via
agreement indices are necessary before their participation. While the
intuitively appealing agreement indices such as coverage probability and total
deviation index, and relative area under coverage probability curve, have been
extended for assessing overall agreement among multiple raters, these
extensions have limitations. The existing overall agreement indices either
require normality and homogeneity assumptions or did not preserve the intuitive
interpretation of the indices originally defined for two raters. In this paper,
we propose a new set of overall agreement indices based on maximum pairwise
differences among all raters. The proposed new overall coverage probability,
overall total deviation index and relative area under overall coverage
probability curve retain the original intuitive interpretation from the
pairwise version. Without making any distributional assumption, we also propose
a new unified nonparametric estimation and inference approach for the overall
indices based on generalized estimating equations that can accommodate
replications made by the same rater. Under mild assumptions, the proposed
variance estimator is shown to achieve efficiency bound under independent
working correlation matrix. Simulation studies under different scenarios are
conducted to assess the performance of the proposed estimation and inference
approach with and without replications. We illustrate the methodology by using
a blood pressure data with three raters who made three replications on each
subjects.
