  A multi-reader multi-case (MRMC) analysis is applied to account for both
reader and case variability when evaluating the clinical performance of a
medical imaging device or reader performance under different reading
modalities. For a clinical task that measures a quantitative biomarker an
agreement analysis, such as limits of agreement (LOA), can be used. In this
work, we decompose the total variation in the data using a three-way mixed
effect ANOVA model to estimate the MRMC variance of individual differences and
the LOA between different reading modalities. There are rules for writing down
the expectation of the mean squares in terms of the variance components for
fully-crossed data, i.e. data where all the readers read all the cases in all
modalities being studied. Sometimes the annotation task is labor-intensive and
time-consuming or distributed across sites, so that a fully-crossed study is
not practical. In this work, we focus on estimating the MRMC variance in the
within- and between-readers and within- and between-modalities LOA for an
arbitrary study design. Simulation studies were conducted to validate the LOA
variance estimates. The method was also applied to a dataset to compare
pathologist performance for assessing the density of stromal tumor infiltrating
lymphocytes on different platforms.
