  In this work, we explore sources of inter-annotator disagreement in narrative
analysis, in light of the question of whether or not a narrative plot exists in
the text. For this purpose, we present a method for a conceptual decomposition
of an existing annotation into two separate levels: (1) \textbf{whether} or not
a narrative plot exists in the text, and (2) \textbf{which} plot elements exist
in the text. We apply this method to an existing dataset of sentences annotated
with three different narrative plot elements: \textit{Complication},
\textit{Resolution} and \textit{Success}. We then employ statistical analysis
in order to quantify how much of the inter-annotator disagreement can be
explained by each of the two levels. We further perform a qualitative analysis
of disagreement cases in each level, observing several sources of disagreement,
such as text ambiguity, scheme definition and personal differences between the
annotators. The insights gathered on the dataset may serve to reduce
inter-annotator disagreement in future annotation endeavors. We conclude with a
broader discussion on the potential implications of our approach in studying
and evaluating inter-annotator disagreement in other settings.
