  Existing temporal relation (TempRel) annotation schemes often have low
inter-annotator agreements (IAA) even between experts, suggesting that the
current annotation task needs a better definition. This paper proposes a new
multi-axis modeling to better capture the temporal structure of events. In
addition, we identify that event end-points are a major source of confusion in
annotation, so we also propose to annotate TempRels based on start-points only.
A pilot expert annotation using the proposed scheme shows significant
improvement in IAA from the conventional 60's to 80's (Cohen's Kappa). This
better-defined annotation scheme further enables the use of crowdsourcing to
alleviate the labor intensity for each annotator. We hope that this work can
foster more interesting studies towards event understanding.
