  Hand-annotated data can vary due to factors such as subjective differences,
intra-rater variability, and differing annotator expertise. We study
annotations from different experts who labelled the same behavior classes on a
set of animal behavior videos, and observe a variation in annotation styles. We
propose a new method using program synthesis to help interpret annotation
differences for behavior analysis. Our model selects relevant trajectory
features and learns a temporal filter as part of a program, which corresponds
to estimated importance an annotator places on that feature at each timestamp.
Our experiments on a dataset from behavioral neuroscience demonstrate that
compared to baseline approaches, our method is more accurate at capturing
annotator labels and learns interpretable temporal filters. We believe that our
method can lead to greater reproducibility of behavior annotations used in
scientific studies. We plan to release our code.
