  Recent protocols and metrics for training and evaluating autonomous robot
navigation through crowds are inconsistent due to diversified definitions of
"social behavior". This makes it difficult, if not impossible, to effectively
compare published navigation algorithms. Furthermore, with the lack of a good
evaluation protocol, resulting algorithms may fail to generalize, due to lack
of diversity in training. To address these gaps, this paper facilitates a more
comprehensive evaluation and objective comparison of crowd navigation
algorithms by proposing a consistent set of metrics that accounts for both
efficiency and social conformity, and a systematic protocol comprising multiple
crowd navigation scenarios of varying complexity for evaluation. We tested four
state-of-the-art algorithms under this protocol. Results revealed that some
state-of-the-art algorithms have much challenge in generalizing, and using our
protocol for training, we were able to improve the algorithm's performance. We
demonstrate that the set of proposed metrics provides more insight and
effectively differentiates the performance of these algorithms with respect to
efficiency and social conformity.
