  Enabling socially acceptable behavior for situated agents is a major goal of
recent robotics research. Robots should not only operate safely around humans,
but also abide by complex social norms. A key challenge for developing
socially-compliant policies is measuring the quality of their behavior. Social
behavior is enormously complex, making it difficult to create reliable metrics
to gauge the performance of algorithms. In this paper, we propose a protocol
for social navigation benchmarking that defines a set of canonical social
navigation scenarios and an in-situ metric for evaluating performance on these
scenarios using questionnaires. Our experiments show this protocol is
realistic, scalable, and repeatable across runs and physical spaces. Our
protocol can be replicated verbatim or it can be used to define a social
navigation benchmark for novel scenarios. Our goal is to introduce a protocol
for benchmarking social scenarios that is homogeneous and comparable.
