  A key challenge on the path to developing agents that learn complex
human-like behavior is the need to quickly and accurately quantify
human-likeness. While human assessments of such behavior can be highly
accurate, speed and scalability are limited. We address these limitations
through a novel automated Navigation Turing Test (ANTT) that learns to predict
human judgments of human-likeness. We demonstrate the effectiveness of our
automated NTT on a navigation task in a complex 3D environment. We investigate
six classification models to shed light on the types of architectures best
suited to this task, and validate them against data collected through a human
NTT. Our best models achieve high accuracy when distinguishing true human and
agent behavior. At the same time, we show that predicting finer-grained human
assessment of agents' progress towards human-like behavior remains unsolved.
Our work takes an important step towards agents that more effectively learn
complex human-like behavior.
