  In autonomous navigation, a planning system reasons about other agents to
plan a safe and plausible trajectory. Before planning starts, agents are
typically processed with computationally intensive models for recognition,
tracking, motion estimation and prediction. With limited computational
resources and a large number of agents to process in real time, it becomes
important to efficiently rank agents according to their impact on the decision
making process. This allows spending more time processing the most important
agents. We propose a system to rank agents around an autonomous vehicle (AV) in
real time. We automatically generate a ranking data set by running the planner
in simulation on real-world logged data, where we can afford to run more
accurate and expensive models on all the agents. The causes of various planner
actions are logged and used for assigning ground truth importance scores. The
generated data set can be used to learn ranking models. In particular, we show
the utility of combining learned features, via a convolutional neural network,
with engineered features designed to capture domain knowledge. We show the
benefits of various design choices experimentally. When tested on real AVs, our
system demonstrates the capability of understanding complex driving situations.
