  Motion prediction of vehicles is critical but challenging due to the
uncertainties in complex environments and the limited visibility caused by
occlusions and limited sensor ranges. In this paper, we study a new task,
safety-aware motion prediction with unseen vehicles for autonomous driving.
Unlike the existing trajectory prediction task for seen vehicles, we aim at
predicting an occupancy map that indicates the earliest time when each location
can be occupied by either seen and unseen vehicles. The ability to predict
unseen vehicles is critical for safety in autonomous driving. To tackle this
challenging task, we propose a safety-aware deep learning model with three new
loss functions to predict the earliest occupancy map. Experiments on the
large-scale autonomous driving nuScenes dataset show that our proposed model
significantly outperforms the state-of-the-art baselines on the safety-aware
motion prediction task. To the best of our knowledge, our approach is the first
one that can predict the existence of unseen vehicles in most cases. Project
page at {\url{https://github.com/xrenaa/Safety-Aware-Motion-Prediction}}.
