  We tackle the problem of predicting saliency maps for videos of dynamic
scenes. We note that the accuracy of the maps reconstructed from the gaze data
of a fixed number of observers varies with the frame, as it depends on the
content of the scene. This issue is particularly pressing when a limited number
of observers are available. In such cases, directly minimizing the discrepancy
between the predicted and measured saliency maps, as traditional deep-learning
methods do, results in overfitting to the noisy data. We propose a noise-aware
training (NAT) paradigm that quantifies and accounts for the uncertainty
arising from frame-specific gaze data inaccuracy. We show that NAT is
especially advantageous when limited training data is available, with
experiments across different models, loss functions, and datasets. We also
introduce a video game-based saliency dataset, with rich temporal semantics,
and multiple gaze attractors per frame. The dataset and source code are
available at https://github.com/NVlabs/NAT-saliency.
