  Reliable uncertainty from deterministic single-forward pass models is sought
after because conventional methods of uncertainty quantification are
computationally expensive. We take two complex single-forward-pass uncertainty
approaches, DUQ and SNGP, and examine whether they mainly rely on a
well-regularized feature space. Crucially, without using their more complex
methods for estimating uncertainty, a single softmax neural net with such a
feature-space, achieved via residual connections and spectral normalization,
*outperforms* DUQ and SNGP's epistemic uncertainty predictions using simple
Gaussian Discriminant Analysis *post-training* as a separate feature-space
density estimator -- without fine-tuning on OoD data, feature ensembling, or
input pre-procressing. This conceptually simple *Deep Deterministic Uncertainty
(DDU)* baseline can also be used to disentangle aleatoric and epistemic
uncertainty and performs as well as Deep Ensembles, the state-of-the art for
uncertainty prediction, on several OoD benchmarks (CIFAR-10/100 vs
SVHN/Tiny-ImageNet, ImageNet vs ImageNet-O) as well as in active learning
settings across different model architectures, yet is *computationally
cheaper*.
