  Deep heteroscedastic regression involves jointly optimizing the mean and
covariance of the predicted distribution using the negative log-likelihood.
However, recent works show that this may result in sub-optimal convergence due
to the challenges associated with covariance estimation. While the literature
addresses this by proposing alternate formulations to mitigate the impact of
the predicted covariance, we focus on improving the predicted covariance
itself. We study two questions: (1) Does the predicted covariance truly capture
the randomness of the predicted mean? (2) In the absence of supervision, how
can we quantify the accuracy of covariance estimation? We address (1) with a
Taylor Induced Covariance (TIC), which captures the randomness of the predicted
mean by incorporating its gradient and curvature through the second order
Taylor polynomial. Furthermore, we tackle (2) by introducing a Task Agnostic
Correlations (TAC) metric, which combines the notion of correlations and
absolute error to evaluate the covariance. We evaluate TIC-TAC across multiple
experiments spanning synthetic and real-world datasets. Our results show that
not only does TIC accurately learn the covariance, it additionally facilitates
an improved convergence of the negative log-likelihood. Our code is available
at https://github.com/vita-epfl/TIC-TAC
