  Maximum likelihood estimation is a popular method in statistical inference.
As a way of assessing the accuracy of the maximum likelihood estimate (MLE),
the calculation of the covariance matrix of the MLE is of great interest in
practice. Standard statistical theory shows that the normalized MLE is
asymptotically normally distributed with covariance matrix being the inverse of
the Fisher information matrix (FIM) at the unknown parameter. Two commonly used
estimates for the covariance of the MLE are the inverse of the observed FIM
(the same as the inverse Hessian of the negative log-likelihood) and the
inverse of the expected FIM (the same as the inverse FIM). Both of the observed
and expected FIM are evaluated at the MLE from the sample data. In this
dissertation, we demonstrate that, under reasonable conditions similar to
standard MLE conditions, the inverse expected FIM outperforms the inverse
observed FIM under a mean squared error criterion. Specifically, in an
asymptotic sense, the inverse expected FIM (evaluated at the MLE) has no
greater mean squared error with respect to the true covariance matrix than the
inverse observed FIM (evaluated at the MLE) at the element level. This result
is different from widely accepted results showing preference for the observed
FIM. In this dissertation, we present theoretical derivations that lead to the
conclusion above. We also present numerical studies on three distinct problems
to support the theoretical result. This dissertation also includes two
appendices on topics of relevance to stochastic systems. The first appendix
discusses optimal perturbation distributions for the simultaneous perturbation
stochastic approximation (SPSA) algorithm. The second appendix considers Monte
Carlo methods for computing FIMs when closed forms are not attainable.
