  We investigate the online overlapping batch-means covariance estimator for
Stochastic Gradient Descent (SGD) under Markovian sampling. Convergence rates
of order $O\big(\sqrt{d}\,n^{-1/8}(\log n)^{1/4}\big)$ and
$O\big(\sqrt{d}\,n^{-1/8}\big)$ are established under state-dependent and
state-independent Markovian sampling, respectively, where $d$ is the
dimensionality and $n$ denotes observations or SGD iterations. These rates
match the best-known convergence rate for independent and identically
distributed (i.i.d) data. Our analysis overcomes significant challenges that
arise due to Markovian sampling, leading to the introduction of additional
error terms and complex dependencies between the blocks of the batch-means
covariance estimator. Moreover, we establish the convergence rate for the first
four moments of the $\ell_2$ norm of the error of SGD dynamics under
state-dependent Markovian data, which holds potential interest as an
independent result. Numerical illustrations provide confidence intervals for
SGD in linear and logistic regression models under Markovian sampling.
Additionally, our method is applied to the strategic classification with
logistic regression, where adversaries adaptively modify features during
training to affect target class classification.
