  We tackle the problem of estimating risk measures of the infinite-horizon
discounted cost within a Markov cost process. The risk measures we study
include variance, Value-at-Risk (VaR), and Conditional Value-at-Risk (CVaR).
First, we show that estimating any of these risk measures with
$\epsilon$-accuracy, either in expected or high-probability sense, requires at
least $\Omega(1/\epsilon^2)$ samples. Then, using a truncation scheme, we
derive an upper bound for the CVaR and variance estimation. This bound matches
our lower bound up to logarithmic factors. Finally, we discuss an extension of
our estimation scheme that covers more general risk measures satisfying a
certain continuity criterion, e.g., spectral risk measures, utility-based
shortfall risk. To the best of our knowledge, our work is the first to provide
lower and upper bounds for estimating any risk measure beyond the mean within a
Markovian setting. Our lower bounds also extend to the infinite-horizon
discounted costs' mean. Even in that case, our lower bound of
$\Omega(1/\epsilon^2) $ improves upon the existing $\Omega(1/\epsilon)$ bound
[13].
