  Stochastic approximation Markov Chain Monte Carlo (SAMCMC) algorithms are a
class of online algorithms having wide-ranging applications, particularly
within Markovian systems. In this work, we study the optimization of
steady-state queueing systems via the general perspective of SAMCMC. Under a
practical and verifiable assumption framework motivated by queueing systems, we
establish a key characteristic of SAMCMC, namely the Lipschitz continuity of
the solution of Poisson equations. This property helps us derive a finite-step
convergence rate and a regret bound for SAMCMC's final output. Leveraging this
rate, we lay the foundation for a functional central limit theory (FCLT)
pertaining to the partial-sum process of the SAMCMC trajectory. This FCLT, in
turn, inspires an online inference approach designed to furnish consistent
confidence intervals for the true solution and to quantify the uncertainties
surrounding SAMCMC outputs. To validate our methodologies, we conduct extensive
numerical experiments on the efficient application of SAMCMC and its inference
techniques in the optimization of GI/GI/1 queues in steady-state conditions.
