  In this paper we consider a family of multiple Hurwitz zeta values with
bi-indices parameterized by $\mu$ with $\Ree(\mu)>0$. These values are equipped
with both the $\mu$-stuffle product from their series definition and the
shuffle product from their integral expressions. We will give a detailed
analysis of the two different products and discuss their regularization when
the bi-indices are non-admissible. Our main goal is to prove the comparison
theorem relating the two ways of regularization, which is an analog of Ihara,
Kaneko, and Zagier's celebrated result concerning the regularization of
multiple zeta values. We will see that the comparison formula is
$\mu$-invariant, that is, it is independent of the parameter $\mu$. As
applications, we will first provide a few interesting closed formulas for some
multi-fold infinite sums that are closely related to the multiple zeta values
of level $N$ studied by Yuan and the second author. We will also derive two sum
formulas for $\mu$-double Hurwitz zeta values which generalize the sum formulas
for double zeta values and double zeta star values, respectively, and a
weighted sum formula which generalizes the weighted sum formulas for both
double zeta values and double $T$-values simultaneously. At the end of the
paper, we will propose a problem extending that for multiple zeta values first
conjectured by Ihara et al.
