  We compute the term $\sim g^3$ in the free energy for a $SU(N)$ gauge theory
with nonzero holonomy at nonzero temperature. If the holonomy is generated
kinematically by the introduction of gauge invariant sources coupled to
Polyakov loops, the contribution of charged (off-diagonal) gluons to the free
energy at order $g^3$, ${\cal F}^{\left( 3\rm{:c.g.} \right)}$, is singular:
${\cal F}^{\left( 3\rm{:c.g.} \right)} \neq 0 $ without holonomy, but ${\cal
F}^{\left( 3 \rm{:c.g.} \right)}= 0$ when the holonomy is nonzero, even
infinitesimally. We show that the absence of the charged gluon contribution is
required by gauge invariance alone and is therefore a universal feature.
