  The Solovay-Kitaev theorem states that universal quantum gate sets can be
exchanged with low overhead. More specifically, any gate on a fixed number of
qudits can be simulated with error $\epsilon$ using merely
$\mathrm{polylog}(1/\epsilon)$ gates from any finite universal quantum gate set
$\mathcal{G}$. One drawback to the theorem is that it requires the gate set
$\mathcal{G}$ to be closed under inversion. Here we show that this restriction
can be traded for the assumption that $\mathcal{G}$ contains an irreducible
representation of any finite group $G$. This extends recent work of
Sardharwalla et al. [arXiv:1602.07963], and applies also to gates from the
special linear group. Our work can be seen as partial progress towards the
long-standing open problem of proving an inverse-free Solovay-Kitaev theorem
[arXiv:quant-ph/0505030, arXiv:0908.0512].
