  While the $S_3$ scalar leptoquark presents a possible tree-level explanation
of the $b \to s \ell \ell$ flavour anomalies, it suffers from two conceptual
problems which are often disregarded by model-builders. Firstly, the quantum
numbers of the $S_3$ allow for a renormalisable diquark operator that would
trigger rapid proton decay unless its coupling were tuned away. Secondly, one
expects the leptoquark to have generic couplings to leptons, which require
tuning to avoid stringent experimental bounds on lepton flavour violation. By
gauging a $U(1)$ current that acts as $L_\mu - L_\tau$ on the Standard Model
(SM) fermions, and under which the leptoquark has charge $-1$, one can remedy
both these problems. The additional $U(1)$, which is spontaneously broken at
some high scale, is associated with a massive $Z^\prime$ gauge boson and a
scalar SM singlet $\Phi$, which play no direct role in mediating the anomalous
$B$ meson decays. By computing one- and two-loop mass corrections, we show that
this pair of particles can be hidden away at much higher mass scales without
destabilising either the Higgs or the leptoquark masses. The only low-energy
relic of gauging $L_\mu - L_\tau$ is thus the accidental global symmetry
structure of the lagrangian. On the other hand, we find quite generally that an
$S_3$ leptoquark that mediates the $b \to s \ell \ell$ anomalies cannot be much
heavier than a few TeV without itself inducing large Higgs mass corrections.
