  We derive constraints on two-dimensional conformal field theories with higher
spin symmetry due to unitarity, modular invariance, and causality. We focus on
CFTs with $\mathcal{W}_N$ symmetry in the "irrational" regime, where $c>N-1$
and the theories have an infinite number of higher-spin primaries. The most
powerful constraints come from positivity of the Kac matrix, which (unlike the
Virasoro case) is non-trivial even when $c>N-1$. This places a lower bound on
the dimension of any non-vacuum higher-spin primary state, which is linear in
the central charge. At large $c$, this implies that the dual holographic
theories of gravity in AdS$_3$, if they exist, have no local, perturbative
degrees of freedom in the semi-classical limit.
