  We consider three dimensional conformal field theories that have a higher
spin symmetry that is slightly broken. The theories have a large N limit, in
the sense that the operators separate into single trace and multitrace and obey
the usual large N factorization properties. We assume that the spectrum of
single trace operators is similar to the one that one gets in the Vasiliev
theories. Namely, the only single trace operators are the higher spin currents
plus an additional scalar. The anomalous dimensions of the higher spin currents
are of order 1/N. Using the slightly broken higher spin symmetry we constrain
the three point functions of the theories to leading order in N. We show that
there are two families of solutions. One family can be realized as a theory of
N fermions with an O(N) Chern-Simons gauge field, the other as a N bosons plus
the Chern-Simons gauge field. The family of solutions is parametrized by the 't
Hooft coupling. At special parity preserving points we get the critical O(N)
models, both the Wilson-Fisher one and the Gross-Neveu one. Our analysis also
fixes the on shell three point functions of Vasiliev's theory on AdS_4 or dS_4.
