  Motivated by GW170817 we examine constraints that observations put on the
angular structure of long gamma-ray burst (GRB) jets. First, the relatively
narrow observed distribution of $E_{X}/E_{\gamma}$ (the isotropic equivalent
early X-ray afterglow to prompt $\gamma$-ray energy ratio) implies that at any
angle that $\gamma$-rays are emitted the Lorentz factor must be high.
Specifically, the Lorentz factor of $\gamma$-ray emitting material cannot drop
rapidly with angle, and must be $\Gamma(\theta)\gtrsim 50$ even if there are
angles for which the gamma-ray received energy is lower by three orders of
magnitude compared to the jet core. Second, jets with an angular structure of
the $\gamma$-ray emission that over-produce events with a $\gamma$-ray
luminosity below the peak of the observed luminosity function are ruled-out.
This eliminates models in which the $\gamma$-ray energy angular distribution
isn't sufficiently steep and the Lorentz factor distribution isn't sufficiently
shallow. Finally, models with a steep structure (e.g. Gaussian) which are
detected away from the jet core generate afterglow light-curves that were never
observed. We conclude that even if the jet kinetic energy distribution drops
continuously with latitude, efficient $\gamma$-ray emission seems to be
restricted to material with $\Gamma\gtrsim 50$ and is most likely confined to a
narrow region around the core. While our study is confined to long GRBs, where
the observed sample is larger and more complete, there are indications that
similar conclusions may be applicable also to short GRBs. We discuss the
possible implications to the $\gamma$-rays observed in GRB 170817A.
