  The partition function of a 3d $\mathcal{N}=4$ gauge theory with rank $N$ can
be computed using supersymmetric localization in terms of a matrix model, which
often can be formulated as an ideal Fermi gas with a non-trivial one-particle
Hamiltonian. We show how OPE coefficients of protected operators correspond in
this formalism to averages of $n$-body operators in the Fermi gas, which can be
computed to all orders in $1/N$ using the WKB expansion. We use this formalism
to compute OPE coefficients in the $U(N)_k\times U(N)_{-k}$ ABJM theory as well
as the $U(N)$ theory with one adjoint and $N_f$ fundamental hypermultiplets,
both of which have weakly coupled M-theory duals in the large $N$ and finite
$k$ or $N_f$ regimes. For ABJM we reproduce known results, while for the $N_f$
theory we compute the all orders in $1/N$ dependence at finite $N_f$ for the
coefficient $c_T$ of the stress tensor two-point function.
