  In order to extract full cosmological information from next-generation large
and high-precision weak lensing (WL) surveys (e.g. Euclid, Roman, LSST),
higher-order statistics that probe the small-scale, non-linear regime of large
scale structure (LSS) need to be utilized. WL peak counts, which trace
overdensities in the cosmic web, are one promising and simple statistic for
constraining cosmological parameters. The physical origin of WL peaks have
previously been linked to dark matter halos along the line of sight and this
peak-halo connection has been used to develop various semi-analytic halo-based
models for predicting peak counts. Here, we study the origin of WL peaks and
the effectiveness of halo-based models for WL peak counts using a suite of
ray-tracing N-body simulations. We compare WL peaks in convergence maps from
the full simulations to those in maps created from only particles associated
with halos -- the latter playing the role of a "perfect" halo model. We find
that while halo-only contributions are able to replicate peak counts
qualitatively well, halos do not explain all WL peaks. Halos particularly
underpredict negative peaks, which are associated with local overdensities in
large-scale underdense regions along the line of sight. In addition, neglecting
non-halo contributions to peaks counts leads to a significant bias on the
parameters ($\Omega_{\rm m}$, $\sigma_{8}$) for surveys larger than $\geq$ 100
deg$^{2}$. We conclude that other elements of the cosmic web, outside and far
away from dark matter halos, need to be incorporated into models of WL peaks in
order to infer unbiased cosmological constraints.
