  Binary neutron stars (BNSs) and neutron star--black hole (NSBH) binaries are
two of the most promising gravitational wave (GW) sources to probe matter
effects. Upcoming observing runs of LIGO-Virgo-KAGRA detectors and future third
generation detectors like Einstein Telescope and Cosmic Explorer will allow the
extraction of detailed information on these matter effects from the GW
signature of BNS and NSBH systems. One subtle effect which may be helpful to
extract more information from the detection of compact binary systems is the
nonlinear memory. In this work, we investigate the observational consequences
of gravitational wave nonlinear memory in the presence of matter effects. We
start by quantifying the impact of nonlinear memory on distinguishing BNS
mergers from binary black holes (BBHs) or NSBH mergers. We find that for the
third generation detectors, the addition of nonlinear memory to the GW signal
model expands the parameter space where BNS signals become distinguishable from
the BBH and NSBH signals. Using numerical relativity simulations, we also study
the nonlinear memory generated from the postmerger phase of BNS systems. We
find that it does not show a strong dependence on the equation of state of the
NS. However, the amplitude of nonlinear memory from the BNS postmerger phase is
much lower than the one from BBH systems of the same masses. Furthermore, we
compute the detection prospects of nonlinear memory from the postmerger phase
of NS systems by accumulating signal strength from a population of BNS mergers
for the current and future detectors. Finally, we discuss the impact of
possible linear memory from the dynamical ejecta of BNS and NSBH systems and
its signal strength relative to the nonlinear memory. We find that linear
memory almost always has a much weaker effect than nonlinear memory.
