  Recent observations of neutron stars with gravitational waves and X-ray
timing provide unprecedented access to the equation of state (EoS) of cold
dense matter at densities difficult to realize in terrestrial experiments. At
the same time, predictions for the EoS with reliable uncertainty estimates from
chiral effective field theory ($\chi$EFT) bound our theoretical ignorance. In
this work, we analyze astrophysical data using a nonparametric representation
of the neutron-star EoS conditioned on $\chi$EFT to directly constrain the
underlying physical properties of the compact objects. We discuss how the data
alone constrain the EoS at high densities when we condition on $\chi$EFT at low
densities. We also demonstrate how to exploit astrophysical data to directly
test the predictions of $\chi$EFT for the EoS up to twice nuclear saturation
density, and estimate the density at which these predictions might break down.
We find that the existence of massive pulsars, gravitational waves from
GW170817, and NICER observations of PSR J0030+0451 favor $\chi$EFT predictions
for the EoS up to nuclear saturation density over a more agnostic analysis by
as much as a factor of 7 for the quantum Monte Carlo (QMC) calculations used in
this work. While $\chi$EFT predictions using QMC are fully consistent with
gravitational-wave data up to twice nuclear saturation density, NICER
observations suggest that the EoS stiffens relative to these predictions at
nuclear saturation density. Additionally, we marginalize over the uncertainty
in the density at which $\chi$EFT begins to break down, constraining the radius
of a $1.4\,M_\odot$ neutron star to $R_{1.4}=11.40^{+1.38}_{-1.04}$
($12.54^{+0.71}_{-0.63}$) km and the pressure at twice nuclear saturation
density to $p(2n_\mathrm{sat})=14.2^{+18.1}_{-8.4}$ ($28.7^{+15.3}_{-15.0}$)
MeV/fm$^3$ with massive pulsar and gravitational-wave (and NICER) data.
