  The dense matter equation of state (EOS) determines neutron star (NS)
structure but can be calculated reliably only up to one to two times the
nuclear saturation density, using accurate many-body methods that employ
nuclear interactions from chiral effective field theory constrained by
scattering data. In this work, we use physically motivated ansatzes for the
speed of sound $c_S$ at high density to extend microscopic calculations of
neutron-rich matter to the highest densities encountered in stable NS cores. We
show how existing and expected astrophysical constraints on NS masses and radii
from X-ray observations can constrain the speed of sound in the NS core. We
confirm earlier expectations that $c_S$ is likely to violate the conformal
limit of $c_S^2\leq c^2/3 $, possibly reaching values closer to the speed of
light $c$ at a few times the nuclear saturation density, independent of the
nuclear Hamiltonian. If QCD obeys the conformal limit, we conclude that the
rapid increase of $c_S$ required to accommodate a $2 $ M$_\odot$ NS suggests a
form of strongly interacting matter where a description in terms of nucleons
will be unwieldy, even between one and two times the nuclear saturation
density. For typical NSs with masses in the range $1.2-1.4~$ M$_\odot$, we find
radii between $10$ and $14$ km, and the smallest possible radius of a $1.4$
M$_{\odot}$ NS consistent with constraints from nuclear physics and
observations is $8.4$ km. We also discuss how future observations could
constrain the EOS and guide theoretical developments in nuclear physics.
