  Realistic fault-tolerant quantum computing at reasonable overhead requires
two-qubit gates with the highest possible fidelity. Typically, an infidelity of
$\lesssim 10^{-4}$ is recommended in the literature. Focusing on the
phase-sensitive architecture used in laboratories and by commercial companies
to implement quantum computers, we show that even under noise-free, ideal
conditions, neglecting the carrier term and linearizing the Lamb-Dicke term in
the Hamiltonian used for control-pulse construction for generating
M{\o}lmer-S{\o}rensen XX gates based on the Raman scheme are not justified if
the goal is an infidelity target of $10^{-4}$. We obtain these results with a
gate simulator code that, in addition to the computational space, explicitly
takes the most relevant part of the phonon space into account. With the help of
a Magnus expansion carried to the third order, keeping terms up to the fourth
order in the Lamb-Dicke parameters, we identify the leading sources of coherent
errors, which we show can be eliminated by adding a single linear equation to
the phase-space closure conditions and subsequently adjusting the amplitude of
the control pulse (calibration). This way, we obtain XX gates with infidelities
$< 10^{-4}$.
