  We introduce a simple determinant diagrammatic Monte Carlo algorithm to
compute the ground-state properties of a particle interacting with a Fermi sea
through a zero-range interaction. The fermionic sign does not cause any
fundamental problem when going to high diagram orders, and we reach order
$N=30$. The data reveal that the diagrammatic series diverges exponentially as
$(-1/R)^{N}$ with a radius of convergence $R<1$. Furthermore, on the polaron
side of the polaron-dimeron transition, the value of $R$ is determined by a
special class of three-body diagrams, corresponding to repeated scattering of
the impurity between two particles of the Fermi sea. A power-counting argument
explains why finite $R$ is possible for zero-range interactions in three
dimensions. Resumming the divergent series through a conformal mapping yields
the polaron energy with record accuracy.
