  We define a necessary and sufficient condition on a polynomial $h\in
\mathbb{Z}[x]$ to guarantee that every set of natural numbers of positive upper
density contains a nonzero difference of the form $h(p)$ for some prime $p$.
Moreover, we establish a quantitative estimate on the size of the largest
subset of ${1,2,\dots,N}$ which lacks the desired arithmetic structure, showing
that if deg$(h)=k$, then the density of such a set is at most a constant times
$(\log N)^{-c}$ for any $c<1/(2k-2)$. We also discuss how an improved version
of this result for $k=2$ and a relative version in the primes can be obtained
with some additional known methods.
