  We show that if $h\in\mathbb{Z}[x]$ is a polynomial of degree $k$ such that
the congruence $h(x)\equiv0\pmod{q}$ has a solution for every positive integer
$q$, then any subset of $\{1,2,\ldots,N\}$ with no two distinct elements with
difference of the form $h(n)$, with $n$ positive integer, has density at most
$(\log N)^{-c\log\log\log N}$, for some constant $c$ that depends only on $k$.
This improves on the best bound in the literature, due to Rice, and generalizes
a recent result of Bloom and Maynard.
