  We investigate sumset decompositions of quite general sets with restricted
prime factors. We manage to handle certain sets, such as the smooth numbers,
even though they have little sieve amenability, and conclude that these sets
cannot be written as a ternary sumset. This proves a conjecture by
S\'{a}rk\"ozy. We also clean up and sharpen existing results on sumset
decompositions of the prime numbers.
