  We derive the lateral Casimir-Polder force on a ground state atom on top of a
corrugated surface, up to first order in the corrugation amplitude. Our
calculation is based on the scattering approach, which takes into account
nonspecular reflections and polarization mixing for electromagnetic quantum
fluctuations impinging on real materials. We compare our first order exact
result with two commonly used approximation methods. We show that the proximity
force approximation (large corrugation wavelengths) overestimates the lateral
force, while the pairwise summation approach underestimates it due to the
non-additivity of dispersion forces. We argue that a frequency shift
measurement for the dipolar lateral oscillations of cold atoms could provide a
striking demonstration of nontrivial geometrical effects on the quantum vacuum.
