  Fuzzy Dark Matter (FDM), consisting of ultralight bosons ($m_{\rm b} \sim
10^{-22}\ \rm eV$), is an intriguing alternative to Cold Dark Matter. Numerical
simulations that solve the Schr\"odinger-Poisson (SP) equation show that FDM
halos consist of a central solitonic core, which is the ground state of the SP
equation, surrounded by an envelope of interfering excited states. These
excited states also interfere with the soliton, causing it to oscillate and
execute a confined random walk with respect to the halo center of mass. Using
high-resolution numerical simulations of a $6.6 \times 10^9 M_{\odot}$ FDM halo
with $m_{\rm b} = 8 \times 10^{-23}\ \rm eV$ in isolation, we demonstrate that
the wobbling, oscillating soliton gravitationally perturbs nuclear objects,
such as supermassive black holes or dense star clusters, causing them to
diffuse outwards. In particular, we show that, on average, objects with mass
$\lesssim 0.3 \%$ of the soliton mass ($M_{\rm sol}$) are expelled from the
soliton in $\sim 3\ \rm Gyr$, after which they continue their outward diffusion
due to gravitational interactions with the soliton and the halo granules. More
massive objects ($\gtrsim 1 \% M_{\rm sol}$), while executing a random walk,
remain largely confined to the soliton due to dynamical friction. We also
present an effective treatment of the diffusion, based on kinetic theory, that
accurately reproduces the outward motion of low mass objects and briefly
discuss how the observed displacements of star clusters and active galactic
nuclei from the centers of their host galaxies can be used to constrain FDM.
