  The arrival times, positions, and fluxes of multiple images in strong lens
systems can be used to infer the presence of dark subhalos in the deflector,
and thus test predictions of cold dark matter models. However, gravitational
lensing does not distinguish between perturbations to a smooth gravitational
potential arising from baryonic and non-baryonic mass. In this work, we
quantify the extent to which the stellar mass distribution of a deflector can
reproduce flux ratio and astrometric anomalies typically associated with the
presence of a dark matter subhalo. Using Hubble Space Telescope images of
nearby galaxies, we simulate strong lens systems with real distributions of
stellar mass as they would be observed at redshift $z_d=0.5$. We add a dark
matter halo and external shear to account for the smooth dark matter field,
omitting dark substructure, and use a Monte Carlo procedure to characterize the
distributions of image positions, time delays, and flux ratios for a compact
background source of diameter 5 pc. By convolving high-resolution images of
real galaxies with a Gaussian PSF, we simulate the most detailed smooth
potential one could construct given high quality data, and find scatter in flux
ratios of $\approx 10\%$, which we interpret as a typical deviation from a
smooth potential caused by large and small scale structure in the lensing
galaxy. We demonstrate that the flux ratio anomalies arising from galaxy-scale
baryonic structure can be minimized by selecting the most massive and round
deflectors, and by simultaneously modeling flux ratio and astrometric data.
