  We examine the ability of gravitational lens time delays to reveal complex
structure in lens potentials. In Congdon, Keeton & Nordgren (2008), we
predicted how the time delay between the bright pair of images in a "fold" lens
scales with the image separation, for smooth lens potentials. Here we show that
the proportionality constant increases with the quadrupole moment of the lens
potential, and depends only weakly on the position of the source along the
caustic. We use Monte Carlo simulations to determine the range of time delays
that can be produced by realistic smooth lens models consisting of isothermal
ellipsoid galaxies with tidal shear. We can then identify outliers as "time
delay anomalies". We find evidence for anomalies in close image pairs in the
cusp lenses RX J1131$-$1231 and B1422+231. The anomalies in RX J1131$-$1231
provide strong evidence for substructure in the lens potential, while at this
point the apparent anomalies in B1422+231 mainly indicate that the time delay
measurements need to be improved. We also find evidence for time delay
anomalies in larger-separation image pairs in the fold lenses, B1608+656 and
WFI 2033$-$4723, and the cusp lens RX J0911+0551. We suggest that these
anomalies are caused by some combination of substructure and a complex lens
environment. Finally, to assist future monitoring campaigns we use our smooth
models with shear to predict the time delays for all known four-image lenses.
