  Recovering shadows is an important step for many vision algorithms. Current
approaches that work with time-lapse sequences are limited to simple
thresholding heuristics. We show these approaches only work with very careful
tuning of parameters, and do not work well for long-term time-lapse sequences
taken over the span of many months. We introduce a parameter-free expectation
maximization approach which simultaneously estimates shadows, albedo, surface
normals, and skylight. This approach is more accurate than previous methods,
works over both very short and very long sequences, and is robust to the
effects of nonlinear camera response. Finally, we demonstrate that the shadow
masks derived through this algorithm substantially improve the performance of
sun-based photometric stereo compared to earlier shadow mask estimation.
