  About half of all optical observations collected via spaceborne satellites
are affected by haze or clouds. Consequently, cloud coverage affects the remote
sensing practitioner's capabilities of a continuous and seamless monitoring of
our planet. This work addresses the challenge of optical satellite image
reconstruction and cloud removal by proposing a novel multi-modal and
multi-temporal data set called SEN12MS-CR-TS. We propose two models
highlighting the benefits and use cases of SEN12MS-CR-TS: First, a multi-modal
multi-temporal 3D-Convolution Neural Network that predicts a cloud-free image
from a sequence of cloudy optical and radar images. Second, a
sequence-to-sequence translation model that predicts a cloud-free time series
from a cloud-covered time series. Both approaches are evaluated experimentally,
with their respective models trained and tested on SEN12MS-CR-TS. The conducted
experiments highlight the contribution of our data set to the remote sensing
community as well as the benefits of multi-modal and multi-temporal information
to reconstruct noisy information. Our data set is available at
https://patrickTUM.github.io/cloud_removal
