  Unprecedented access to multi-temporal satellite imagery has opened new
perspectives for a variety of Earth observation tasks. Among them,
pixel-precise panoptic segmentation of agricultural parcels has major economic
and environmental implications. While researchers have explored this problem
for single images, we argue that the complex temporal patterns of crop
phenology are better addressed with temporal sequences of images. In this
paper, we present the first end-to-end, single-stage method for panoptic
segmentation of Satellite Image Time Series (SITS). This module can be combined
with our novel image sequence encoding network which relies on temporal
self-attention to extract rich and adaptive multi-scale spatio-temporal
features. We also introduce PASTIS, the first open-access SITS dataset with
panoptic annotations. We demonstrate the superiority of our encoder for
semantic segmentation against multiple competing architectures, and set up the
first state-of-the-art of panoptic segmentation of SITS. Our implementation and
PASTIS are publicly available.
