  Restoring face images from distortions is important in face recognition
applications and is challenged by multiple scale issues, which is still not
well-solved in research area. In this paper, we present a Sequential Gating
Ensemble Network (SGEN) for multi-scale face restoration issue. We first employ
the principle of ensemble learning into SGEN architecture design to reinforce
predictive performance of the network. The SGEN aggregates multi-level
base-encoders and base-decoders into the network, which enables the network to
contain multiple scales of receptive field. Instead of combining these
base-en/decoders directly with non-sequential operations, the SGEN takes
base-en/decoders from different levels as sequential data. Specifically, the
SGEN learns to sequentially extract high level information from base-encoders
in bottom-up manner and restore low level information from base-decoders in
top-down manner. Besides, we propose to realize bottom-up and top-down
information combination and selection with Sequential Gating Unit (SGU). The
SGU sequentially takes two inputs from different levels and decides the output
based on one active input. Experiment results demonstrate that our SGEN is more
effective at multi-scale human face restoration with more image details and
less noise than state-of-the-art image restoration models. By using adversarial
training, SGEN also produces more visually preferred results than other models
through subjective evaluation.
