  The sky is a major component of the appearance of a photograph, and its color
and tone can strongly influence the mood of a picture. In nighttime
photography, the sky can also suffer from noise and color artifacts. For this
reason, there is a strong desire to process the sky in isolation from the rest
of the scene to achieve an optimal look. In this work, we propose an automated
method, which can run as a part of a camera pipeline, for creating accurate sky
alpha-masks and using them to improve the appearance of the sky. Our method
performs end-to-end sky optimization in less than half a second per image on a
mobile device. We introduce a method for creating an accurate sky-mask dataset
that is based on partially annotated images that are inpainted and refined by
our modified weighted guided filter. We use this dataset to train a neural
network for semantic sky segmentation. Due to the compute and power constraints
of mobile devices, sky segmentation is performed at a low image resolution. Our
modified weighted guided filter is used for edge-aware upsampling to resize the
alpha-mask to a higher resolution. With this detailed mask we automatically
apply post-processing steps to the sky in isolation, such as automatic
spatially varying white-balance, brightness adjustments, contrast enhancement,
and noise reduction.
