  Super-resolution (SR) has been widely used to convert low-resolution legacy
videos to high-resolution (HR) ones, to suit the increasing resolution of
displays (e.g. UHD TVs). However, it becomes easier for humans to notice motion
artifacts (e.g. motion judder) in HR videos being rendered on larger-sized
display devices. Thus, broadcasting standards support higher frame rates for
UHD (Ultra High Definition) videos (4K@60 fps, 8K@120 fps), meaning that
applying SR only is insufficient to produce genuine high quality videos. Hence,
to up-convert legacy videos for realistic applications, not only SR but also
video frame interpolation (VFI) is necessitated. In this paper, we first
propose a joint VFI-SR framework for up-scaling the spatio-temporal resolution
of videos from 2K 30 fps to 4K 60 fps. For this, we propose a novel training
scheme with a multi-scale temporal loss that imposes temporal regularization on
the input video sequence, which can be applied to any general video-related
task. The proposed structure is analyzed in depth with extensive experiments.
