  Recording fast motion in a high FPS (frame-per-second) requires expensive
high-speed cameras. As an alternative, interpolating low-FPS videos from
commodity cameras has attracted significant attention. If only low-FPS videos
are available, motion assumptions (linear or quadratic) are necessary to infer
intermediate frames, which fail to model complex motions. Event camera, a new
camera with pixels producing events of brightness change at the temporal
resolution of $\mu s$ $(10^{-6}$ second $)$, is a game-changing device to
enable video interpolation at the presence of arbitrarily complex motion. Since
event camera is a novel sensor, its potential has not been fulfilled due to the
lack of processing algorithms. The pioneering work Time Lens introduced event
cameras to video interpolation by designing optical devices to collect a large
amount of paired training data of high-speed frames and events, which is too
costly to scale. To fully unlock the potential of event cameras, this paper
proposes a novel TimeReplayer algorithm to interpolate videos captured by
commodity cameras with events. It is trained in an unsupervised
cycle-consistent style, canceling the necessity of high-speed training data and
bringing the additional ability of video extrapolation. Its state-of-the-art
results and demo videos in supplementary reveal the promising future of
event-based vision.
