  We propose a deep representation of appearance, i. e., the relation of color,
surface orientation, viewer position, material and illumination. Previous
approaches have useddeep learning to extract classic appearance
representationsrelating to reflectance model parameters (e. g., Phong)
orillumination (e. g., HDR environment maps). We suggest todirectly represent
appearance itself as a network we call aDeep Appearance Map (DAM). This is a 4D
generalizationover 2D reflectance maps, which held the view direction fixed.
First, we show how a DAM can be learned from images or video frames and later
be used to synthesize appearance, given new surface orientations and viewer
positions. Second, we demonstrate how another network can be used to map from
an image or video frames to a DAM network to reproduce this appearance, without
using a lengthy optimization such as stochastic gradient descent
(learning-to-learn). Finally, we show the example of an appearance
estimation-and-segmentation task, mapping from an image showingmultiple
materials to multiple deep appearance maps.
