  Most conventional photometric stereo algorithms inversely solve a BRDF-based
image formation model. However, the actual imaging process is often far more
complex due to the global light transport on the non-convex surfaces. This
paper presents a photometric stereo network that directly learns relationships
between the photometric stereo input and surface normals of a scene. For
handling unordered, arbitrary number of input images, we merge all the input
data to the intermediate representation called {\it observation map} that has a
fixed shape, is able to be fed into a CNN. To improve both training and
prediction, we take into account the rotational pseudo-invariance of the
observation map that is derived from the isotropic constraint. For training the
network, we create a synthetic photometric stereo dataset that is generated by
a physics-based renderer, therefore the global light transport is considered.
Our experimental results on both synthetic and real datasets show that our
method outperforms conventional BRDF-based photometric stereo algorithms
especially when scenes are highly non-convex.
