  State-of-the-art face recognition systems are based on deep (convolutional)
neural networks. Therefore, it is imperative to determine to what extent face
templates derived from deep networks can be inverted to obtain the original
face image. In this paper, we study the vulnerabilities of a state-of-the-art
face recognition system based on template reconstruction attack. We propose a
neighborly de-convolutional neural network (\textit{NbNet}) to reconstruct face
images from their deep templates. In our experiments, we assumed that no
knowledge about the target subject and the deep network are available. To train
the \textit{NbNet} reconstruction models, we augmented two benchmark face
datasets (VGG-Face and Multi-PIE) with a large collection of images synthesized
using a face generator. The proposed reconstruction was evaluated using type-I
(comparing the reconstructed images against the original face images used to
generate the deep template) and type-II (comparing the reconstructed images
against a different face image of the same subject) attacks. Given the images
reconstructed from \textit{NbNets}, we show that for verification, we achieve
TAR of 95.20\% (58.05\%) on LFW under type-I (type-II) attacks @ FAR of 0.1\%.
Besides, 96.58\% (92.84\%) of the images reconstruction from templates of
partition \textit{fa} (\textit{fb}) can be identified from partition
\textit{fa} in color FERET. Our study demonstrates the need to secure deep
templates in face recognition systems.
