  Lensless cameras, innovatively replacing traditional lenses for ultra-thin,
flat optics, encode light directly onto sensors, producing images that are not
immediately recognizable. This compact, lightweight, and cost-effective imaging
solution offers inherent privacy advantages, making it attractive for
privacy-sensitive applications like face verification. Typical lensless face
verification adopts a two-stage process of reconstruction followed by
verification, incurring privacy risks from reconstructed faces and high
computational costs. This paper presents an end-to-end optimization approach
for privacy-preserving face verification directly on encoded lensless captures,
ensuring that the entire software pipeline remains encoded with no visible
faces as intermediate results. To achieve this, we propose several techniques
to address unique challenges from the lensless setup which precludes
traditional face detection and alignment. Specifically, we propose a face
center alignment scheme, an augmentation curriculum to build robustness against
variations, and a knowledge distillation method to smooth optimization and
enhance performance. Evaluations under both simulation and real environment
demonstrate our method outperforms two-stage lensless verification while
enhancing privacy and efficiency. Project website:
\url{lenslessface.github.io}.
