  Lip-based biometric authentication (LBBA) is an authentication method based
on a person's lip movements during speech in the form of video data captured by
a camera sensor. LBBA can utilize both physical and behavioral characteristics
of lip movements without requiring any additional sensory equipment apart from
an RGB camera. State-of-the-art (SOTA) approaches use one-shot learning to
train deep siamese neural networks which produce an embedding vector out of
these features. Embeddings are further used to compute the similarity between
an enrolled user and a user being authenticated. A flaw of these approaches is
that they model behavioral features as style-of-speech without relation to what
is being said. This makes the system vulnerable to video replay attacks of the
client speaking any phrase. To solve this problem we propose a one-shot
approach which models behavioral features to discriminate against what is being
said in addition to style-of-speech. We achieve this by customizing the GRID
dataset to obtain required triplets and training a siamese neural network based
on 3D convolutions and recurrent neural network layers. A custom triplet loss
for batch-wise hard-negative mining is proposed. Obtained results using an
open-set protocol are 3.2% FAR and 3.8% FRR on the test set of the customized
GRID dataset. Additional analysis of the results was done to quantify the
influence and discriminatory power of behavioral and physical features for
LBBA.
