  Socially Interactive Agents (SIAs) are physical or virtual embodied agents
that display similar behavior as human multimodal behavior. Modeling SIAs'
non-verbal behavior, such as speech and facial gestures, has always been a
challenging task, given that a SIA can take the role of a speaker or a
listener. A SIA must emit appropriate behavior adapted to its own speech, its
previous behaviors (intra-personal), and the User's behaviors (inter-personal)
for both roles. We propose AMII, a novel approach to synthesize adaptive facial
gestures for SIAs while interacting with Users and acting interchangeably as a
speaker or as a listener. AMII is characterized by modality memory encoding
schema - where modality corresponds to either speech or facial gestures - and
makes use of attention mechanisms to capture the intra-personal and
inter-personal relationships. We validate our approach by conducting objective
evaluations and comparing it with the state-of-the-art approaches.
