  This paper presents our latest investigation on modeling backchannel in
conversations. Motivated by a proactive backchanneling theory, we aim at
developing a system which acts as a proactive listener by inserting
backchannels, such as continuers and assessment, to influence speakers. Our
model takes into account not only lexical and acoustic cues, but also
introduces the simple and novel idea of using listener embeddings to mimic
different backchanneling behaviours. Our experimental results on the
Switchboard benchmark dataset reveal that acoustic cues are more important than
lexical cues in this task and their combination with listener embeddings works
best on both, manual transcriptions and automatically generated transcriptions.
