  To tackle the vocabulary problem in conversational systems, previous work has
applied unsupervised learning approaches on co-occurring speech and eye gaze
during interaction to automatically acquire new words. Although these
approaches have shown promise, several issues related to human language
behavior and human-machine conversation have not been addressed. First,
psycholinguistic studies have shown certain temporal regularities between human
eye movement and language production. While these regularities can potentially
guide the acquisition process, they have not been incorporated in the previous
unsupervised approaches. Second, conversational systems generally have an
existing knowledge base about the domain and vocabulary. While the existing
knowledge can potentially help bootstrap and constrain the acquired new words,
it has not been incorporated in the previous models. Third, eye gaze could
serve different functions in human-machine conversation. Some gaze streams may
not be closely coupled with speech stream, and thus are potentially detrimental
to word acquisition. Automated recognition of closely-coupled speech-gaze
streams based on conversation context is important. To address these issues, we
developed new approaches that incorporate user language behavior, domain
knowledge, and conversation context in word acquisition. We evaluated these
approaches in the context of situated dialogue in a virtual world. Our
experimental results have shown that incorporating the above three types of
contextual information significantly improves word acquisition performance.
