  End-to-end learning framework is useful for building dialog systems for its
simplicity in training and efficiency in model updating. However, current
end-to-end approaches only consider user semantic inputs in learning and
under-utilize other user information. Therefore, we propose to include user
sentiment obtained through multimodal information (acoustic, dialogic and
textual), in the end-to-end learning framework to make systems more
user-adaptive and effective. We incorporated user sentiment information in both
supervised and reinforcement learning settings. In both settings, adding
sentiment information reduced the dialog length and improved the task success
rate on a bus information search task. This work is the first attempt to
incorporate multimodal user information in the adaptive end-to-end dialog
system training framework and attained state-of-the-art performance.
