  In spite of the recent success of Dialogue Act (DA) classification, the
majority of prior works focus on text-based classification with oracle
transcriptions, i.e. human transcriptions, instead of Automatic Speech
Recognition (ASR)'s transcriptions. In spoken dialog systems, however, the
agent would only have access to noisy ASR transcriptions, which may further
suffer performance degradation due to domain shift. In this paper, we explore
the effectiveness of using both acoustic and textual signals, either oracle or
ASR transcriptions, and investigate speaker domain adaptation for DA
classification. Our multimodal model proves to be superior to the unimodal
models, particularly when the oracle transcriptions are not available. We also
propose an effective method for speaker domain adaptation, which achieves
competitive results.
