  Despite recent progress in open-domain dialogue evaluation, how to develop
automatic metrics remains an open problem. We explore the potential of dialogue
evaluation featuring dialog act information, which was hardly explicitly
modeled in previous methods. However, defined at the utterance level in
general, dialog act is of coarse granularity, as an utterance can contain
multiple segments possessing different functions. Hence, we propose segment
act, an extension of dialog act from utterance level to segment level, and
crowdsource a large-scale dataset for it. To utilize segment act flows,
sequences of segment acts, for evaluation, we develop the first consensus-based
dialogue evaluation framework, FlowEval. This framework provides a
reference-free approach for dialog evaluation by finding pseudo-references.
Extensive experiments against strong baselines on three benchmark datasets
demonstrate the effectiveness and other desirable characteristics of our
FlowEval, pointing out a potential path for better dialogue evaluation.
