  Goal-oriented dialogue systems aim to help users achieve certain goals.
Therefore, how humans perceive their helpfulness is important. However, neither
the human-perceived helpfulness of goal-oriented dialogue systems nor its
fairness implication has been well studied. In this paper, we study
computational measurements of helpfulness. We first formally define a dialogue
response as helpful if it is relevant & coherent, useful, and informative to a
query. Then, we collect human annotations for the helpfulness of dialogue
responses based on our definition and build a classifier to automatically
determine the helpfulness of a response. We further propose to use the
helpfulness level of a dialogue system towards different user queries to
measure the fairness of a dialogue system. Experiments with state-of-the-art
dialogue systems under three information-seeking scenarios reveal that existing
systems tend to be more helpful for questions regarding concepts from
highly-developed countries than less-developed countries, uncovering potential
fairness concerns underlying the current goal-oriented dialogue systems.
