  The popularity of conversational digital assistants has resulted in the
availability of large amounts of conversational data which can be utilized for
improved user experience and personalized response generation. Building these
assistants using popular large language models like ChatGPT also require
additional emphasis on prompt engineering and evaluation methods. Textual
similarity metrics are a key ingredient for such analysis and evaluations.
While many similarity metrics have been proposed in the literature, they have
not proven effective for task-oriented conversations as they do not take
advantage of unique conversational features. To address this gap, we present
TaskDiff, a novel conversational similarity metric that utilizes different
dialogue components (utterances, intents, and slots) and their distributions to
compute similarity. Extensive experimental evaluation of TaskDiff on a
benchmark dataset demonstrates its superior performance and improved robustness
over other related approaches.
