  Recent years have seen an increasing trend in the volume of personal media
captured by users, thanks to the advent of smartphones and smart glasses,
resulting in large media collections. Despite conversation being an intuitive
human-computer interface, current efforts focus mostly on single-shot natural
language based media retrieval to aid users query their media and re-live their
memories. This severely limits the search functionality as users can neither
ask follow-up queries nor obtain information without first formulating a
single-turn query.
  In this work, we propose dialogs for connected memories as a powerful tool to
empower users to search their media collection through a multi-turn,
interactive conversation. Towards this, we collect a new task-oriented dialog
dataset COMET, which contains $11.5k$ user<->assistant dialogs (totaling $103k$
utterances), grounded in simulated personal memory graphs. We employ a
resource-efficient, two-phase data collection pipeline that uses: (1) a novel
multimodal dialog simulator that generates synthetic dialog flows grounded in
memory graphs, and, (2) manual paraphrasing to obtain natural language
utterances. We analyze COMET, formulate four main tasks to benchmark meaningful
progress, and adopt state-of-the-art language models as strong baselines, in
order to highlight the multimodal challenges captured by our dataset.
