  Open-domain Multi-Document Summarization (ODMDS) is a critical tool for
condensing vast arrays of documents into coherent, concise summaries. With a
more inter-related document set, there does not necessarily exist a correct
answer for the retrieval, making it hard to measure the retrieving performance.
We propose a rule-based method to process query-based document summarization
datasets into ODMDS datasets. Based on this method, we introduce a novel
dataset, ODSum, a sophisticated case with its document index interdependent and
often interrelated. We tackle ODMDS with the \textit{retrieve-then-summarize}
method, and the performance of a list of retrievers and summarizers is
investigated. Through extensive experiments, we identify variances in
evaluation metrics and provide insights into their reliability. We also found
that LLMs suffer great performance loss from retrieving errors. We further
experimented methods to improve the performance as well as investigate their
robustness against imperfect retrieval. We will release our data and code at
https://github.com/yale-nlp/ODSum.
