  The shift towards publicly available text sources has enabled language
processing at unprecedented scale, yet leaves under-serviced the domains where
public and openly licensed data is scarce. Proactively collecting text data for
research is a viable strategy to address this scarcity, but lacks systematic
methodology taking into account the many ethical, legal and
confidentiality-related aspects of data collection. Our work presents a case
study on proactive data collection in peer review -- a challenging and
under-resourced NLP domain. We outline ethical and legal desiderata for
proactive data collection and introduce "Yes-Yes-Yes", the first donation-based
peer reviewing data collection workflow that meets these requirements. We
report on the implementation of Yes-Yes-Yes at ACL Rolling Review and
empirically study the implications of proactive data collection for the dataset
size and the biases induced by the donation behavior on the peer reviewing
platform.
