  Automated event extraction in social science applications often requires
corpus-level evaluations: for example, aggregating text predictions across
metadata and unbiased estimates of recall. We combine corpus-level evaluation
requirements with a real-world, social science setting and introduce the
IndiaPoliceEvents corpus--all 21,391 sentences from 1,257 English-language
Times of India articles about events in the state of Gujarat during March 2002.
Our trained annotators read and label every document for mentions of police
activity events, allowing for unbiased recall evaluations. In contrast to other
datasets with structured event representations, we gather annotations by posing
natural questions, and evaluate off-the-shelf models for three different tasks:
sentence classification, document ranking, and temporal aggregation of target
events. We present baseline results from zero-shot BERT-based models fine-tuned
on natural language inference and passage retrieval tasks. Our novel
corpus-level evaluations and annotation approach can guide creation of similar
social-science-oriented resources in the future.
