  The probing classifiers framework has been employed for interpreting deep
neural network models for a variety of natural language processing (NLP)
applications. Studies, however, have largely focused on sentencelevel NLP
tasks. This work is the first to apply the probing paradigm to representations
learned for document-level information extraction (IE). We designed eight
embedding probes to analyze surface, semantic, and event-understanding
capabilities relevant to document-level event extraction. We apply them to the
representations acquired by learning models from three different LLM-based
document-level IE approaches on a standard dataset. We found that trained
encoders from these models yield embeddings that can modestly improve argument
detections and labeling but only slightly enhance event-level tasks, albeit
trade-offs in information helpful for coherence and event-type prediction. We
further found that encoder models struggle with document length and
cross-sentence discourse.
