  Language models have become nearly ubiquitous in natural language processing
applications achieving state-of-the-art results in many tasks including
prosody. As the model design does not define predetermined linguistic targets
during training but rather aims at learning generalized representations of the
language, analyzing and interpreting the representations that models implicitly
capture is important in bridging the gap between interpretability and model
performance. Several studies have explored the linguistic information that
models capture providing some insights on their representational capacity.
However, the current studies have not explored whether prosody is part of the
structural information of the language that models learn. In this work, we
perform a series of experiments on BERT probing the representations captured at
different layers. Our results show that information about prosodic prominence
spans across many layers but is mostly focused in middle layers suggesting that
BERT relies mostly on syntactic and semantic information.
