  Positional Encodings (PEs) are used to inject word-order information into
transformer-based language models. While they can significantly enhance the
quality of sentence representations, their specific contribution to language
models is not fully understood, especially given recent findings that various
positional encodings are insensitive to word order. In this work, we conduct a
systematic study of positional encodings in \textbf{Bidirectional Masked
Language Models} (BERT-style) , which complements existing work in three
aspects: (1) We uncover the core function of PEs by identifying two common
properties, Locality and Symmetry; (2) We show that the two properties are
closely correlated with the performances of downstream tasks; (3) We quantify
the weakness of current PEs by introducing two new probing tasks, on which
current PEs perform poorly. We believe that these results are the basis for
developing better PEs for transformer-based language models. The code is
available at \faGithub~ \url{https://github.com/tigerchen52/locality\_symmetry}
