  Previous Part-Of-Speech (POS) induction models usually assume certain
independence assumptions (e.g., Markov, unidirectional, local dependency) that
do not hold in real languages. For example, the subject-verb agreement can be
both long-term and bidirectional. To facilitate flexible dependency modeling,
we propose a Masked Part-of-Speech Model (MPoSM), inspired by the recent
success of Masked Language Models (MLM). MPoSM can model arbitrary tag
dependency and perform POS induction through the objective of masked POS
reconstruction. We achieve competitive results on both the English Penn WSJ
dataset as well as the universal treebank containing 10 diverse languages.
Though modeling the long-term dependency should ideally help this task, our
ablation study shows mixed trends in different languages. To better understand
this phenomenon, we design a novel synthetic experiment that can specifically
diagnose the model's ability to learn tag agreement. Surprisingly, we find that
even strong baselines fail to solve this problem consistently in a very
simplified setting: the agreement between adjacent words. Nonetheless, MPoSM
achieves overall better performance. Lastly, we conduct a detailed error
analysis to shed light on other remaining challenges. Our code is available at
https://github.com/owenzx/MPoSM
