  Developing neural architectures that are capable of logical reasoning has
become increasingly important for a wide range of applications (e.g., natural
language processing). Towards this grand objective, we propose a symbolic
reasoning architecture that chains many join operators together to model output
logical expressions. In particular, we demonstrate that such an ensemble of
join-chains can express a broad subset of ''tree-structured'' first-order
logical expressions, named FOET, which is particularly useful for modeling
natural languages. To endow it with differentiable learning capability, we
closely examine various neural operators for approximating the symbolic
join-chains. Interestingly, we find that the widely used multi-head
self-attention module in transformer can be understood as a special neural
operator that implements the union bound of the join operator in probabilistic
predicate space. Our analysis not only provides a new perspective on the
mechanism of the pretrained models such as BERT for natural language
understanding but also suggests several important future improvement
directions.
