  In recent years BERT shows apparent advantages and great potential in natural
language processing tasks. However, both training and applying BERT requires
intensive time and resources for computing contextual language representations,
which hinders its universality and applicability. To overcome this bottleneck,
we propose a deep bidirectional language model by using window masking
mechanism at attention layer. This work computes contextual language
representations without random masking as does in BERT and maintains the deep
bidirectional architecture like BERT. To compute the same sentence
representation, our method shows O(n) complexity less compared to other
transformer-based models with O($n^2$). To further demonstrate its superiority,
computing context language representations on CPU environments is conducted, by
using the embeddings from the proposed method, logistic regression shows much
higher accuracy in terms of SMS classification. Moverover, the proposed method
also achieves significant higher performance in semantic similarity tasks.
