  BERT is a popular language model whose main pre-training task is to fill in
the blank, i.e., predicting a word that was masked out of a sentence, based on
the remaining words. In some applications, however, having an additional
context can help the model make the right prediction, e.g., by taking the
domain or the time of writing into account. This motivates us to advance the
BERT architecture by adding a global state for conditioning on a fixed-sized
context. We present our two novel approaches and apply them to an industry
use-case, where we complete fashion outfits with missing articles, conditioned
on a specific customer. An experimental comparison to other methods from the
literature shows that our methods improve personalization significantly.
