  Gender, race and social biases have recently been detected as evident
examples of unfairness in applications of Natural Language Processing. A key
path towards fairness is to understand, analyse and interpret our data and
algorithms. Recent studies have shown that the human-generated data used in
training is an apparent factor of getting biases. In addition, current
algorithms have also been proven to amplify biases from data.
  To further address these concerns, in this paper, we study how an
state-of-the-art recurrent neural language model behaves when trained on data,
which under-represents females, using pre-trained standard and debiased word
embeddings. Results show that language models inherit higher bias when trained
on unbalanced data when using pre-trained embeddings, in comparison with using
embeddings trained within the task. Moreover, results show that, on the same
data, language models inherit lower bias when using debiased pre-trained
emdeddings, compared to using standard pre-trained embeddings.
