  Increasing utilization of machine learning based decision support systems
emphasizes the need for resulting predictions to be both accurate and fair to
all stakeholders. In this work we present a novel approach to increase a Neural
Network model's fairness during training. We introduce a family of fairness
enhancing regularization components that we use in conjunction with the
traditional binary-cross-entropy based accuracy loss. These loss functions are
based on Bias Parity Score (BPS), a score that helps quantify bias in the
models with a single number. In the current work we investigate the behavior
and effect of these regularization components on bias. We deploy them in the
context of a recidivism prediction task as well as on a census-based adult
income dataset. The results demonstrate that with a good choice of fairness
loss function we can reduce the trained model's bias without deteriorating
accuracy even in unbalanced dataset.
