  Unfairness in mortgage lending has created generational inequality among
racial and ethnic groups in the US. Many studies address this problem, but most
existing work focuses on correlation-based techniques. In our work, we use the
framework of counterfactual fairness to train fair machine learning models. We
propose a new causal graph for the variables available in the Home Mortgage
Disclosure Act (HMDA) data. We use a matching-based approach instead of the
latent variable modeling approach, because the former approach does not rely on
any modeling assumptions. Furthermore, matching provides us with counterfactual
pairs in which the race variable is isolated. We first demonstrate the
unfairness in mortgage approval and interest rates between African-American and
non-Hispanic White sub-populations. Then, we show that having balanced data
using matching does not guarantee perfect counterfactual fairness of the
machine learning models.
