  Fairness in machine learning (ML) has received much attention. However,
existing studies have mainly focused on the distributive fairness of ML models.
The other dimension of fairness, i.e., procedural fairness, has been neglected.
In this paper, we first define the procedural fairness of ML models, and then
give formal definitions of individual and group procedural fairness. We propose
a novel metric to evaluate the group procedural fairness of ML models, called
$GPF_{FAE}$, which utilizes a widely used explainable artificial intelligence
technique, namely feature attribution explanation (FAE), to capture the
decision process of the ML models. We validate the effectiveness of $GPF_{FAE}$
on a synthetic dataset and eight real-world datasets. Our experiments reveal
the relationship between procedural and distributive fairness of the ML model.
Based on our analysis, we propose a method for identifying the features that
lead to the procedural unfairness of the model and propose two methods to
improve procedural fairness after identifying unfair features. Our experimental
results demonstrate that we can accurately identify the features that lead to
procedural unfairness in the ML model, and both of our proposed methods can
significantly improve procedural fairness with a slight impact on model
performance, while also improving distributive fairness.
