  Machine learning is used to make decisions for individuals in various fields,
which require us to achieve good prediction accuracy while ensuring fairness
with respect to sensitive features (e.g., race and gender). This problem,
however, remains difficult in complex real-world scenarios. To quantify
unfairness under such situations, existing methods utilize {\it path-specific
causal effects}. However, none of them can ensure fairness for each individual
without making impractical functional assumptions on the data. In this paper,
we propose a far more practical framework for learning an individually fair
classifier. To avoid restrictive functional assumptions, we define the {\it
probability of individual unfairness} (PIU) and solve an optimization problem
where PIU's upper bound, which can be estimated from data, is controlled to be
close to zero. We elucidate why our method can guarantee fairness for each
individual. Experimental results show that our method can learn an individually
fair classifier at a slight cost of accuracy.
