  Machine learning (ML) systems have achieved remarkable performance across a
wide area of applications. However, they frequently exhibit unfair behaviors in
sensitive application domains, raising severe fairness concerns. To evaluate
and test fairness, engineers often generate individual discriminatory instances
to expose unfair behaviors before model deployment. However, existing baselines
ignore the naturalness of generation and produce instances that deviate from
the real data distribution, which may fail to reveal the actual model fairness
since these unnatural discriminatory instances are unlikely to appear in
practice. To address the problem, this paper proposes a framework named Latent
Imitator (LIMI) to generate more natural individual discriminatory instances
with the help of a generative adversarial network (GAN), where we imitate the
decision boundary of the target model in the semantic latent space of GAN and
further samples latent instances on it. Specifically, we first derive a
surrogate linear boundary to coarsely approximate the decision boundary of the
target model, which reflects the nature of the original data distribution.
Subsequently, to obtain more natural instances, we manipulate random latent
vectors to the surrogate boundary with a one-step movement, and further conduct
vector calculation to probe two potential discriminatory candidates that may be
more closely located in the real decision boundary. Extensive experiments on
various datasets demonstrate that our LIMI outperforms other baselines largely
in effectiveness ($\times$9.42 instances), efficiency ($\times$8.71 speeds),
and naturalness (+19.65%) on average. In addition, we empirically demonstrate
that retraining on test samples generated by our approach can lead to
improvements in both individual fairness (45.67% on $IF_r$ and 32.81% on
$IF_o$) and group fairness (9.86% on $SPD$ and 28.38% on $AOD$}).
