  We present a human-in-the-loop approach to generate counterfactual (CF)
explanations that preserve global and local feasibility constraints. Global
feasibility constraints refer to the causal constraints that are necessary for
generating actionable CF explanation. Assuming a domain expert with knowledge
on unary and binary causal constraints, our approach efficiently employs this
knowledge to generate CF explanation by rejecting gradient steps that violate
these constraints. Local feasibility constraints encode end-user's constraints
for generating desirable CF explanation. We extract these constraints from the
end-user of the model and exploit them during CF generation via user-defined
distance metric. Through user studies, we demonstrate that incorporating causal
constraints during CF generation results in significantly better explanations
in terms of feasibility and desirability for participants. Adopting local and
global feasibility constraints simultaneously, although improves user
satisfaction, does not significantly improve desirability of the participants
compared to only incorporating global constraints.
