  Data-driven software solutions have significantly been used in critical
domains with significant socio-economic, legal, and ethical implications. The
rapid adoptions of data-driven solutions, however, pose major threats to the
trustworthiness of automated decision-support software. A diminished
understanding of the solution by the developer and historical/current biases in
the data sets are primary challenges.
  To aid data-driven software developers and end-users, we present \toolname, a
debugging tool to test and explain the fairness implications of data-driven
solutions. \toolname visualizes the logic of datasets, trained models, and
decisions for a given data point. In addition, it trains various models with
varying fairness-accuracy trade-offs. Crucially, \toolname incorporates
counterfactual fairness testing that finds bugs beyond the development
datasets. We conducted two studies through \toolname that allowed us to measure
false positives/negatives in prevalent counterfactual testing and understand
the human perception of counterfactual test cases in a class survey. \toolname
and its benchmarks are publicly available
at~\url{https://github.com/Pennswood/FairLay-ML}. The live version of the tool
is available at~\url{https://fairlayml-v2.streamlit.app/}. We provide a video
demo of the tool at https://youtu.be/wNI9UWkywVU?t=127
