  Despite outperforming the human in many tasks, deep neural network models are
also criticized for the lack of transparency and interpretability in decision
making. The opaqueness results in uncertainty and low confidence when deploying
such a model in model sharing scenarios, when the model is developed by a third
party. For a supervised machine learning model, sharing training process
including training data provides an effective way to gain trust and to better
understand model predictions. However, it is not always possible to share all
training data due to privacy and policy constraints. In this paper, we propose
a method to disclose a small set of training data that is just sufficient for
users to get the insight of a complicated model. The method constructs a
boundary tree using selected training data and the tree is able to approximate
the complicated model with high fidelity. We show that traversing data points
in the tree gives users significantly better understanding of the model and
paves the way for trustworthy model sharing.
