  Recent deep-learning models have achieved impressive prediction performance,
but often sacrifice interpretability and computational efficiency.
Interpretability is crucial in many disciplines, such as science and medicine,
where models must be carefully vetted or where interpretation is the goal
itself. Moreover, interpretable models are concise and often yield
computational efficiency. Here, we propose adaptive wavelet distillation (AWD),
a method which aims to distill information from a trained neural network into a
wavelet transform. Specifically, AWD penalizes feature attributions of a neural
network in the wavelet domain to learn an effective multi-resolution wavelet
transform. The resulting model is highly predictive, concise, computationally
efficient, and has properties (such as a multi-scale structure) which make it
easy to interpret. In close collaboration with domain experts, we showcase how
AWD addresses challenges in two real-world settings: cosmological parameter
inference and molecular-partner prediction. In both cases, AWD yields a
scientifically interpretable and concise model which gives predictive
performance better than state-of-the-art neural networks. Moreover, AWD
identifies predictive features that are scientifically meaningful in the
context of respective domains. All code and models are released in a
full-fledged package available on Github
(https://github.com/Yu-Group/adaptive-wavelets).
