  To learn about real world phenomena, scientists have traditionally used
models with clearly interpretable elements. However, modern machine learning
(ML) models, while powerful predictors, lack this direct elementwise
interpretability (e.g. neural network weights). Interpretable machine learning
(IML) offers a solution by analyzing models holistically to derive
interpretations. Yet, current IML research is focused on auditing ML models
rather than leveraging them for scientific inference. Our work bridges this
gap, presenting a framework for designing IML methods-termed 'property
descriptors' -- that illuminate not just the model, but also the phenomenon it
represents. We demonstrate that property descriptors, grounded in statistical
learning theory, can effectively reveal relevant properties of the joint
probability distribution of the observational data. We identify existing IML
methods suited for scientific inference and provide a guide for developing new
descriptors with quantified epistemic uncertainty. Our framework empowers
scientists to harness ML models for inference, and provides directions for
future IML research to support scientific understanding.
