  To develop rigorous knowledge about ML models -- and the systems in which
they are embedded -- we need reliable measurements. But reliable measurement is
fundamentally challenging, and touches on issues of reproducibility,
scalability, uncertainty quantification, epistemology, and more. This
dissertation addresses criteria needed to take reliability seriously: both
criteria for designing meaningful metrics, and for methodologies that ensure
that we can dependably and efficiently measure these metrics at scale and in
practice. In doing so, this dissertation articulates a research vision for a
new field of scholarship at the intersection of machine learning, law, and
policy. Within this frame, we cover topics that fit under three different
themes: (1) quantifying and mitigating sources of arbitrariness in ML, (2)
taming randomness in uncertainty estimation and optimization algorithms, in
order to achieve scalability without sacrificing reliability, and (3) providing
methods for evaluating generative-AI systems, with specific focuses on
quantifying memorization in language models and training latent diffusion
models on open-licensed data. By making contributions in these three themes,
this dissertation serves as an empirical proof by example that research on
reliable measurement for machine learning is intimately and inescapably bound
up with research in law and policy. These different disciplines pose similar
research questions about reliable measurement in machine learning. They are, in
fact, two complementary sides of the same research vision, which, broadly
construed, aims to construct machine-learning systems that cohere with broader
societal values.
