  Consumers, regulators, and ISPs all use client-based "speed tests" to measure
network performance, both in single-user settings and in aggregate. Two
prevalent speed tests, Ookla's Speedtest and Measurement Lab's Network
Diagnostic Test (NDT), are often used for similar purposes, despite having
significant differences in both the test design and implementation, and in the
infrastructure used to perform measurements. In this paper, we present the
first-ever comparative evaluation of Ookla and NDT7 (the latest version of
NDT), both in controlled and wide-area settings. Our goal is to characterize
when and to what extent these two speed tests yield different results, as well
as the factors that contribute to the differences. To study the effects of the
test design, we conduct a series of controlled, in-lab experiments under a
comprehensive set of network conditions and usage modes (e.g., TCP congestion
control, native vs. browser client). Our results show that Ookla and NDT7
report similar speeds under most in-lab conditions, with the exception of
networks that experience high latency, where Ookla consistently reports higher
throughput. To characterize the behavior of these tools in wide-area
deployment, we collect more than 80,000 pairs of Ookla and NDT7 measurements
across nine months and 126 households, with a range of ISPs and speed tiers.
This first-of-its-kind paired-test analysis reveals many previously unknown
systemic issues, including high variability in NDT7 test results and
systematically under-performing servers in the Ookla network.
