  In order to evaluate the prevalence of security and privacy practices on a
representative sample of the Web, researchers rely on website popularity
rankings such as the Alexa list. While the validity and representativeness of
these rankings are rarely questioned, our findings show the contrary: we show
for four main rankings how their inherent properties (similarity, stability,
representativeness, responsiveness and benignness) affect their composition and
therefore potentially skew the conclusions made in studies. Moreover, we find
that it is trivial for an adversary to manipulate the composition of these
lists. We are the first to empirically validate that the ranks of domains in
each of the lists are easily altered, in the case of Alexa through as little as
a single HTTP request. This allows adversaries to manipulate rankings on a
large scale and insert malicious domains into whitelists or bend the outcome of
research studies to their will. To overcome the limitations of such rankings,
we propose improvements to reduce the fluctuations in list composition and
guarantee better defenses against manipulation. To allow the research community
to work with reliable and reproducible rankings, we provide Tranco, an improved
ranking that we offer through an online service available at
https://tranco-list.eu.
