  We use sequential large-scale crawl data to empirically investigate and
validate the dynamics that underlie the evolution of the structure of the web.
We find that the overall structure of the web is defined by an intricate
interplay between experience or entitlement of the pages (as measured by the
number of inbound hyperlinks a page already has), inherent talent or fitness of
the pages (as measured by the likelihood that someone visiting the page would
give a hyperlink to it), and the continual high rates of birth and death of
pages on the web. We find that the web is conservative in judging talent and
the overall fitness distribution is exponential, showing low variability. The
small variance in talent, however, is enough to lead to experience
distributions with high variance: The preferential attachment mechanism
amplifies these small biases and leads to heavy-tailed power-law (PL) inbound
degree distributions over all pages, as well as over pages that are of the same
age. The balancing act between experience and talent on the web allows newly
introduced pages with novel and interesting content to grow quickly and surpass
older pages. In this regard, it is much like what we observe in high-mobility
and meritocratic societies: People with entitlement continue to have access to
the best resources, but there is just enough screening for fitness that allows
for talented winners to emerge and join the ranks of the leaders. Finally, we
show that the fitness estimates have potential practical applications in
ranking query results.
