  A leaderboard is a tabular presentation of performance scores of the best
competing techniques that address a specific scientific problem. Manually
maintained leaderboards take time to emerge, which induces a latency in
performance discovery and meaningful comparison. This can delay dissemination
of best practices to non-experts and practitioners. Regarding papers as proxies
for techniques, we present a new system to automatically discover and maintain
leaderboards in the form of partial orders between papers, based on performance
reported therein. In principle, a leaderboard depends on the task, data set,
other experimental settings, and the choice of performance metrics. Often there
are also tradeoffs between different metrics. Thus, leaderboard discovery is
not just a matter of accurately extracting performance numbers and comparing
them. In fact, the levels of noise and uncertainty around performance
comparisons are so large that reliable traditional extraction is infeasible. We
mitigate these challenges by using relatively cleaner, structured parts of the
papers, e.g., performance tables. We propose a novel performance improvement
graph with papers as nodes, where edges encode noisy performance comparison
information extracted from tables. Every individual performance edge is
extracted from a table with citations to other papers. These extractions
resemble (noisy) outcomes of 'matches' in an incomplete tournament. We propose
several approaches to rank papers from these noisy 'match' outcomes. We show
that our ranking scheme can reproduce various manually curated leaderboards
very well. Using widely-used lists of state-of-the-art papers in 27 areas of
Computer Science, we demonstrate that our system produces very reliable
rankings.
