  The most popular code review tools (e.g., Gerrit and GitHub) present the
files to review sorted in alphabetical order. Could this choice or, more
generally, the relative position in which a file is presented bias the outcome
of code reviews? We investigate this hypothesis by triangulating complementary
evidence in a two-step study.
  First, we observe developers' code review activity. We analyze the review
comments pertaining to 219,476 Pull Requests (PRs) from 138 popular Java
projects on GitHub. We found files shown earlier in a PR to receive more
comments than files shown later, also when controlling for possible confounding
factors: e.g., the presence of discussion threads or the lines added in a file.
Second, we measure the impact of file position on defect finding in code
review. Recruiting 106 participants, we conduct an online controlled experiment
in which we measure participants' performance in detecting two unrelated
defects seeded into two different files. Participants are assigned to one of
two treatments in which the position of the defective files is switched. For
one type of defect, participants are not affected by its file's position; for
the other, they have 64% lower odds to identify it when its file is last as
opposed to first. Overall, our findings provide evidence that the relative
position in which files are presented has an impact on code reviews' outcome;
we discuss these results and implications for tool design and code review.
  Data and materials: https://doi.org/10.5281/zenodo.6901285
