  Active learning (AL) is a human-and-model-in-the-loop paradigm that
iteratively selects informative unlabeled data for human annotation, aiming to
improve over random sampling. However, performing AL experiments with human
annotations on-the-fly is a laborious and expensive process, thus unrealistic
for academic research. An easy fix to this impediment is to simulate AL, by
treating an already labeled and publicly available dataset as the pool of
unlabeled data. In this position paper, we first survey recent literature and
highlight the challenges across all different steps within the AL loop. We
further unveil neglected caveats in the experimental setup that can
significantly affect the quality of AL research. We continue with an
exploration of how the simulation setting can govern empirical findings,
arguing that it might be one of the answers behind the ever posed question
``why do active learning algorithms sometimes fail to outperform random
sampling?''. We argue that evaluating AL algorithms on available labeled
datasets might provide a lower bound as to their effectiveness in real data. We
believe it is essential to collectively shape the best practices for AL
research, particularly as engineering advancements in LLMs push the research
focus towards data-driven approaches (e.g., data efficiency, alignment,
fairness). In light of this, we have developed guidelines for future work. Our
aim is to draw attention to these limitations within the community, in the hope
of finding ways to address them.
