  Algorithm audits have increased in recent years due to a growing need to
independently assess the performance of automatically curated services that
process, filter, and rank the large and dynamic amount of information available
on the internet. Among several methodologies to perform such audits, virtual
agents stand out because they offer the ability to perform systematic
experiments, simulating human behaviour without the associated costs of
recruiting participants. Motivated by the importance of research transparency
and replicability of results, this paper focuses on the challenges of such an
approach. It provides methodological details, recommendations, lessons learned,
and limitations based on our experience of setting up experiments for eight
search engines (including main, news, image and video sections) with hundreds
of virtual agents placed in different regions. We demonstrate the successful
performance of our research infrastructure across multiple data collections,
with diverse experimental designs, and point to different changes and
strategies that improve the quality of the method. We conclude that virtual
agents are a promising venue for monitoring the performance of algorithms
across long periods of time, and we hope that this paper can serve as a basis
for further research in this area.
