  Machine Learning and AI have the potential to transform data-driven
scientific discovery, enabling accurate predictions for several scientific
phenomena. As many scientific questions are inherently causal, this paper looks
at the causal inference task of treatment effect estimation, where we assume
binary effects that are recorded as high-dimensional images in a Randomized
Controlled Trial (RCT). Despite being the simplest possible setting and a
perfect fit for deep learning, we theoretically find that many common choices
in the literature may lead to biased estimates. To test the practical impact of
these considerations, we recorded the first real-world benchmark for causal
inference downstream tasks on high-dimensional observations as an RCT studying
how garden ants (Lasius neglectus) respond to microparticles applied onto their
colony members by hygienic grooming. Comparing 6 480 models fine-tuned from
state-of-the-art visual backbones, we find that the sampling and modeling
choices significantly affect the accuracy of the causal estimate, and that
classification accuracy is not a proxy thereof. We further validated the
analysis, repeating it on a synthetically generated visual data set controlling
the causal model. Our results suggest that future benchmarks should carefully
consider real downstream scientific questions, especially causal ones. Further,
we highlight guidelines for representation learning methods to help answer
causal questions in the sciences. All code and data will be released.
