  Can the causal effects estimated in experiment be generalized to real-world
scenarios? This question lies at the heart of social science studies. External
validity primarily assesses whether experimental effects persist across
different settings, implicitly presuming the experiment's ecological
validity-that is, the consistency of experimental effects with their real-life
counterparts. However, we argue that this presumed consistency may not always
hold, especially in experiments involving multidimensional decision processes,
such as conjoint experiments. We introduce a formal model to elucidate how
attention and salience effects lead to three types of inconsistencies between
experimental findings and real-world phenomena: amplified effect magnitude,
effect sign reversal, and effect importance reversal. We derive testable
hypotheses from each theoretical outcome and test these hypotheses using data
from various existing conjoint experiments and our own experiments. Drawing on
our theoretical framework, we propose several recommendations for experimental
design aimed at enhancing the generalizability of survey experiment findings.
