  Bayesian evidence ratios are widely used to quantify the statistical
consistency between different experiments. However, since the evidence ratio is
prior dependent, the precise translation between its value and the degree of
concordance/discordance requires additional information. The most commonly
adopted metric, the Jeffreys scale, can falsely suggest agreement between
datasets when priors are chosen to be sufficiently wide. In this work, we
examine evidence ratios in a DES-Y1 simulated analysis, focusing on the
internal consistency between weak lensing and galaxy clustering. We study two
scenarios using simulated data in controlled experiments. First, we calibrate
the expected evidence ratio distribution given noise realizations around the
best fit DES-Y1 $\Lambda$CDM cosmology. Second, we show the behavior of
evidence ratios for noiseless fiducial data vectors simulated using a modified
gravity model, which generates internal tension in the $\Lambda$CDM analysis.
We show that the choice of prior could conceal the discrepancies between weak
lensing and galaxy clustering induced by such models and that the evidence
ratio in a DES-Y1 study is, indeed, biased towards agreement.
