The released model seems always favors system B
#1
by
sedecim
- opened
The released model seems always favors system B. The result after switching audio files for input A and B stays the same ~60% of the time.
Hi @sedecim , thanks for pointing out this! The position bias is indeed an important issue. Some existing works suppose this might be raised by the training data, or inherent to the left-to-right architecture of causal transformers (https://arxiv.org/abs/2306.05685).
We will take a closer look for this issue. We hope to fix this in the next version of GRM.