  Financial forecasting has been an important and active area of machine
learning research, as even the most modest advantage in predictive accuracy can
be parlayed into significant financial gains. Recent advances in natural
language processing (NLP) bring the opportunity to leverage textual data, such
as earnings reports of publicly traded companies, to predict the return rate
for an asset. However, when dealing with such a sensitive task, the consistency
of models -- their invariance under meaning-preserving alternations in input --
is a crucial property for building user trust. Despite this, current financial
forecasting methods do not consider consistency. To address this problem, we
propose FinTrust, an evaluation tool that assesses logical consistency in
financial text. Using FinTrust, we show that the consistency of
state-of-the-art NLP models for financial forecasting is poor. Our analysis of
the performance degradation caused by meaning-preserving alternations suggests
that current text-based methods are not suitable for robustly predicting market
information. All resources are available at
https://github.com/yingpengma/fintrust.
