  Recent innovations in transformers have shown their superior performance in
natural language processing (NLP) and computer vision (CV). The ability to
capture long-range dependencies and interactions in sequential data has also
triggered a great interest in time series modeling, leading to the widespread
use of transformers in many time series applications. However, being the most
common and crucial application, the adaptation of transformers to time series
forecasting has remained limited, with both promising and inconsistent results.
In contrast to the challenges in NLP and CV, time series problems not only add
the complexity of order or temporal dependence among input sequences but also
consider trend, level, and seasonality information that much of this data is
valuable for decision making. The conventional training scheme has shown
deficiencies regarding model overfitting, data scarcity, and privacy issues
when working with transformers for a forecasting task. In this work, we propose
attentive federated transformers for time series stock forecasting with better
performance while preserving the privacy of participating enterprises.
Empirical results on various stock data from the Yahoo! Finance website
indicate the superiority of our proposed scheme in dealing with the above
challenges and data heterogeneity in federated learning.
