  Transfer entropy (TE) is a measurement in information theory that reveals the
directional flow of information between processes, providing valuable insights
for a wide range of real-world applications. This work proposes Transfer
Entropy Estimation via Transformers (TREET), a novel transformer-based approach
for estimating the TE for stationary processes. The proposed approach employs
Donsker-Vardhan (DV) representation to TE and leverages the attention mechanism
for the task of neural estimation. We propose a detailed theoretical and
empirical study of the TREET, comparing it to existing methods. To increase its
applicability, we design an estimated TE optimization scheme that is motivated
by the functional representation lemma. Afterwards, we take advantage of the
joint optimization scheme to optimize the capacity of communication channels
with memory, which is a canonical optimization problem in information theory,
and show the memory capabilities of our estimator. Finally, we apply TREET to
real-world feature analysis. Our work, applied with state-of-the-art deep
learning methods, opens a new door for communication problems which are yet to
be solved.
