  Deep neural networks are largely used for complex prediction tasks. There is
plenty of empirical evidence of their successful end-to-end training for a
diversity of tasks. Success is often measured based solely on the final
performance of the trained network, and explanations on when, why and how they
work are less emphasized. In this paper we study encoder-decoder recurrent
neural networks with attention mechanisms for the task of reading handwritten
chess scoresheets. Rather than prediction performance, our concern is to better
understand how learning occurs in these type of networks. We characterize the
task in terms of three subtasks, namely input-output alignment, sequential
pattern recognition, and handwriting recognition, and experimentally
investigate which factors affect their learning. We identify competition,
collaboration and dependence relations between the subtasks, and argue that
such knowledge might help one to better balance factors to properly train a
network.
