  Neural networks are increasingly employed to model, analyze and control
non-linear dynamical systems ranging from physics to biology. Owing to their
universal approximation capabilities, they regularly outperform
state-of-the-art model-driven methods in terms of accuracy, computational
speed, and/or control capabilities. On the other hand, neural networks are very
often they are taken as black boxes whose explainability is challenged, among
others, by huge amounts of trainable parameters. In this paper, we tackle the
outstanding issue of analyzing the inner workings of neural networks trained to
classify regular-versus-chaotic time series. This setting, well-studied in
dynamical systems, enables thorough formal analyses. We focus specifically on a
family of networks dubbed Large Kernel Convolutional Neural Networks (LKCNN),
recently introduced by Boull\'{e} et al. (2021). These non-recursive networks
have been shown to outperform other established architectures (e.g. residual
networks, shallow neural networks and fully convolutional networks) at this
classification task. Furthermore, they outperform ``manual'' classification
approaches based on direct reconstruction of the Lyapunov exponent. We find
that LKCNNs use qualitative properties of the input sequence. In particular, we
show that the relation between input periodicity and activation periodicity is
key for the performance of LKCNN models. Low performing models show, in fact,
analogous periodic activations to random untrained models. This could give very
general criteria for identifying, a priori, trained models that have poor
accuracy.
