  Free-running Recurrent Neural Networks (RNNs), especially probabilistic
models, generate an ongoing information flux that can be quantified with the
mutual information $I\left[\vec{x}(t),\vec{x}(t\!+\!1)\right]$ between
subsequent system states $\vec{x}$. Although, former studies have shown that
$I$ depends on the statistics of the network's connection weights, it is
unclear (1) how to maximize $I$ systematically and (2) how to quantify the flux
in large systems where computing the mutual information becomes intractable.
Here, we address these questions using Boltzmann machines as model systems. We
find that in networks with moderately strong connections, the mutual
information $I$ is approximately a monotonic transformation of the
root-mean-square averaged Pearson correlations between neuron-pairs, a quantity
that can be efficiently computed even in large systems. Furthermore,
evolutionary maximization of $I\left[\vec{x}(t),\vec{x}(t\!+\!1)\right]$
reveals a general design principle for the weight matrices enabling the
systematic construction of systems with a high spontaneous information flux.
Finally, we simultaneously maximize information flux and the mean period length
of cyclic attractors in the state space of these dynamical networks. Our
results are potentially useful for the construction of RNNs that serve as
short-time memories or pattern generators.
