  Is cognition a collection of loosely connected functions tuned to different
tasks, or can there be a general learning algorithm? If such an hypothetical
general algorithm did exist, tuned to our world, could it adapt seamlessly to a
world with different laws of nature? We consider the theory that predictive
coding is such a general rule, and falsify it for one specific neural
architecture known for high-performance predictions on natural videos and
replication of human visual illusions: PredNet. Our results show that PredNet's
high performance generalizes without retraining on a completely different
natural video dataset. Yet PredNet cannot be trained to reach even mediocre
accuracy on an artificial video dataset created with the rules of the Game of
Life (GoL). We also find that a submodule of PredNet, a Convolutional Neural
Network trained alone, reaches perfect accuracy on the GoL while being mediocre
for natural videos, showing that PredNet's architecture itself is responsible
for both the high performance on natural videos and the loss of performance on
the GoL. Just as humans cannot predict the dynamics of the GoL, our results
suggest that there might be a trade-off between high performance on sensory
inputs with different sets of rules.
