  Humans sometimes have an insight that leads to a sudden and drastic
performance improvement on the task they are working on. Sudden strategy
adaptations are often linked to insights, considered to be a unique aspect of
human cognition tied to complex processes such as creativity or meta-cognitive
reasoning. Here, we take a learning perspective and ask whether insight-like
behaviour can occur in simple artificial neural networks, even when the models
only learn to form input-output associations through gradual gradient descent.
We compared learning dynamics in humans and regularised neural networks in a
perceptual decision task that included a hidden regularity to solve the task
more efficiently. Our results show that only some humans discover this
regularity, whose behaviour was marked by a sudden and abrupt strategy switch
that reflects an aha-moment. Notably, we find that simple neural networks with
a gradual learning rule and a constant learning rate closely mimicked
behavioural characteristics of human insight-like switches, exhibiting delay of
insight, suddenness and selective occurrence in only some networks. Analyses of
network architectures and learning dynamics revealed that insight-like
behaviour crucially depended on a regularised gating mechanism and noise added
to gradient updates, which allowed the networks to accumulate "silent
knowledge" that is initially suppressed by regularised (attentional) gating.
This suggests that insight-like behaviour can arise naturally from gradual
learning in simple neural networks, where it reflects the combined influences
of noise, gating and regularisation.
