  The ability to learn new tasks and generalize performance to others is one of
the most remarkable characteristics of the human brain and of recent AI
systems. The ability to perform multiple tasks simultaneously is also a
signature characteristic of large-scale parallel architectures, that is evident
in the human brain, and has been exploited effectively more traditional,
massively parallel computational architectures. Here, we show that these two
characteristics are in tension, reflecting a fundamental tradeoff between
interactive parallelism that supports learning and generalization, and
independent parallelism that supports processing efficiency through concurrent
multitasking. We formally show that, while the maximum number of tasks that can
be performed simultaneously grows linearly with network size, under realistic
scenarios (e.g. in an unpredictable environment), the expected number that can
be performed concurrently grows radically sub-linearly with network size.
Hence, even modest reliance on shared representation strictly constrains the
number of tasks that can be performed simultaneously, implying profound
consequences for the development of artificial intelligence that optimally
manages the tradeoff between learning and processing, and for understanding the
human brains remarkably puzzling mix of sequential and parallel capabilities.
