  What is the prospect of developing artificial general intelligence (AGI)? I
investigate this question by systematically comparing living and algorithmic
systems, with a special focus on the notion of "agency." There are three
fundamental differences to consider: (1) Living systems are autopoietic, that
is, self-manufacturing, and therefore able to set their own intrinsic goals,
while algorithms exist in a computational environment with target functions
that are both provided by an external agent. (2) Living systems are embodied in
the sense that there is no separation between their symbolic and physical
aspects, while algorithms run on computational architectures that maximally
isolate software from hardware. (3) Living systems experience a large world, in
which most problems are ill-defined (and not all definable), while algorithms
exist in a small world, in which all problems are well-defined. These three
differences imply that living and algorithmic systems have very different
capabilities and limitations. In particular, it is extremely unlikely that true
AGI (beyond mere mimicry) can be developed in the current algorithmic framework
of AI research. Consequently, discussions about the proper development and
deployment of algorithmic tools should be shaped around the dangers and
opportunities of current narrow AI, not the extremely unlikely prospect of the
emergence of true agency in artificial systems.
