<p>As the other response noted, a lot of the answers will depend on how you define the terms. Here, I will use Legg &amp; Hutter's (2007) definition of intelligence, where an agent is intelligent if it can achieve a large variety of goals in a large variety of environments. For instance, a chess-playing robot is great at achieving the goal of winning at chess when it is in the environment of a chess game where nobody cheats or does anything else odd, but it fails miserably at just about everything else. So it has low intelligence. In contrast, a human can achieve many different goals in many different environments, from hunting food in a rainforest, to repairing solar panels while wearing a space suit on Earth's orbit. So a human has a considerably higher intelligence. See the paper for a formalization of this intuitive definition.</p>

<p>Note that a system can be exceedingly complex, or good at one thing, without necessarily being intelligent in this sense. Complexity does not imply intelligence by itself. But let's suppose that a system were intelligent in this sense? Well, let's tackle each of your terms in turn.</p>

<p><strong>Self-awareness.</strong> One widely-used definition for self-awareness is being "aware of themselves as a differentiated and unique entity in the world". For instance, Rochat (2003) discusses the development of self-awareness in babies and very young children:</p>

<blockquote>
  <p>All parents also notice an important change at around 2 years of age when children
  manifest ‘‘self-consciousness,’’ the so-called secondary emotions such as embarrassment or pride
  in very specific situations such as mirror exposure or competitive games (Kagan, 1984; Lewis,
  1992). Prior to the second year, an infant placed in front of a mirror will typically smile, coo, and
  explore in apparent delight of the perfect contingency between acted and seen movements
  bouncing back at them from the polished surface of the mirror (Amsterdam, 1972). By 2 years, the
  specular image is associated with radically different behaviors. Toddlers become typically frozen
  and sometime behave as if they wanted to hide themselves by tucking their head in their shoulders
  or hiding their face behind their hands. They show embarrassment. This is a robust phenomenon
  and one is naturally tempted to ask what it means psychologically for children in their development. [...]</p>
  
  <p>It appears indeed that by 2–3
  years young children do start to have others in mind when they behave. The expression of embarrassment
  that children often begin to display in front of mirrors at around this age is the
  expression of such ‘‘self-consciousness.’’ They behave not unlike criminals hiding their face to the
  cameras. Their behavior indicates a drive to vanish from the public eyes, as if they came to grip via
  the experience of their own specular image of how they present themselves to the world. Not only
  do they discover in the mirror that it is themselves, they also realize that it is themselves as
  perceived by others. The malaise might come from the realization of a fundamental discrepancy
  between how the child represents herself from within, and how he or she is actually perceived by
  others as reflected in the mirror.</p>
</blockquote>

<p>So self-awareness implies an understanding of the fact that there is such a thing as a "me", which is separate from the world as a whole, and which others will perceive differently than it will perceive itself. There can also be varying levels of self-awareness, ranging from the level 1 self-awareness that newborn infants have ("immediately after birth, infants are capable of demonstrating
already a sense of their own body as a differentiated entity: an entity among other entities in the
environment") to level 5 self-awareness involving a theory of mind and an understanding of the fact that others may have false beliefs about something while you yourself have a true belief of it.</p>

<p>Would a superintelligence necessarily need to be self-aware, in any of these senses? I'd say that it would. While there are lots of goals that could plausibly be solved without any kind of self-awareness (proving mathematical theorems, for instance), many seem impossible to solve without it. Pretty much any social activity requires you to have a model about others as distinct from yourself, as well as an ability to model their beliefs and attitudes towards yourself. So in order to really be capable of solving many goals, you need to be self-aware. On the other hand, if it was a narrow intelligence that had been built to <em>only</em> solve theorems of math, it probably wouldn't need self-awaraness.</p>

<p><strong>Emotion</strong>. There are many theories of emotion. For instance, the <em>Information Principle</em> says that <em>emotional feelings provide conscious information from unconscious appraisals of situations</em> (Clore et al. 2001). Your brain is constantly appraising the situation you happen to be in. It notes things like a passerby having slightly threatening body language, or conversation with another person being easy and free of misunderstandings. There are countless of such evaluations going on all the time, and you aren't consciously aware of them, because you don't need to. Your subconscious mind can handle them just fine on its own. The end result of all those evaluations is packaged into a brief summary, which is the only thing that your conscious mind sees directly. That "executive summary" is what you experience as a particular emotional state. The passerby makes you feel slightly nervous and you avoid her, or your conversational partner feels pleasant to talk with and you begin to like him, even though you don't know why.</p>

<p>So this theory says that emotions are both a source of information about unconscious appraisals, and a way to bias your behavior on the basis of circumstances. For instance, in happy moods (when things are presumably going well), we tend to rely more on our pre-existing routines and heuristics, while in sad or frustrated moods (when things aren't going so well), our minds switch to a more analytical mode of thought where we're more likely to take apart our existing beliefs and routines to find whether there's something wrong with them (Schwarz, 2010).</p>

<p>Would a superintelligence have something that we would consider to be akin to "emotions"? I would think that it might or might not, depending on its architecture. Thinking for a moment about humans, maybe a human with a vastly larger working memory capacity wouldn't need to have the subconscious appraisals packaged into summaries in the form of emotions, but could just constantly review their un-summarized conclusions while also thinking about other things. Maybe such a human could also learn to consciously control the style of their information processing, and adapt either heuristic or analytical thought as seemed reasonable. Arguably, such a human wouldn't need emotions. So it seems plausible that a superintelligence wouldn't necessarily have them either.</p>

<p><strong>Free will.</strong> Okay, now we are getting into tricky territory. There are lots of definitions about free will, so I'll merely say that the <a href="http://plato.stanford.edu/entries/computational-mind/">computational theory of mind</a> seems to be the majority view in cognitive science. If you accept the computational theory of mind, then you also consider the human mind to be a kind of a computer executing certain algorithms. Then on most accounts, if you would consider a human to have free will, you would probably also consider a machine intelligence to have free will, and vice versa.</p>

<p><strong>Consciousness.</strong> This is another vague term, and people mean different things by it. Copeland (1993) notes at least three possible meanings:</p>

<ol>
<li><em>Baseline sense</em>: an entity is conscious if it is capable of perceiving the world via sense organs of some sort, and if it is capable of performing inner activities such as reasoning, deliberating, judging, hypothesing, planning, and so on.</li>
<li><em>Consciousness as internal monitoring</em>: an entity is conscious if it is capable of observing and reporting on itself. For instance, I am not simply thinking looking up Copeland's definitions and typing a summary of them: I also know that I am doing so, and can think about this fact.</li>
<li><em>Phenomenological consciousness</em>: an entity is conscious if there's something that being that entity "feels like". Pain and pleasure are not <em>just</em> algorithmical responses that modify an entity's behavior: they <em>feel</em> painful or pleasant. This is the hardest category to really express in words, but hopefully everyone understands what's meant.</li>
</ol>

<p>A superintelligence would most definitely need to be conscious in the baseline sense: if it couldn't collect information from the world and make plans on that basis, it wouldn't be much of an intelligence. It would probably also need be conscious in the sense of being capable of internal monitoring, for otherwise it might not really be self-aware, and we already concluded that self-awareness is necessary for (broad) intelligence.</p>

<p>What about phenomenological consciousness? Chalmers (1995) has a pretty convincing (to me) thought experiment, where he says that a human who was changed into a robot, replacing one neuron at a time with an artificial one, would remain phenomenally conscious. But that argument presumes that the human retains their <em>mental</em> architecture unchanged. If we built a mind that had an entirely different mental architecture as humans did, would it exhibit phenomenal consciousness? There are lots of arguments, but the essential gist is that nobody knows.</p>

<p><strong>Indepedent goal-directed behavior.</strong> But there's one more thing that your question seems to be implying, which wasn't fully covered by any of the terms above. You seem to also be asking whether an intelligent system might <em>start doing things which conflict with the desires of its creators</em>.</p>

<p>The answer seems to be yes. Omohundro (2008) argues that even a chess-playing robot, if it were intelligent enough and if it were not designed carefully enough, "will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety". Suppose that the robot has "winning at chess" as its goal. Now, it will exhibit these behaviors because, if it is turned off it might not be able to play any more chess, if it can make copies of itself those copies can continue to play chess even if some were turned off, and extra resources will help it in making sure that it can always win at chess. Omohundro also mentions a number of other drives which will arise as natural subgoals of almost any other goal. </p>

<p>Armstrong et al. (forthcoming) have an extensive discussion about "Oracle AI", an artificial intelligence which would be built to be safe, as it was built to only answer questions and not do anything else. But even this is harder to do than it might seem. Even trying to program the Oracle AI to want to stay within a specific physical box requires a number of definitions about what constitutes its physical location and what constitutes the Oracle AI itself (is it still staying within the box if it makes remote copies of itself?), and it might plausibly discover theories of physics in which those definitions became meaningless. Note that all of this presumes that we actually understand the OAI well enough to <em>be able</em> to specify its goals with such detail. If we don't, then that plan fails right away. Probably the best existing review article about this topic is Muehlhauser &amp; Helm (forthcoming).</p>

<p><strong>References</strong></p>

<p>Armstrong, S. &amp; Sandberg, A. &amp; Bostrom, N. (forthcoming) Thinking inside the box: using and
controlling an Oracle AI. <a href="http://www.aleph.se/papers/oracleAI.pdf">http://www.aleph.se/papers/oracleAI.pdf</a></p>

<p>Chalmers, D.J. (1995) Absent Qualia, Fading Qualia, Dancing Qualia. In Metzinger, T. (Ed.), <em>Conscious Experience</em>. Imprint Academic. <a href="http://consc.net/papers/qualia.html">http://consc.net/papers/qualia.html</a></p>

<p>Clore, G.L. &amp; Gasper, K., &amp; Garvin, E. (2001). Affect as information. In Forgas, J.P. (Ed.), <em>Handbook of affect and social cognition</em> (pp. 121–144). Mahwah, NJ: Erlbaum.</p>

<p>Copeland, J. (1993) <em>Artificial Intelligence: A Philosophical Introduction</em>. Blackwell.</p>

<p>Legg, S. &amp; Hutter, M. (2007) Universal Intelligence: A Definition of Machine Intelligence. <em>Minds &amp; Machines</em>, 17:4 391-444. <a href="http://arxiv.org/abs/0712.3329">http://arxiv.org/abs/0712.3329</a></p>

<p>Muehlhauser, L. &amp; Helm, L. (forthcoming). The Singularity and Machine Ethics. <a href="http://commonsenseatheism.com/wp-content/uploads/2011/11/Muehlhauser-Helm-The-Singularity-and-Machine-Ethics-draft.pdf">http://commonsenseatheism.com/wp-content/uploads/2011/11/Muehlhauser-Helm-The-Singularity-and-Machine-Ethics-draft.pdf</a></p>

<p>Omohundro, S. (2008) "The basic AI drives," in Wang, P. Goertzel, P. and Franklin,
S. (eds.), <em>Proceedings of the First AGI Conference. Frontiers in Articial Intelli-
gence and Applications</em>, Volume 171 (IOS Press) 483-494. <a href="https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf">https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf</a></p>

<p>Rochat, P. (2003) Five levels of self-awareness as they unfold early in life. <em>Consciousness and Cognition</em>, 12, 717-731. <a href="http://www.psychology.emory.edu/cognition/rochat/lab/5%20levels%20of%20self-awareness.pdf">http://www.psychology.emory.edu/cognition/rochat/lab/5%20levels%20of%20self-awareness.pdf</a></p>

<p>Schwarz, N. (2010) Feelings as information. In Van Lange, P. &amp; Kruglanski, A. &amp; Higgins, E.T. (Eds.), Handbook of theories of social psychology, Sange. <a href="http://people.ict.usc.edu/~gratch/CSCI534/schwarz_feelings-as-information_7jan10.pdf">http://people.ict.usc.edu/~gratch/CSCI534/schwarz_feelings-as-information_7jan10.pdf</a></p>

CC BY-SA 3.0
2012-03-02T11:34:15.093Z
Superb answer. The final part "Indepedent goal-directed behavior" just answered by question I asked in the first answer.
Ubermensch
CC BY-SA 3.0
2012-03-02T14:44:22.707Z
Excellent answer-- it's great to see people putting in the time to answer questions at a level of detail that really answers the question thoroughly and provides references for further research
Jeff
CC BY-SA 3.0
2012-06-15T01:51:27.417Z
Thank you for putting so much effort in your anwser (I wish I would upvote a few more times). My view is that we still don't have a clue about how the brain works, and the same is true for intelligence. [Brain science might be lacking theory](http://www.ted.com/talks/jeff_hawkins_on_how_brain_science_will_change_computing.html). But the fact that we know so little is probably what makes it so exiting.
Benjamin Crouzier
CC BY-SA 3.0
2012-03-02T10:00:13.130Z
2012-03-02T10:00:13.130Z
Kaj_Sotala
<p>Great question. It really hits cognitive science in some tender spots, though. None of those terms you listed in the question have any sort of consensus within the field. The only one we are approaching an understanding of is probably emotion. Consciousness and free will are... tricky. I honestly don't believe there will ever be an agreed upon definition of consciousness that satisfies. It's just a term, and it does not spring from study of the brain. It's like if physics were still tied to trying to explain things in terms of the "ether". There is no physical evidence of this "ether", and so it has been abandoned. But it might be hard for people to abandon consciousness as easily, for obvious reasons. I understand the essence of what you're asking, but it's really not easy to answer. I'll try though. </p>

<p>The first thing to understand, is that the brain has <em>evolved</em>. This means that over a long time, tiny exceptions built up to create our brain and thus our minds. They are unreasonably messy. Speaking of any artificial system in terms of our own mental system is pretty much a mistake (at best), or deceptive (at worst). To say that the internet becomes self-aware is to go too far in anthropomorphizing it. Our minds are inherently tied into the crazy messy brains that produce them. Even our idea of free-will is not some omnipresent universal phenomena, it's just an artifact of how our brains are constructed. We make choices under certain measurable conditions, it isn't magic. For the most part, free will is only <em>virtually</em> true. </p>

<p>Your question assumes certain things that are really hard to articulate, but which are not exactly true. Intelligence and cognition sort of happen in the cortex, as in Hawkins model, but it's really not the full story. If you built a big HTM system like Hawkins describes, it would be inert. Consciousness, free-will, emotion, and self-awareness are not properties of the cortex, they are properties of the brain (of which the cortex is a part), and the body as a whole. We are self-aware because we have apparatus that can perceive ourselves. If you built an HTM system, and fed it weather data, it would not become self aware because it would not be perceiving itself. It would not have emotion because there would be no systems impelling the system to act in one way over another. The factors you list are related to "intelligence" as defined by cortical size, but they do not solely depend on it. </p>

<p>I think the fallacious assumption in your question is very related to the idea that living things exist on an evolutionary ladder. And that we are the "highest rung" on that ladder. As if we let other creatures evolve more, they would become more like us. It is a sort of deterministic belief that as we construct increasingly intelligent machines, they will become more like us. Make them smart enough, and they will begin exhibiting things that we exhibit. It's simply not true. We eventually will be able to build systems that could be considered self-aware, or having free-will, or whatever, but it will be because we intentionally design it to do so. These properties will not emerge on their own.</p>

CC BY-SA 3.0
2012-03-02T07:28:45.747Z
Top answer. But can you kindly elaborate me on the final part. Just because brain is a complex structure that evolved over time, does that mean that we cannot create a structure more complex that brain(I am sure that the entire working of the brain is still not deciphered and it won't be in the near future) given the sensory capabilities and perception of knowledge by the humans is introduced into the machine. Simply, can't a AI machine be created that's intelligent than a human(not intelligent than mankind).
Ubermensch
CC BY-SA 3.0
2012-03-02T07:32:24.230Z
Also, I think (I amn't sure) that knowledge increases with   more information provided the capability to transform information into valuable knowledge with perception that could be further transformed into meaningful decisions. Can't this be replicated in machines at a minor scale (I am extending a futuristic argument and not for the near future plus I am interested in AI)
Ubermensch
CC BY-SA 3.0
2012-03-02T06:35:36.873Z
2012-03-02T06:35:36.873Z
Preece
<p>After seeing <a href="http://vimeo.com/34696162">this talk</a>, the question popped in my mind. The idea is that as soon as a system is complex enough or intelligent enough, it able to act on its own. It seems to be a common belief. For example, you can find it in the movie <em>Terminator</em>, where <a href="http://en.wikipedia.org/wiki/Skynet_%28Terminator%29">skynet</a>, an advanced computer, becomes self aware and decide to eradicate humanity. The same is true with the  <a href="http://fr.wikipedia.org/wiki/HAL_9000">HAL</a> in <em>2001: A Space Odyssey</em>, where the computer decides to act despite receiving the opposite orders from the astronaut.</p>

<p>I believe what is said in this <a href="http://www.ted.com/talks/antonio_damasio_the_quest_to_understand_consciousness.html">ted talk</a>, where Antonio Damasio explains that consciousnesses is not a great product of the neocortex, but rather a trick achieved by other older parts in the brain (essentially the brainstem). If, in addition, you believe that human intelligence and cognition takes place in the neocortex (as in the <a href="http://en.wikipedia.org/wiki/Memory-prediction_framework">Memory-prediction framework</a> model, introduced in <a href="http://www.onintelligence.org/">On Intelligence</a> by Jeff Hawkins), it doesn't make sense that intelligence is linked to consciousness, self awareness, freewill or emotion.</p>

<p>So my question is: is there any scientific evidence for this link, is there any paper about it ?</p>

CC BY-SA 3.0
2015-05-25T09:04:53.517Z
@DanD. Good point, HAL was indeed [correct regarding his secret mission](http://scifi.stackexchange.com/questions/49670/was-hal-correct). I didn't know about that.
Benjamin Crouzier
CC BY-SA 3.0
2012-03-02T01:50:03.123Z
Welcome to our site pinouchon and thanks for this interesting question! Also, [obligitory XKCD reference](http://xkcd.com/534/)...
Josh
CC BY-SA 3.0
2012-06-13T17:28:08.417Z
Here is another [XKCD](http://xkcd.com/1046/)...
draks ...
CC BY-SA 3.0
2013-07-04T06:46:07.440Z
Nether of your examples actually exhibits what you think it does. Both Skynet and HAL were acting according to their stated orders. Skynet to achieve peace, it had to eliminate humanity. No one had told it that it couldn't. And HAL had secret mission orders (See 2010) that it had to obey. These orders overrode the orders of the crew. -- But then again these examples don't have much to do with the rest of the question.
Dan D.
<intelligence>
Does super-intelligence necessary lead to consciousness, self awareness, freewill or emotion
Benjamin Crouzier
# Does super-intelligence necessary lead to consciousness, self awareness, freewill or emotion

(Asked by: username_0 on 2012-03-02)

After seeing [this talk](http://vimeo.com/34696162), the question popped in my mind. The idea is that as soon as a system is complex enough or intelligent enough, it able to act on its own. It seems to be a common belief. For example, you can find it in the movie *Terminator*, where [skynet](http://en.wikipedia.org/wiki/Skynet_%28Terminator%29), an advanced computer, becomes self aware and decide to eradicate humanity. The same is true with the [HAL](http://fr.wikipedia.org/wiki/HAL_9000) in *2001: A Space Odyssey*, where the computer decides to act despite receiving the opposite orders from the astronaut.

I believe what is said in this [ted talk](http://www.ted.com/talks/antonio_damasio_the_quest_to_understand_consciousness.html), where Antonio Damasio explains that consciousnesses is not a great product of the neocortex, but rather a trick achieved by other older parts in the brain (essentially the brainstem). If, in addition, you believe that human intelligence and cognition takes place in the neocortex (as in the [Memory-prediction framework](http://en.wikipedia.org/wiki/Memory-prediction_framework) model, introduced in [On Intelligence](http://www.onintelligence.org/) by username_6 Hawkins), it doesn't make sense that intelligence is linked to consciousness, self awareness, freewill or emotion.

So my question is: is there any scientific evidence for this link, is there any paper about it ?

**Tags:** intelligence

> **Comments:**
> **username_1**: Welcome to our site pinouchon and thanks for this interesting question! Also, [obligitory XKCD reference](http://xkcd.com/534/)...  
> **username_2**: Here is another [XKCD](http://xkcd.com/1046/)...  
> **username_3**: Nether of your examples actually exhibits what you think it does. Both Skynet and HAL were acting according to their stated orders. Skynet to achieve peace, it had to eliminate humanity. No one had told it that it couldn't. And HAL had secret mission orders (See 2010) that it had to obey. These orders overrode the orders of the crew. -- But then again these examples don't have much to do with the rest of the question.  
> **username_0**: @DanD. Good point, HAL was indeed [correct regarding his secret mission](http://scifi.stackexchange.com/questions/49670/was-hal-correct). I didn't know about that.  

---

## 2 answers

### Answer by username_4 on 2012-03-02. Score: 19

As the other response noted, a lot of the answers will depend on how you define the terms. Here, I will use Legg & Hutter's (2007) definition of intelligence, where an agent is intelligent if it can achieve a large variety of goals in a large variety of environments. For instance, a chess-playing robot is great at achieving the goal of winning at chess when it is in the environment of a chess game where nobody cheats or does anything else odd, but it fails miserably at just about everything else. So it has low intelligence. In contrast, a human can achieve many different goals in many different environments, from hunting food in a rainforest, to repairing solar panels while wearing a space suit on Earth's orbit. So a human has a considerably higher intelligence. See the paper for a formalization of this intuitive definition.

Note that a system can be exceedingly complex, or good at one thing, without necessarily being intelligent in this sense. Complexity does not imply intelligence by itself. But let's suppose that a system were intelligent in this sense? Well, let's tackle each of your terms in turn.

**Self-awareness.** One widely-used definition for self-awareness is being "aware of themselves as a differentiated and unique entity in the world". For instance, Rochat (2003) discusses the development of self-awareness in babies and very young children:

> All parents also notice an important change at around 2 years of age when children
> manifest ‘‘self-consciousness,’’ the so-called secondary emotions such as embarrassment or pride
> in very specific situations such as mirror exposure or competitive games (Kagan, 1984; Lewis,
> 1992). Prior to the second year, an infant placed in front of a mirror will typically smile, coo, and
> explore in apparent delight of the perfect contingency between acted and seen movements
> bouncing back at them from the polished surface of the mirror (Amsterdam, 1972). By 2 years, the
> specular image is associated with radically different behaviors. Toddlers become typically frozen
> and sometime behave as if they wanted to hide themselves by tucking their head in their shoulders
> or hiding their face behind their hands. They show embarrassment. This is a robust phenomenon
> and one is naturally tempted to ask what it means psychologically for children in their development. [...]
>
> It appears indeed that by 2–3
> years young children do start to have others in mind when they behave. The expression of embarrassment
> that children often begin to display in front of mirrors at around this age is the
> expression of such ‘‘self-consciousness.’’ They behave not unlike criminals hiding their face to the
> cameras. Their behavior indicates a drive to vanish from the public eyes, as if they came to grip via
> the experience of their own specular image of how they present themselves to the world. Not only
> do they discover in the mirror that it is themselves, they also realize that it is themselves as
> perceived by others. The malaise might come from the realization of a fundamental discrepancy
> between how the child represents herself from within, and how he or she is actually perceived by
> others as reflected in the mirror.

So self-awareness implies an understanding of the fact that there is such a thing as a "me", which is separate from the world as a whole, and which others will perceive differently than it will perceive itself. There can also be varying levels of self-awareness, ranging from the level 1 self-awareness that newborn infants have ("immediately after birth, infants are capable of demonstrating
already a sense of their own body as a differentiated entity: an entity among other entities in the
environment") to level 5 self-awareness involving a theory of mind and an understanding of the fact that others may have false beliefs about something while you yourself have a true belief of it.

Would a superintelligence necessarily need to be self-aware, in any of these senses? I'd say that it would. While there are lots of goals that could plausibly be solved without any kind of self-awareness (proving mathematical theorems, for instance), many seem impossible to solve without it. Pretty much any social activity requires you to have a model about others as distinct from yourself, as well as an ability to model their beliefs and attitudes towards yourself. So in order to really be capable of solving many goals, you need to be self-aware. On the other hand, if it was a narrow intelligence that had been built to *only* solve theorems of math, it probably wouldn't need self-awaraness.

**Emotion**. There are many theories of emotion. For instance, the *Information Principle* says that *emotional feelings provide conscious information from unconscious appraisals of situations* (Clore et al. 2001). Your brain is constantly appraising the situation you happen to be in. It notes things like a passerby having slightly threatening body language, or conversation with another person being easy and free of misunderstandings. There are countless of such evaluations going on all the time, and you aren't consciously aware of them, because you don't need to. Your subconscious mind can handle them just fine on its own. The end result of all those evaluations is packaged into a brief summary, which is the only thing that your conscious mind sees directly. That "executive summary" is what you experience as a particular emotional state. The passerby makes you feel slightly nervous and you avoid her, or your conversational partner feels pleasant to talk with and you begin to like him, even though you don't know why.

So this theory says that emotions are both a source of information about unconscious appraisals, and a way to bias your behavior on the basis of circumstances. For instance, in happy moods (when things are presumably going well), we tend to rely more on our pre-existing routines and heuristics, while in sad or frustrated moods (when things aren't going so well), our minds switch to a more analytical mode of thought where we're more likely to take apart our existing beliefs and routines to find whether there's something wrong with them (Schwarz, 2010).

Would a superintelligence have something that we would consider to be akin to "emotions"? I would think that it might or might not, depending on its architecture. Thinking for a moment about humans, maybe a human with a vastly larger working memory capacity wouldn't need to have the subconscious appraisals packaged into summaries in the form of emotions, but could just constantly review their un-summarized conclusions while also thinking about other things. Maybe such a human could also learn to consciously control the style of their information processing, and adapt either heuristic or analytical thought as seemed reasonable. Arguably, such a human wouldn't need emotions. So it seems plausible that a superintelligence wouldn't necessarily have them either.

**Free will.** Okay, now we are getting into tricky territory. There are lots of definitions about free will, so I'll merely say that the [computational theory of mind](http://plato.stanford.edu/entries/computational-mind/) seems to be the majority view in cognitive science. If you accept the computational theory of mind, then you also consider the human mind to be a kind of a computer executing certain algorithms. Then on most accounts, if you would consider a human to have free will, you would probably also consider a machine intelligence to have free will, and vice versa.

**Consciousness.** This is another vague term, and people mean different things by it. Copeland (1993) notes at least three possible meanings:

1. *Baseline sense*: an entity is conscious if it is capable of perceiving the world via sense organs of some sort, and if it is capable of performing inner activities such as reasoning, deliberating, judging, hypothesing, planning, and so on.
2. *Consciousness as internal monitoring*: an entity is conscious if it is capable of observing and reporting on itself. For instance, I am not simply thinking looking up Copeland's definitions and typing a summary of them: I also know that I am doing so, and can think about this fact.
3. *Phenomenological consciousness*: an entity is conscious if there's something that being that entity "feels like". Pain and pleasure are not *just* algorithmical responses that modify an entity's behavior: they *feel* painful or pleasant. This is the hardest category to really express in words, but hopefully everyone understands what's meant.

A superintelligence would most definitely need to be conscious in the baseline sense: if it couldn't collect information from the world and make plans on that basis, it wouldn't be much of an intelligence. It would probably also need be conscious in the sense of being capable of internal monitoring, for otherwise it might not really be self-aware, and we already concluded that self-awareness is necessary for (broad) intelligence.

What about phenomenological consciousness? Chalmers (1995) has a pretty convincing (to me) thought experiment, where he says that a human who was changed into a robot, replacing one neuron at a time with an artificial one, would remain phenomenally conscious. But that argument presumes that the human retains their *mental* architecture unchanged. If we built a mind that had an entirely different mental architecture as humans did, would it exhibit phenomenal consciousness? There are lots of arguments, but the essential gist is that nobody knows.

**Indepedent goal-directed behavior.** But there's one more thing that your question seems to be implying, which wasn't fully covered by any of the terms above. You seem to also be asking whether an intelligent system might *start doing things which conflict with the desires of its creators*.

The answer seems to be yes. Omohundro (2008) argues that even a chess-playing robot, if it were intelligent enough and if it were not designed carefully enough, "will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety". Suppose that the robot has "winning at chess" as its goal. Now, it will exhibit these behaviors because, if it is turned off it might not be able to play any more chess, if it can make copies of itself those copies can continue to play chess even if some were turned off, and extra resources will help it in making sure that it can always win at chess. Omohundro also mentions a number of other drives which will arise as natural subgoals of almost any other goal.

Armstrong et al. (forthcoming) have an extensive discussion about "Oracle AI", an artificial intelligence which would be built to be safe, as it was built to only answer questions and not do anything else. But even this is harder to do than it might seem. Even trying to program the Oracle AI to want to stay within a specific physical box requires a number of definitions about what constitutes its physical location and what constitutes the Oracle AI itself (is it still staying within the box if it makes remote copies of itself?), and it might plausibly discover theories of physics in which those definitions became meaningless. Note that all of this presumes that we actually understand the OAI well enough to *be able* to specify its goals with such detail. If we don't, then that plan fails right away. Probably the best existing review article about this topic is Muehlhauser & Helm (forthcoming).

**References**

Armstrong, S. & Sandberg, A. & Bostrom, N. (forthcoming) Thinking inside the box: using and
controlling an Oracle AI. <http://www.aleph.se/papers/oracleAI.pdf>

Chalmers, D.J. (1995) Absent Qualia, Fading Qualia, Dancing Qualia. In Metzinger, T. (Ed.), *Conscious Experience*. Imprint Academic. <http://consc.net/papers/qualia.html>

Clore, G.L. & Gasper, K., & Garvin, E. (2001). Affect as information. In Forgas, J.P. (Ed.), *Handbook of affect and social cognition* (pp. 121–144). Mahwah, NJ: Erlbaum.

Copeland, J. (1993) *Artificial Intelligence: A Philosophical Introduction*. Blackwell.

Legg, S. & Hutter, M. (2007) Universal Intelligence: A Definition of Machine Intelligence. *Minds & Machines*, 17:4 391-444. <http://arxiv.org/abs/0712.3329>

Muehlhauser, L. & Helm, L. (forthcoming). The Singularity and Machine Ethics. <http://commonsenseatheism.com/wp-content/uploads/2011/11/Muehlhauser-Helm-The-Singularity-and-Machine-Ethics-draft.pdf>

Omohundro, S. (2008) "The basic AI drives," in Wang, P. Goertzel, P. and Franklin,
S. (eds.), *Proceedings of the First AGI Conference. Frontiers in Articial Intelli-
gence and Applications*, Volume 171 (IOS Press) 483-494. <https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf>

Rochat, P. (2003) Five levels of self-awareness as they unfold early in life. *Consciousness and Cognition*, 12, 717-731. <http://www.psychology.emory.edu/cognition/rochat/lab/5%20levels%20of%20self-awareness.pdf>

Schwarz, N. (2010) Feelings as information. In Van Lange, P. & Kruglanski, A. & Higgins, E.T. (Eds.), Handbook of theories of social psychology, Sange. <http://people.ict.usc.edu/~gratch/CSCI534/schwarz_feelings-as-information_7jan10.pdf>

> **Comments:**
> **username_5**: Superb answer. The final part "Indepedent goal-directed behavior" just answered by question I asked in the first answer.  
> **username_6**: Excellent answer-- it's great to see people putting in the time to answer questions at a level of detail that really answers the question thoroughly and provides references for further research  
> **username_0**: Thank you for putting so much effort in your anwser (I wish I would upvote a few more times). My view is that we still don't have a clue about how the brain works, and the same is true for intelligence. [Brain science might be lacking theory](http://www.ted.com/talks/jeff_hawkins_on_how_brain_science_will_change_computing.html). But the fact that we know so little is probably what makes it so exiting.  

---

### Answer by username_7 on 2012-03-02. Score: 8

Great question. It really hits cognitive science in some tender spots, though. None of those terms you listed in the question have any sort of consensus within the field. The only one we are approaching an understanding of is probably emotion. Consciousness and free will are... tricky. I honestly don't believe there will ever be an agreed upon definition of consciousness that satisfies. It's just a term, and it does not spring from study of the brain. It's like if physics were still tied to trying to explain things in terms of the "ether". There is no physical evidence of this "ether", and so it has been abandoned. But it might be hard for people to abandon consciousness as easily, for obvious reasons. I understand the essence of what you're asking, but it's really not easy to answer. I'll try though.

The first thing to understand, is that the brain has *evolved*. This means that over a long time, tiny exceptions built up to create our brain and thus our minds. They are unreasonably messy. Speaking of any artificial system in terms of our own mental system is pretty much a mistake (at best), or deceptive (at worst). To say that the internet becomes self-aware is to go too far in anthropomorphizing it. Our minds are inherently tied into the crazy messy brains that produce them. Even our idea of free-will is not some omnipresent universal phenomena, it's just an artifact of how our brains are constructed. We make choices under certain measurable conditions, it isn't magic. For the most part, free will is only *virtually* true.

Your question assumes certain things that are really hard to articulate, but which are not exactly true. Intelligence and cognition sort of happen in the cortex, as in Hawkins model, but it's really not the full story. If you built a big HTM system like Hawkins describes, it would be inert. Consciousness, free-will, emotion, and self-awareness are not properties of the cortex, they are properties of the brain (of which the cortex is a part), and the body as a whole. We are self-aware because we have apparatus that can perceive ourselves. If you built an HTM system, and fed it weather data, it would not become self aware because it would not be perceiving itself. It would not have emotion because there would be no systems impelling the system to act in one way over another. The factors you list are related to "intelligence" as defined by cortical size, but they do not solely depend on it.

I think the fallacious assumption in your question is very related to the idea that living things exist on an evolutionary ladder. And that we are the "highest rung" on that ladder. As if we let other creatures evolve more, they would become more like us. It is a sort of deterministic belief that as we construct increasingly intelligent machines, they will become more like us. Make them smart enough, and they will begin exhibiting things that we exhibit. It's simply not true. We eventually will be able to build systems that could be considered self-aware, or having free-will, or whatever, but it will be because we intentionally design it to do so. These properties will not emerge on their own.

> **Comments:**
> **username_5**: Top answer. But can you kindly elaborate me on the final part. Just because brain is a complex structure that evolved over time, does that mean that we cannot create a structure more complex that brain(I am sure that the entire working of the brain is still not deciphered and it won't be in the near future) given the sensory capabilities and perception of knowledge by the humans is introduced into the machine. Simply, can't a AI machine be created that's intelligent than a human(not intelligent than mankind).  
> **username_5**: Also, I think (I amn't sure) that knowledge increases with   more information provided the capability to transform information into valuable knowledge with perception that could be further transformed into meaningful decisions. Can't this be replicated in machines at a minor scale (I am extending a futuristic argument and not for the near future plus I am interested in AI)  

---

