<h1>Some things omitted</h1>

<ol>
<li><p>Although the criteria addresses learning new tasks in Adaptibility, it doesn't address the ability to switch between tasks easily.</p></li>
<li><p>Although Triangulation does address matching data, it doesn't discuss grounding the data in the real world vs. data fed from a dataset. In other words, it doesn't address the quality of the data matched.</p></li>
<li><p>Compactness doesn't address how easy anything is to explain or grasp. This might actually be wise, as understanding is highly dependent on tools available to the person grasping at comprehension.</p></li>
</ol>

<h1>Are there more complete criterion?</h1>

<p>The criterion provided above provide a high-level view of the goals that should be sought after by any cognitive system, however this does not address <em>how</em> a system should pursue this goal.</p>

<p>For a more complete evaluation of the methods and tools that should be used, please see Terry Stewart's PHD thesis <a href="https://curve.carleton.ca/system/files/theses/27919.pdf" rel="nofollow noreferrer">A Methodology for Computational Cognitive Modelling</a> which outlines a set of statistical methods and <a href="https://github.com/tcstewar/ccmsuite" rel="nofollow noreferrer">tools</a> for cognitive scientists (even without a significant statistical background) to use.</p>

CC BY-SA 4.0
2015-04-22T13:51:38.783Z
2019-03-08T01:31:21.177Z
2019-03-08T01:31:21.177Z
Seanny123
<p>Hofstadter provides a detailed set of criteria in <a href="https://en.wikipedia.org/wiki/Fluid_Concepts_and_Creative_Analogies" rel="nofollow">Fluid Concepts and Creative Analogies.</a>  Some of his criteria that I do not see mentioned above concern <em>the flow of information-processing</em>:  whether or not the model evaluated <em>flows through possibility space in a psychologically plausible way</em>, and how to measure that.</p>

<p>For instance...  Suppose the following: "A bat and a ball cost usd110. The bat costs usd100 more than the ball. How much is the ball?" </p>

<p>Now, cognitive psychology has shown that people are primed to say <a href="https://en.wikipedia.org/wiki/Cognitive_Reflection_Test" rel="nofollow">"the ball costs usd10."</a> Which is wrong! A good cognitive model should also "think" usd10, before considering usd5. Otherwise, it doesn't respect information-processing flow.</p>

<p>He also mentions, metaphorically, the idea of including "lesions" in the model, such that, if some component of the model is "lesioned", the expected outcome from that component should drastically change.  </p>

CC BY-SA 3.0
2014-07-14T05:33:58.847Z
Info flow is an interesting addition, though (without reading your reference or the OP's) I wonder how it differs from compositionality and systematicity. Lesions seem covered by robustness.
Nick Stauner
CC BY-SA 3.0
2014-07-14T15:51:46.383Z
I agree with Nick, would you mind elaborating and including an example of *flow of information-processing*. Namely what a good system and what a bad system would do.
Seanny123
CC BY-SA 3.0
2014-07-15T01:54:21.763Z
Sure.  Suppose the following:  "A bat and a ball cost usd110.  The bat costs usd100 more than the ball.  How much is the ball?"  


Now, cognitive psychology has shown that people are primed to say "the ball costs usd10."  Which is wrong. 



A good cognitive model should *also* "think" usd10, before considering usd5.  Otherwise, it doesn't respect information-processing flow.
linhares
CC BY-SA 3.0
2014-07-15T14:00:48.783Z
That seems to fall under "triangulation"? I think I'm mis-understanding here. Could you describe how they're distinct?
Seanny123
CC BY-SA 3.0
2014-07-15T19:28:26.457Z
Perhaps, but triangulation seems too broad based to me.  It doesn't include the info processing pathways explicitly, which is something I care very deeply about.  So, while I would place information-processing flow on its own header, I see how one could throw it into triangulation just fine.
linhares
CC BY-SA 3.0
2014-07-14T05:23:35.527Z
2015-04-24T08:04:35.040Z
2015-04-24T08:04:35.040Z
linhares
<p>In the first chapter of the book "How to Build a Brain", Chris Eliasmith quickly establishes some criteria which he will use to evaluate Spaun, the cognitive system described in the book. He describes them in great detail in Chapter 8, but below is my attempt at summarizing them.</p>

<hr>

<h1>1. Representational Structure</h1>

<h2>Systematicity</h2>

<p>The system should connect concepts in an inter-connected way and understand there are classes of concepts, since human thought is systematic. For example, given <code>blue ball</code> and <code>red square</code>, it should be possible for the system to reason about the concept of <code>blue square</code>.</p>

<h2>Compositionality</h2>

<p>Concepts should be created off of the combination of concepts to the degree supported by real cognitive systems. For example, given the concepts of <code>pet</code> and <code>fish</code>, it should be possible to reason about the novel concept <code>pet fish</code>.</p>

<h2>Productivity</h2>

<p>The system should be able to create many representation based off of a few basic representations to the degree supported by real cognitive systems.</p>

<h2><a href="https://en.wikipedia.org/wiki/Binding_problem" rel="nofollow noreferrer">The Massive Binding Problem</a></h2>

<p>The system should identify a binding operation that scales well.</p>

<h1>2. Performance Concerns</h1>

<h2>Syntactic generalization</h2>

<p>Ability to exploit the structure of language regardless of content.</p>

<h2>Robustness</h2>

<p>Losing a few neurons or being exposed to noise shouldn't break the system.</p>

<h2>Adaptability</h2>

<p>You should be able to use a single system for multiple task.</p>

<h2>Memory</h2>

<p>The system should be able to show the relation to the various types of memory (working, long-term).</p>

<h2>Scalability</h2>

<p>Large portions of the brain should be modeled and able to complete a wide variety of tasks.</p>

<h1>3. Scientific merit</h1>

<h2>Triangulation</h2>

<p>Contact with the most sources of experimental data as possible.</p>

<h2>Compactness</h2>

<p>Good theories can be stated compactly and without ad-hoc additions.</p>

<hr>

<p>He explains in the chapter that this is actually a synthesis of various other criteria proposed by various authors. Although the scientific merit seems rather straightforward to me and has been elaborated upon on great length, the other criteria I am less certain about in terms of completeness and acceptance. Have attempts at unified criteria been made before? Are there more complete criteria? Are these criteria at odds with any other ideas about what a cognitive system should embody?</p>

<reference-request><theoretical-neuroscience><computational-modeling>
Criteria for evaluating cognitive systems
Seanny123
# Criteria for evaluating cognitive systems

(Asked by: username_0 on 2014-04-16)

In the first chapter of the book "How to Build a Brain", Chris Eliasmith quickly establishes some criteria which he will use to evaluate Spaun, the cognitive system described in the book. He describes them in great detail in Chapter 8, but below is my attempt at summarizing them.

---

# 1. Representational Structure

## Systematicity

The system should connect concepts in an inter-connected way and understand there are classes of concepts, since human thought is systematic. For example, given `blue ball` and `red square`, it should be possible for the system to reason about the concept of `blue square`.

## Compositionality

Concepts should be created off of the combination of concepts to the degree supported by real cognitive systems. For example, given the concepts of `pet` and `fish`, it should be possible to reason about the novel concept `pet fish`.

## Productivity

The system should be able to create many representation based off of a few basic representations to the degree supported by real cognitive systems.

## [The Massive Binding Problem](https://en.wikipedia.org/wiki/Binding_problem)

The system should identify a binding operation that scales well.

# 2. Performance Concerns

## Syntactic generalization

Ability to exploit the structure of language regardless of content.

## Robustness

Losing a few neurons or being exposed to noise shouldn't break the system.

## Adaptability

You should be able to use a single system for multiple task.

## Memory

The system should be able to show the relation to the various types of memory (working, long-term).

## Scalability

Large portions of the brain should be modeled and able to complete a wide variety of tasks.

# 3. Scientific merit

## Triangulation

Contact with the most sources of experimental data as possible.

## Compactness

Good theories can be stated compactly and without ad-hoc additions.

---

He explains in the chapter that this is actually a synthesis of various other criteria proposed by various authors. Although the scientific merit seems rather straightforward to me and has been elaborated upon on great length, the other criteria I am less certain about in terms of completeness and acceptance. Have attempts at unified criteria been made before? Are there more complete criteria? Are these criteria at odds with any other ideas about what a cognitive system should embody?

**Tags:** reference-request, theoretical-neuroscience, computational-modeling


---

## 2 answers

### Answer by username_0 on 2015-04-22. Score: 3 | Accepted answer

# Some things omitted

1. Although the criteria addresses learning new tasks in Adaptibility, it doesn't address the ability to switch between tasks easily.
2. Although Triangulation does address matching data, it doesn't discuss grounding the data in the real world vs. data fed from a dataset. In other words, it doesn't address the quality of the data matched.
3. Compactness doesn't address how easy anything is to explain or grasp. This might actually be wise, as understanding is highly dependent on tools available to the person grasping at comprehension.

# Are there more complete criterion?

The criterion provided above provide a high-level view of the goals that should be sought after by any cognitive system, however this does not address *how* a system should pursue this goal.

For a more complete evaluation of the methods and tools that should be used, please see Terry Stewart's PHD thesis [A Methodology for Computational Cognitive Modelling](https://curve.carleton.ca/system/files/theses/27919.pdf) which outlines a set of statistical methods and [tools](https://github.com/tcstewar/ccmsuite) for cognitive scientists (even without a significant statistical background) to use.


---

### Answer by username_1 on 2014-07-14. Score: 4

Hofstadter provides a detailed set of criteria in [Fluid Concepts and Creative Analogies.](https://en.wikipedia.org/wiki/Fluid_Concepts_and_Creative_Analogies) Some of his criteria that I do not see mentioned above concern *the flow of information-processing*: whether or not the model evaluated *flows through possibility space in a psychologically plausible way*, and how to measure that.

For instance... Suppose the following: "A bat and a ball cost usd110. The bat costs usd100 more than the ball. How much is the ball?"

Now, cognitive psychology has shown that people are primed to say ["the ball costs usd10."](https://en.wikipedia.org/wiki/Cognitive_Reflection_Test) Which is wrong! A good cognitive model should also "think" usd10, before considering usd5. Otherwise, it doesn't respect information-processing flow.

He also mentions, metaphorically, the idea of including "lesions" in the model, such that, if some component of the model is "lesioned", the expected outcome from that component should drastically change.

> **Comments:**
> **username_2**: Info flow is an interesting addition, though (without reading your reference or the OP's) I wonder how it differs from compositionality and systematicity. Lesions seem covered by robustness.  
> **username_0**: I agree with Nick, would you mind elaborating and including an example of *flow of information-processing*. Namely what a good system and what a bad system would do.  
> **username_1**: Sure.  Suppose the following:  "A bat and a ball cost usd110.  The bat costs usd100 more than the ball.  How much is the ball?"     Now, cognitive psychology has shown that people are primed to say "the ball costs usd10."  Which is wrong.     A good cognitive model should *also* "think" usd10, before considering usd5.  Otherwise, it doesn't respect information-processing flow.  
> **username_0**: That seems to fall under "triangulation"? I think I'm mis-understanding here. Could you describe how they're distinct?  
> **username_1**: Perhaps, but triangulation seems too broad based to me.  It doesn't include the info processing pathways explicitly, which is something I care very deeply about.  So, while I would place information-processing flow on its own header, I see how one could throw it into triangulation just fine.  

---

