<p>More than two years later, I now have a heuristic (which is a close as I can get to an objective metric) that I rely on. As I write about in <a href="https://medium.com/@seanaubin/why-does-ai-still-suck-and-can-it-suck-less-9db36be294dc#.1l42uxoei" rel="nofollow noreferrer">one of my blog posts</a>, there are a couple of tasks that humans are capable of that are still really hard for machines. They require the following things for a machine (or a neural model):</p>

<ul>
<li><strong>Balancing Planning and Exploration:</strong> Modelling the environment's future state and the possible future states via exploration efficiently.</li>
<li><strong>Scaling Skills:</strong> Transfering skills between tasks and building more complex skills atop of basic skills.</li>
<li><strong>Knowledge Representation:</strong> Keeping relations between skills, environmental variables, past knowledge and priorities all usefully organised.</li>
</ul>

<p>Any neural model that begins to leverage these things is evaluated as impressive in my eyes.</p>

<p>However, these requirements are mostly for Artificial Intelligence. What about neural models specifically? All neural models that complete a task are, by definition, cognitive models, so in addition to the above criteria they must satisfy various <a href="https://cogsci.stackexchange.com/q/6247/4397">Cognitive Criteria</a> to prove the system is achieving it's goals in a human-like manner.</p>

CC BY-SA 3.0
2016-05-23T18:03:52.900Z
2016-05-23T22:08:49.013Z
2017-04-13T12:56:16.467Z
Seanny123
<p>When I first saw this video of Spaun and the tasks it can complete (solving the Towers of Hanoi problem, completing the Raven matrices), I was really impressed, but then I realized I didn't really know what other tasks were being completed by other neural architectures. Consequently, I almost asked a question here that was basically "How does Spaun compare in terms of task complexity to other neural architectures?". However, I realized that it would be better to if I could learn how to do this evaluation myself.</p>

<p>So, what metrics or heuristics can be used to evaluate the complexity of a task completed by a neural architecture? How can one evaluate the complexity of a problem/puzzle such at the Tower of Hanoi, to another problem, such as modelling how people solve algebraic equations, like ACT-R has accomplished? </p>

CC BY-SA 3.0
2014-04-14T07:12:15.060Z
This is a nice question. One approach is to go as you are proposing from the direction of task difficulty, a dual idea would be to look at the complexity of the neural networks that solve the tasks. For that you might find [this CogSci question](http://cogsci.stackexchange.com/q/252/29) and [this cstheory question](http://cstheory.stackexchange.com/q/2698/1037) useful.
Artem Kaznatcheev
<theoretical-neuroscience><computational-modeling>
How to compare tasks completed by neural architectures objectively?
Seanny123
# How to compare tasks completed by neural architectures objectively?

(Asked by: username_0 on 2014-04-14)

When I first saw this video of Spaun and the tasks it can complete (solving the Towers of Hanoi problem, completing the Raven matrices), I was really impressed, but then I realized I didn't really know what other tasks were being completed by other neural architectures. Consequently, I almost asked a question here that was basically "How does Spaun compare in terms of task complexity to other neural architectures?". However, I realized that it would be better to if I could learn how to do this evaluation myself.

So, what metrics or heuristics can be used to evaluate the complexity of a task completed by a neural architecture? How can one evaluate the complexity of a problem/puzzle such at the Tower of Hanoi, to another problem, such as modelling how people solve algebraic equations, like ACT-R has accomplished?

**Tags:** theoretical-neuroscience, computational-modeling

> **Comments:**
> **username_1**: This is a nice question. One approach is to go as you are proposing from the direction of task difficulty, a dual idea would be to look at the complexity of the neural networks that solve the tasks. For that you might find [this CogSci question](http://cogsci.stackexchange.com/q/252/29) and [this cstheory question](http://cstheory.stackexchange.com/q/2698/1037) useful.  

---

## 1 answers

### Answer by username_0 on 2016-05-23. Score: 1

More than two years later, I now have a heuristic (which is a close as I can get to an objective metric) that I rely on. As I write about in [one of my blog posts](https://medium.com/@seanaubin/why-does-ai-still-suck-and-can-it-suck-less-9db36be294dc#.1l42uxoei), there are a couple of tasks that humans are capable of that are still really hard for machines. They require the following things for a machine (or a neural model):

* **Balancing Planning and Exploration:** Modelling the environment's future state and the possible future states via exploration efficiently.
* **Scaling Skills:** Transfering skills between tasks and building more complex skills atop of basic skills.
* **Knowledge Representation:** Keeping relations between skills, environmental variables, past knowledge and priorities all usefully organised.

Any neural model that begins to leverage these things is evaluated as impressive in my eyes.

However, these requirements are mostly for Artificial Intelligence. What about neural models specifically? All neural models that complete a task are, by definition, cognitive models, so in addition to the above criteria they must satisfy various [Cognitive Criteria](https://cogsci.stackexchange.com/q/6247/4397) to prove the system is achieving it's goals in a human-like manner.


---

