<p>There's a related saying in statistics:</p>

<blockquote>
  <p>you cannot do inference without making assumptions.</p>
</blockquote>

<p>From <a href="http://www.inference.phy.cam.ac.uk/mackay/itila/" rel="nofollow">Information Theory, Inference, and Learning Algorithms</a> (page 26) by David J. C. Mackay.</p>

<p>Inference about the world is learning, and if you don't assume anything about the world, you can't learn to generalize. Assumptions limit your hypothesis space.</p>

CC BY-SA 3.0
2015-09-04T14:20:21.187Z
2015-09-04T14:20:21.187Z
Memming
<p>If that suffices, I can give you the classical article from the domain of language learning: <a href="http://web.mit.edu/6.863/www/spring2010/readings/gold67limit.pdf" rel="nofollow">Gold 1967</a>.</p>

<p>The basic intuition is that an infinite number of grammars could explain any given set of strings. Analogously, you can probably consider fitting a polynomial to a set of points, or instances of the inverse problem (reconstructing a source from observations). In language acquisition, without some prior constraint on the to be acquired grammar, therefore, no unique convergence will happen. This has led to the Poverty of the Stimulus class of arguments, and the wide range of responses to it. There is a wide range of responses to the Gold paper - if you want more, it really depends on what specifically you're looking for.</p>

<p>In practice, I don't think anyone can imagine a learner in the real world that is without constraints, first and foremost those of storage and computational power.</p>

CC BY-SA 3.0
2015-08-31T18:04:05.657Z
This is definitely along the lines of what I was looking for, and is very helpful.
Josh
CC BY-SA 3.0
2015-08-31T19:34:09.417Z
Glad I could help. If you specify your question, I'll specify my answer.
jona
CC BY-SA 3.0
2015-08-31T16:01:48.203Z
2015-08-31T16:01:48.203Z
jona
<p>I've read theoretical papers on why constraints are a necessary part of any learning system. The papers were aimed at human learning. I'm having trouble finding relevant references now. </p>

<p>What papers deal with the theoretical issue of whether or not learning must be constrained?</p>

CC BY-SA 3.0
2015-08-28T00:24:02.437Z
Constrained in what way?
Seanny123
CC BY-SA 3.0
2015-08-28T02:07:32.453Z
In any way. The basic argument is that it is impossible to have learning unless there is some limit to what can be learned, if I remember correctly.
Josh
CC BY-SA 3.0
2015-08-28T03:03:55.720Z
By 'constraints', are you referring to inter-subject (i.e. Calculus) constraints or constraints on learning as a whole?
Sydney Maples
CC BY-SA 3.0
2015-08-28T12:45:39.353Z
Learning as a whole. It's a very broad, general argument. I just can't remember where I found it!
Josh
<learning><reference-request>
Reference request: the necessity of constraints in learning
Josh
# Reference request: the necessity of constraints in learning

(Asked by: username_0 on 2015-08-27)

I've read theoretical papers on why constraints are a necessary part of any learning system. The papers were aimed at human learning. I'm having trouble finding relevant references now.

What papers deal with the theoretical issue of whether or not learning must be constrained?

**Tags:** learning, reference-request

> **Comments:**
> **username_1**: Constrained in what way?  
> **username_0**: In any way. The basic argument is that it is impossible to have learning unless there is some limit to what can be learned, if I remember correctly.  
> **username_2**: By 'constraints', are you referring to inter-subject (i.e. Calculus) constraints or constraints on learning as a whole?  
> **username_0**: Learning as a whole. It's a very broad, general argument. I just can't remember where I found it!  

---

## 2 answers

### Answer by username_4 on 2015-08-31. Score: 4 | Accepted answer

If that suffices, I can give you the classical article from the domain of language learning: [Gold 1967](http://web.mit.edu/6.863/www/spring2010/readings/gold67limit.pdf).

The basic intuition is that an infinite number of grammars could explain any given set of strings. Analogously, you can probably consider fitting a polynomial to a set of points, or instances of the inverse problem (reconstructing a source from observations). In language acquisition, without some prior constraint on the to be acquired grammar, therefore, no unique convergence will happen. This has led to the Poverty of the Stimulus class of arguments, and the wide range of responses to it. There is a wide range of responses to the Gold paper - if you want more, it really depends on what specifically you're looking for.

In practice, I don't think anyone can imagine a learner in the real world that is without constraints, first and foremost those of storage and computational power.

> **Comments:**
> **username_0**: This is definitely along the lines of what I was looking for, and is very helpful.  
> **username_4**: Glad I could help. If you specify your question, I'll specify my answer.  

---

### Answer by username_3 on 2015-09-04. Score: 1

There's a related saying in statistics:

> you cannot do inference without making assumptions.

From [Information Theory, Inference, and Learning Algorithms](http://www.inference.phy.cam.ac.uk/mackay/itila/) (page 26) by David J. C. Mackay.

Inference about the world is learning, and if you don't assume anything about the world, you can't learn to generalize. Assumptions limit your hypothesis space.


---

