<p>This isn't my area of expertise, but it's an interesting question so I did some poking around and ran across a couple of relevant papers.</p>

<h2>Human active learning</h2>

<p><a href="http://papers.nips.cc/paper/3456-human-active-learning" rel="nofollow">http://papers.nips.cc/paper/3456-human-active-learning</a></p>

<p>Castro, R. M., Kalish, C., Nowak, R., Qian, R., Rogers, T., &amp; Zhu, X. (2009). Human active learning. In Advances in neural information processing systems (pp. 241-248).</p>

<p>In this paper, the authors compare human performance on a classification task using (a) randomly selected samples, (b) human-selected examples (i.e., "active learning"), and (c) machine-selected samples ("machine-assisted active learning"). Participants in all three conditions were tested regularly on the classification task as they accumulated examples. Somewhat surprisingly (for me -- I would have expected the human-selected condition to learn the fastest), participants in the machine-selected example case were the fastest learners, particularly when label noise was large.</p>

<h2>Category learning through active sampling</h2>

<p><a href="http://palm.mindmodeling.org/cogsci2010/papers/0041/paper0041.pdf" rel="nofollow">http://palm.mindmodeling.org/cogsci2010/papers/0041/paper0041.pdf</a></p>

<p>Markant, D., &amp; Gureckis, T. M. (2010). Category learning through active sampling. In Proceedings of the 32nd Annual Meeting of the Cognitive Science Society (pp. 248-253).</p>

<p>The task in this paper is a linear separation-boundary location task like the example you provide. It might be worth a look just to see whether it is similar to what you were describing.</p>

<p>(Also here's a more recent journal-length paper from the same authors: Markant, D. B., &amp; Gureckis, T. M. (2014). Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1), 94.)</p>

CC BY-SA 3.0
2015-07-22T05:38:55.517Z
2015-07-22T05:38:55.517Z
lmjohns3
<p>I am looking for a specific type of experimental test of active learning. Given some artificial or natural learning task that consists of classifying inputs <span class="math-container">$x$</span> from a large input space <span class="math-container">$X$</span>. There is some underlying function <span class="math-container">$f: X \rightarrow \{0,1\}$</span> to be learned, this function can come from some restricted function class <span class="math-container">$C$</span>. The subject knows the description of the function class, but has to learn the specific parameters that define the function.
I am looking for experiments that had two conditions of this type:</p>
<ol>
<li>The subject is given labeled samples <span class="math-container">$(x,f(x))$</span> from some distribution on <span class="math-container">$X$</span> that they have no control over.</li>
<li>The subject is allowed to ask for specific <span class="math-container">$x \in X$</span> and receive the labeled sample <span class="math-container">$(x,f(x))$</span>.</li>
</ol>
<p>The first condition would correspond to passive learning, while the second to active learning.</p>
<p>From arm-chair expectations (or formal theories like CoLT), one would expect that on certain tasks the subjects in the active condition would be able to significantly outperform (predict the underlying function more accurately) than in the passive condition.</p>
<p><strong>Can you provide references to specific experiments that would fit in this paradigm? What were their results? On what sort of tasks does active learning provide an advantage? How significant is this advantage?</strong></p>
<hr />
<h3>Example</h3>
<p>An example would be a subject sitting in front of a blank screen. They are told &quot;there is a straight line that divides this screen into a red and green region. Can you find it?&quot;. Thus, the set <span class="math-container">$X$</span> would correspond to coordinates, and an input <span class="math-container">$x$</span> would be a specific coordinate, say <span class="math-container">$x = (1,3)$</span>. The function class would be the set of straight line partitions of the screen. A specific function with <span class="math-container">$0$</span> meaning red and <span class="math-container">$1$</span> - green might be:</p>
<p><span class="math-container">$$f((a,b)) = \begin{cases}0 &amp; \mathrm{if} \; a &gt; b \\ 1 &amp; \mathrm{otherwise}\end{cases}$$</span></p>
<p>In the active task, the participant can click on any point <span class="math-container">$x$</span> on the screen to turn it from blank to either red (if <span class="math-container">$f(x) = 0$</span>) or green (if <span class="math-container">$f(x) = 1$</span>). In the passive trial, some number of points are selected and revealed at random to the subject.</p>

<learning><experimental-psychology>
Advantage of active learning on classification tasks
Artem Kaznatcheev
# Advantage of active learning on classification tasks

(Asked by: username_0 on 2012-07-26)

I am looking for a specific type of experimental test of active learning. Given some artificial or natural learning task that consists of classifying inputs $x$ from a large input space $X$. There is some underlying function $f: X \rightarrow \{0,1\}$ to be learned, this function can come from some restricted function class $C$. The subject knows the description of the function class, but has to learn the specific parameters that define the function.
I am looking for experiments that had two conditions of this type:

1. The subject is given labeled samples $(x,f(x))$ from some distribution on $X$ that they have no control over.
2. The subject is allowed to ask for specific $x \in X$ and receive the labeled sample $(x,f(x))$.

The first condition would correspond to passive learning, while the second to active learning.

From arm-chair expectations (or formal theories like CoLT), one would expect that on certain tasks the subjects in the active condition would be able to significantly outperform (predict the underlying function more accurately) than in the passive condition.

**Can you provide references to specific experiments that would fit in this paradigm? What were their results? On what sort of tasks does active learning provide an advantage? How significant is this advantage?**

---

### Example

An example would be a subject sitting in front of a blank screen. They are told "there is a straight line that divides this screen into a red and green region. Can you find it?". Thus, the set $X$ would correspond to coordinates, and an input $x$ would be a specific coordinate, say $x = (1,3)$. The function class would be the set of straight line partitions of the screen. A specific function with $0$ meaning red and $1$ - green might be:

$$f((a,b)) = \begin{cases}0 & \mathrm{if} \; a > b \\ 1 & \mathrm{otherwise}\end{cases}$$

In the active task, the participant can click on any point $x$ on the screen to turn it from blank to either red (if $f(x) = 0$) or green (if $f(x) = 1$). In the passive trial, some number of points are selected and revealed at random to the subject.

**Tags:** learning, experimental-psychology


---

## 1 answers

### Answer by username_1 on 2015-07-22. Score: 1

This isn't my area of expertise, but it's an interesting question so I did some poking around and ran across a couple of relevant papers.

## Human active learning

<http://papers.nips.cc/paper/3456-human-active-learning>

Castro, R. M., Kalish, C., Nowak, R., Qian, R., Rogers, T., & Zhu, X. (2009). Human active learning. In Advances in neural information processing systems (pp. 241-248).

In this paper, the authors compare human performance on a classification task using (a) randomly selected samples, (b) human-selected examples (i.e., "active learning"), and (c) machine-selected samples ("machine-assisted active learning"). Participants in all three conditions were tested regularly on the classification task as they accumulated examples. Somewhat surprisingly (for me -- I would have expected the human-selected condition to learn the fastest), participants in the machine-selected example case were the fastest learners, particularly when label noise was large.

## Category learning through active sampling

<http://palm.mindmodeling.org/cogsci2010/papers/0041/paper0041.pdf>

Markant, D., & Gureckis, T. M. (2010). Category learning through active sampling. In Proceedings of the 32nd Annual Meeting of the Cognitive Science Society (pp. 248-253).

The task in this paper is a linear separation-boundary location task like the example you provide. It might be worth a look just to see whether it is similar to what you were describing.

(Also here's a more recent journal-length paper from the same authors: Markant, D. B., & Gureckis, T. M. (2014). Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1), 94.)


---

