<p>Humans are, as you would expect, much noisier than physical systems. As such, replication of experiments in psychology is very important. Replication provides evidence that the original results were not spurious.</p>

<p>As pointed out in the comments, replication <em>with the same subject pool</em> is not always the way to go. Many tasks have learning effects, and so you would not expect to achieve the same results by re-administering the experiment.</p>

<p>There is a vast statistical literature on reliability testing. A good place to start would be at the Wikipedia article for <a href="http://en.wikipedia.org/wiki/Reliability_%28psychometrics%29">Reliability (psychometrics)</a></p>

CC BY-SA 3.0
2012-11-19T04:01:00.123Z
If I understand this correctly, there are two types of experiments then: the one shot ones, and those focused on learning, right?
Alex Stone
CC BY-SA 3.0
2012-11-19T04:36:02.830Z
I suppose you could say that, but no one actually does. It's just as arbitrary as saying "There are two types of experiments: those that study the effects of aging on memory, and those that don't." Sure it's true, but it's meaningless.
Jeff
CC BY-SA 3.0
2012-11-19T00:32:14.680Z
2012-11-19T00:32:14.680Z
Jeff
<p>Thanks to this website, I've seen a number of papers and scholarly articles that deal with cognition. I'm interested in how repeatable the findings are that are discovered as a result of experiments.</p>

<p>For example, <strong>when a mechanical system is tested</strong>, it is tested to work the same way every single time:</p>

<ul>
<li>A lock opens and closes when a key is turned.</li>
<li>A car door opens up to the maximum degree allowed by design.</li>
</ul>

<p><strong>How much is a cognitive system like a mechanical system?</strong> Will it work the same way every time?</p>

<p>For example, I'm thinking about the design of an experiment I've been reading about: <a href="http://www.hbs.edu/faculty/Pages/download.aspx?name=11-091.pdf">The Ikea Effect Experiment</a>. Volunteers are asked to assemble Ikea boxes and bid on the box after assembly. The experiment shows that people value things they built themselves more than things that are just offered to them.</p>

<p>Now if the machine-like definition of repeatability is applied to this experiment: Would test subjects value boxes they built themselves next time they do an experiment? How about next month? What about in 3 years? Would people in the 1940's have the same response?</p>

<p>Have there been any studies that deal with this "repeatability" question? For example, <strong>studies that used the same test subjects doing the same experiment multiple times?</strong></p>

CC BY-SA 3.0
2012-11-18T01:10:03.487Z
When dealing with humans, you always [have to take into account learning effects in experimental design](http://www.yorku.ca/mack/RN-Counterbalancing.html). The main difference with the type of machines you are referring to is those machines don't have a memory. Humans do, and subsequent tests will always be influenced (in predictable and unpredictable ways) by previous experiments.
Steven Jeuris
CC BY-SA 3.0
2012-11-18T09:24:07.410Z
I think that this question is very broad and could be improved if you specified which area you are interested in. As Steven pointed out running the same experiment twice in human subjects is done when effects of learning are investigated. If no learning effects are expected and performance is hypothesized to be stable, researchers often investigate 'retest-reliability'. So if you google/pubmed the area you are intetested in (e.g. memory) plus 'learning' or 'retest-reliability' you will find lots of studies.
jokel
CC BY-SA 3.0
2012-11-18T19:30:48.747Z
The area that I was originally interested in is the "stimulus-response" or "sequence of tasks performed - result". I don't know what the correct word for this, but it's not memory. Is it behavior?
Alex Stone
<cognitive-psychology><reference-request>
How repeatable are cognitive science findings within the same individual?
Alex Stone
# How repeatable are cognitive science findings within the same individual?

(Asked by: username_0 on 2012-11-17)

Thanks to this website, I've seen a number of papers and scholarly articles that deal with cognition. I'm interested in how repeatable the findings are that are discovered as a result of experiments.

For example, **when a mechanical system is tested**, it is tested to work the same way every single time:

* A lock opens and closes when a key is turned.
* A car door opens up to the maximum degree allowed by design.

**How much is a cognitive system like a mechanical system?** Will it work the same way every time?

For example, I'm thinking about the design of an experiment I've been reading about: [The Ikea Effect Experiment](http://www.hbs.edu/faculty/Pages/download.aspx?name=11-091.pdf). Volunteers are asked to assemble Ikea boxes and bid on the box after assembly. The experiment shows that people value things they built themselves more than things that are just offered to them.

Now if the machine-like definition of repeatability is applied to this experiment: Would test subjects value boxes they built themselves next time they do an experiment? How about next month? What about in 3 years? Would people in the 1940's have the same response?

Have there been any studies that deal with this "repeatability" question? For example, **studies that used the same test subjects doing the same experiment multiple times?**

**Tags:** cognitive-psychology, reference-request

> **Comments:**
> **username_1**: When dealing with humans, you always [have to take into account learning effects in experimental design](http://www.yorku.ca/mack/RN-Counterbalancing.html). The main difference with the type of machines you are referring to is those machines don't have a memory. Humans do, and subsequent tests will always be influenced (in predictable and unpredictable ways) by previous experiments.  
> **username_2**: I think that this question is very broad and could be improved if you specified which area you are interested in. As Steven pointed out running the same experiment twice in human subjects is done when effects of learning are investigated. If no learning effects are expected and performance is hypothesized to be stable, researchers often investigate 'retest-reliability'. So if you google/pubmed the area you are intetested in (e.g. memory) plus 'learning' or 'retest-reliability' you will find lots of studies.  
> **username_0**: The area that I was originally interested in is the "stimulus-response" or "sequence of tasks performed - result". I don't know what the correct word for this, but it's not memory. Is it behavior?  

---

## 1 answers

### Answer by username_3 on 2012-11-19. Score: 5 | Accepted answer

Humans are, as you would expect, much noisier than physical systems. As such, replication of experiments in psychology is very important. Replication provides evidence that the original results were not spurious.

As pointed out in the comments, replication *with the same subject pool* is not always the way to go. Many tasks have learning effects, and so you would not expect to achieve the same results by re-administering the experiment.

There is a vast statistical literature on reliability testing. A good place to start would be at the Wikipedia article for [Reliability (psychometrics)](http://en.wikipedia.org/wiki/Reliability_%28psychometrics%29)

> **Comments:**
> **username_0**: If I understand this correctly, there are two types of experiments then: the one shot ones, and those focused on learning, right?  
> **username_3**: I suppose you could say that, but no one actually does. It's just as arbitrary as saying "There are two types of experiments: those that study the effects of aging on memory, and those that don't." Sure it's true, but it's meaningless.  

---

