<p>I haven't heard Fineman's speech, but from your description it seem that the problem he is getting at is that just because a certain finding has been made in one experiment, it is not a universal truth. It is quite possible that, when you try to replicate the findings, you will not be able to obtain them again. Jeromy already said something along these lines when he mentioned meta-analysis. </p>

<p>Fineman goes so far as to say, that the only way really to be sure is to conduct the experiment yourself. In other words, when you want to know the difference between treatment X and treatment Y, you have to design an experiment with both of these treatments yourself rather than rely on findings from someone else.</p>

CC BY-SA 3.0
2013-12-11T23:03:07.893Z
2013-12-11T23:03:07.893Z
Jens Kouros
<p>I think it's a basic concept of experimental design that you have control groups. So for example, if you want to study the effect of X (e.g., a drug) on Y (e.g., performance), you could manipulate X (e.g., give half the drug and the other half no drug) and randomly assign the participants to either group.
<a href="http://en.wikipedia.org/wiki/Random_assignment">Random assignment</a> is a strategy for making the groups on average equivalent. Thus, it is an attempt to control for confounding factors that might cause Y.</p>

<p>Of course, you could say that other researchers have conducted the equivalent of the control group condition, and just do a one sample study with just the drug condition. However, studies differ in all sorts of ways (participants differ, your exact measures might differ, the context might differ, etc.). Thus, you couldn't unambiguously attribute any changes to the effect of the drug.</p>

<p>That said, we often do compare the experimental outcome of different studies that differ in some respect. For example, meta-analyses often examine moderators which compare effect sizes across study features. While this is useful, you'll often get a cleaner answer to your research question by incorporating the moderator into a specific study where there are fewer alternative sources of explanation.</p>

CC BY-SA 3.0
2013-12-11T04:56:37.523Z
2013-12-11T04:56:37.523Z
Jeromy Anglim
<p>In Richard Feynman's classic <a href="http://neurotheory.columbia.edu/%7Eken/cargo_cult.html" rel="noreferrer">Cargo Cult Science speech</a>. He talks about a type of scientific error made, where a person takes a previous experimental result for granted when designing an experiment.</p>
<blockquote>
<p>Other kinds of errors are more characteristic of poor science. When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A.</p>
<p>So her proposal was to do the experiment under circumstances Y and see if they still did A. I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the the real difference was the thing she thought she had under control.</p>
</blockquote>
<p>I'm not sure I understand what sort of fallacy is being committed in this instance. What exactly is the problem?</p>
<p>Thanks.</p>

CC BY-SA 3.0
2013-12-12T11:43:11.347Z
Also see _pathological science_
Dan D.
<methodology><experimental-psychology>
Why did Richard Feynman appear to recommend replication before manipulation when conducting lab experiments?
klinger
# Why did Richard Feynman appear to recommend replication before manipulation when conducting lab experiments?

(Asked by: username_0 on 2013-12-11)

In Richard Feynman's classic [Cargo Cult Science speech](http://neurotheory.columbia.edu/%7Eken/cargo_cult.html). He talks about a type of scientific error made, where a person takes a previous experimental result for granted when designing an experiment.

> Other kinds of errors are more characteristic of poor science. When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A.
>
> So her proposal was to do the experiment under circumstances Y and see if they still did A. I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the the real difference was the thing she thought she had under control.

I'm not sure I understand what sort of fallacy is being committed in this instance. What exactly is the problem?

Thanks.

**Tags:** methodology, experimental-psychology

> **Comments:**
> **username_1**: Also see _pathological science_  

---

## 2 answers

### Answer by username_3 on 2013-12-11. Score: 8 | Accepted answer

I think it's a basic concept of experimental design that you have control groups. So for example, if you want to study the effect of X (e.g., a drug) on Y (e.g., performance), you could manipulate X (e.g., give half the drug and the other half no drug) and randomly assign the participants to either group.
[Random assignment](http://en.wikipedia.org/wiki/Random_assignment) is a strategy for making the groups on average equivalent. Thus, it is an attempt to control for confounding factors that might cause Y.

Of course, you could say that other researchers have conducted the equivalent of the control group condition, and just do a one sample study with just the drug condition. However, studies differ in all sorts of ways (participants differ, your exact measures might differ, the context might differ, etc.). Thus, you couldn't unambiguously attribute any changes to the effect of the drug.

That said, we often do compare the experimental outcome of different studies that differ in some respect. For example, meta-analyses often examine moderators which compare effect sizes across study features. While this is useful, you'll often get a cleaner answer to your research question by incorporating the moderator into a specific study where there are fewer alternative sources of explanation.


---

### Answer by username_2 on 2013-12-11. Score: 2

I haven't heard Fineman's speech, but from your description it seem that the problem he is getting at is that just because a certain finding has been made in one experiment, it is not a universal truth. It is quite possible that, when you try to replicate the findings, you will not be able to obtain them again. Jeromy already said something along these lines when he mentioned meta-analysis.

Fineman goes so far as to say, that the only way really to be sure is to conduct the experiment yourself. In other words, when you want to know the difference between treatment X and treatment Y, you have to design an experiment with both of these treatments yourself rather than rely on findings from someone else.


---

