<p>I realize this is an old question but i was googling searching for the paper in question and found this and thought you might like it. An actual example of a paper where RCT and equilibrium disagree substantially about the efficacy of a program.</p>
<p>Denmark did a RCT i 2005 in two regions with more intensity in the active labor market policy. RCT study found a big impact. But this other study made later points out that the RCT violates SUTVA as the treatment and control group are competing for the same job openings. You could observe how people in the control group had their unemployment spell increased by the existence of the experiment by comparing them to neighboring regions with no experiment. The effect decreased substantially in size when these spill-over effects were taken into account.</p>
<p><a href="https://www.iza.org/publications/dp/6748/estimating-equilibrium-effects-of-job-search-assistance" rel="nofollow noreferrer">https://www.iza.org/publications/dp/6748/estimating-equilibrium-effects-of-job-search-assistance</a></p>

CC BY-SA 4.0
2023-11-12T19:05:07.247Z
2023-11-12T19:05:07.247Z
Søren Nørgard
<p>One of the concerns that one can have regarding the use of RCTs for evaluating policies is that while when done in small scales, the RCTs may show the result in one direction, scaling the policies may involve general equilibrium effects that may possibly make the relationship of interest nil or even the opposite of what was found in the small scale RCTs.</p>
<p>Several possible examples, intuitively, can be thought of immediately.</p>
<p>But I would like to have some real world examples where this has happened. Links to any relevant papers that demonstrate this would be helpful.</p>

CC BY-SA 4.0
2023-04-29T11:49:38.557Z
What examples do you have in mind as you have said that "Several possible examples, intuitively, can be thought of immediately"?
Amit
CC BY-SA 4.0
2023-04-29T19:23:44.993Z
Umm one example can be: we do an RCT where the treatment is a job training programme. The treatment group's income increases as a result of the treatment when it is small, but as we scale it economy-wide, trainees lose the competitive advantage they have.
Ishan Kashyap Hazarika
CC BY-SA 4.0
2023-04-29T19:27:55.913Z
Another can be, when a new farming technique when tried on a small treatment group leads to increase in yields, but when implemented in the entire region, the land quality may degrade leading to probably even a fall in the average yield
Ishan Kashyap Hazarika
CC BY-SA 4.0
2023-04-29T19:30:18.143Z
Or a sales idea that when implemented by a small treatment group leads to rise in sales but when everybody adopts it, we are back to square one
Ishan Kashyap Hazarika
CC BY-SA 4.0
2023-04-30T23:39:29.550Z
You're right - such a thing can happen. The treatment effects of a policy on a small sample can be misleading if we aim to use the results to implement the policy on a large scale. To avoid this, we need to take into account the equilibrium while measuring these effects.
Amit
CC BY-SA 4.0
2023-12-12T20:19:15.930Z
(-1)  "*Several possible examples, intuitively, can be thought of immediately.*" And then none are listed! This is poor scholarship.
Giskard
<reference-request><general-equilibrium><randomized-experiment>
Interesting example of general equilibrium effects contradicting RCT results
Ishan Kashyap Hazarika
# Interesting example of general equilibrium effects contradicting RCT results

(Asked by: username_0 on 2022-06-10)

One of the concerns that one can have regarding the use of RCTs for evaluating policies is that while when done in small scales, the RCTs may show the result in one direction, scaling the policies may involve general equilibrium effects that may possibly make the relationship of interest nil or even the opposite of what was found in the small scale RCTs.

Several possible examples, intuitively, can be thought of immediately.

But I would like to have some real world examples where this has happened. Links to any relevant papers that demonstrate this would be helpful.

**Tags:** reference-request, general-equilibrium, randomized-experiment

> **Comments:**
> **username_1**: What examples do you have in mind as you have said that "Several possible examples, intuitively, can be thought of immediately"?  
> **username_0**: Umm one example can be: we do an RCT where the treatment is a job training programme. The treatment group's income increases as a result of the treatment when it is small, but as we scale it economy-wide, trainees lose the competitive advantage they have.  
> **username_0**: Another can be, when a new farming technique when tried on a small treatment group leads to increase in yields, but when implemented in the entire region, the land quality may degrade leading to probably even a fall in the average yield  
> **username_0**: Or a sales idea that when implemented by a small treatment group leads to rise in sales but when everybody adopts it, we are back to square one  
> **username_1**: You're right - such a thing can happen. The treatment effects of a policy on a small sample can be misleading if we aim to use the results to implement the policy on a large scale. To avoid this, we need to take into account the equilibrium while measuring these effects.  
> **username_2**: (-1)  "*Several possible examples, intuitively, can be thought of immediately.*" And then none are listed! This is poor scholarship.  

---

## 1 answers

### Answer by username_3 on 2023-11-12. Score: 1

I realize this is an old question but i was googling searching for the paper in question and found this and thought you might like it. An actual example of a paper where RCT and equilibrium disagree substantially about the efficacy of a program.

Denmark did a RCT i 2005 in two regions with more intensity in the active labor market policy. RCT study found a big impact. But this other study made later points out that the RCT violates SUTVA as the treatment and control group are competing for the same job openings. You could observe how people in the control group had their unemployment spell increased by the existence of the experiment by comparing them to neighboring regions with no experiment. The effect decreased substantially in size when these spill-over effects were taken into account.

<https://www.iza.org/publications/dp/6748/estimating-equilibrium-effects-of-job-search-assistance>


---

