<p>Interesting question. I don't have any <em>empirical</em> evidence to offer, so I can't answer your question if that's what you're asking for.</p>

<p>But it's easy to provide "<em>theoretical</em> evidence", as it were, that your concerns may be well-founded, even without leaving the EU model.</p>

<p>You simply need to add a decision cost --- or a cost to figure out your true WTP --- to the decision-theoretic model of your participants. 
Then, as the EU stakes decrease (because <span class="math-container">$p$</span> decreases), participants are less likely to invest in the introspection cost of discovering their true WTP.
As a consequence, participants' behavior become a much noisier indication of their WTP: </p>

<ul>
<li>Some of your participants will decide to buy the good although their true WTP is <em>lower</em> than the price. This could happen if the participant decided not to invest in introspection and therefore got a sufficiently upward-biased signal of their true WTP. </li>
<li>Other participants will decide <em>not</em> to buy the good although their true WTP is <em>above</em>  the price. This could happen if the participant decided not to invest in introspection and as a consequence got a sufficiently downward-biased signal of their true WTP. </li>
</ul>

<p>Assuming the cost of introspection does not change with <span class="math-container">$p$</span>, participants become less likely to invest in introspection as <span class="math-container">$p$</span> decreases (because decreasing <span class="math-container">$p$</span> decreases the stakes, and therefore the benefits of introspection), and you'll get more of this noise as <span class="math-container">$p$</span> decreases.</p>

CC BY-SA 4.0
2020-05-14T14:18:11.157Z
I saw your comment "Right, I would think that this procedure assumes EU (and also zero decision-making costs) " after writing this answer. So yes, as you suggested, my point is that adding decision costs (even without dropping EU) makes the accuracy of the approach questionable. Given some budget constraint on the cost of the experiment, there's maybe a trade-off here between getting a few accurate observations (with high $p$) versus a lot of noisy observations (with low $p$).
Martin Van der Linden
CC BY-SA 4.0
2020-05-14T14:11:31.873Z
2020-05-14T14:11:31.873Z
Martin Van der Linden
<p><a href="https://link.springer.com/article/10.1007/s10683-018-9586-z" rel="nofollow noreferrer">Freeman and Mayraz (2018)</a> document that the <em>presentation of safe-vs-risky decisions in a choice list</em> has a distinct effect on people's decisions, whether or not payment is based on predeterminted or randomly selected items from that list. The abstract of their paper reads: </p>

<blockquote>
  <p>Choice lists with random incentives are widely used for preference
  elicitation. It is commonly assumed that subjects choose the same
  option in each question as they would have if it were the only
  question, but recent findings challenge this assumption. We conduct a
  large sample experiment varying incentives and presentation
  independently, and examine choices both near and away from certainty.
  We consistently find more risk taking when a choice between a safe
  prize and a risky lottery is embedded in a choice list than when it is
  presented on its own. This difference remains when we inform subjects
  of the paid choice in advance, implying that isolation fails not
  because of the random incentives scheme, but simply because the choice
  appears in a list together with others. We conjecture that subjects
  are uncertain about their preferences, reduce this uncertainty through
  considering the choices that confront them, and make cautious
  decisions in the interim. Other conditions and non-choice data support
  this interpretation. Our results open up the possibility that
  preferences inferred from choice lists offer a better indication of
  informed preferences than preferences inferred from single choices.</p>
</blockquote>

<p>In a subsequent paper, <a href="http://qeconomics.org/ojs/index.php/qe/article/viewFile/646/638" rel="nofollow noreferrer">Freeman, Halevy and Kneeland (2019)</a> provided a non-EU theory that rationalizes the finding.</p>

CC BY-SA 4.0
2020-05-16T06:49:37.007Z
2020-05-16T06:49:37.007Z
Herr K.
<p>I have read several experimental papers which ask people to make decisions which only matter with a certain probability. <a href="https://www.aeaweb.org/articles?id=10.1257/aer.97.4.1407" rel="nofollow noreferrer">For example</a>, individuals might be asked to submit bids in an auction, but told that their bids only 'count' with a certain probability <span class="math-container">$p$</span> (with probability <span class="math-container">$1 - p$</span>, their payoff is 0 regardless of what they do). Alternatively, if we wanted to learn whether an individual's valuation for a good exceeds £10, we could offer to sell her that good for £10; but tell her that the good will be sold to her (and the payment taken) only with a certain probability. In both cases, the idea is that individuals should act as they would if <span class="math-container">$p = 1$</span> (i.e. the mechanism is 'incentive compatible') which then allows researchers to run these experiments more cheaply than they would be able to otherwise.</p>

<p>Is there any evidence about the general validity of this approach? More concretely, is there evidence as to whether the outputs of such experiments vary with <span class="math-container">$p$</span>? Intuitively, I would have expected that subjects become inattentive if <span class="math-container">$p$</span> is very low (since in absolute terms, it doesn't matter much what they do), which could then add noise to the elicited estimates.</p>

CC BY-SA 4.0
2020-05-12T19:26:41.353Z
There is some evidence to the contrary. At least it seems to me that the setting in the [Allais Paradox](https://en.wikipedia.org/wiki/Allais_paradox) is similar, though not identical.
Giskard
CC BY-SA 4.0
2020-05-13T08:04:22.120Z
Right, I would think that this procedure assumes EU (and also zero decision making costs)
Itzhak Rasooly
<experimental-economics>
Does it matter if your decision might not matter?
Itzhak Rasooly
# Does it matter if your decision might not matter?

(Asked by: username_0 on 2020-05-12)

I have read several experimental papers which ask people to make decisions which only matter with a certain probability. [For example](https://www.aeaweb.org/articles?id=10.1257/aer.97.4.1407), individuals might be asked to submit bids in an auction, but told that their bids only 'count' with a certain probability $p$ (with probability $1 - p$, their payoff is 0 regardless of what they do). Alternatively, if we wanted to learn whether an individual's valuation for a good exceeds £10, we could offer to sell her that good for £10; but tell her that the good will be sold to her (and the payment taken) only with a certain probability. In both cases, the idea is that individuals should act as they would if $p = 1$ (i.e. the mechanism is 'incentive compatible') which then allows researchers to run these experiments more cheaply than they would be able to otherwise.

Is there any evidence about the general validity of this approach? More concretely, is there evidence as to whether the outputs of such experiments vary with $p$? Intuitively, I would have expected that subjects become inattentive if $p$ is very low (since in absolute terms, it doesn't matter much what they do), which could then add noise to the elicited estimates.

**Tags:** experimental-economics

> **Comments:**
> **username_1**: There is some evidence to the contrary. At least it seems to me that the setting in the [Allais Paradox](https://en.wikipedia.org/wiki/Allais_paradox) is similar, though not identical.  
> **username_0**: Right, I would think that this procedure assumes EU (and also zero decision making costs)  

---

## 2 answers

### Answer by username_2 on 2020-05-14. Score: 1

Interesting question. I don't have any *empirical* evidence to offer, so I can't answer your question if that's what you're asking for.

But it's easy to provide "*theoretical* evidence", as it were, that your concerns may be well-founded, even without leaving the EU model.

You simply need to add a decision cost --- or a cost to figure out your true WTP --- to the decision-theoretic model of your participants.
Then, as the EU stakes decrease (because $p$ decreases), participants are less likely to invest in the introspection cost of discovering their true WTP.
As a consequence, participants' behavior become a much noisier indication of their WTP:

* Some of your participants will decide to buy the good although their true WTP is *lower* than the price. This could happen if the participant decided not to invest in introspection and therefore got a sufficiently upward-biased signal of their true WTP.
* Other participants will decide *not* to buy the good although their true WTP is *above* the price. This could happen if the participant decided not to invest in introspection and as a consequence got a sufficiently downward-biased signal of their true WTP.

Assuming the cost of introspection does not change with $p$, participants become less likely to invest in introspection as $p$ decreases (because decreasing $p$ decreases the stakes, and therefore the benefits of introspection), and you'll get more of this noise as $p$ decreases.

> **Comments:**
> **username_2**: I saw your comment "Right, I would think that this procedure assumes EU (and also zero decision-making costs) " after writing this answer. So yes, as you suggested, my point is that adding decision costs (even without dropping EU) makes the accuracy of the approach questionable. Given some budget constraint on the cost of the experiment, there's maybe a trade-off here between getting a few accurate observations (with high $p$) versus a lot of noisy observations (with low $p$).  

---

### Answer by username_3 on 2020-05-16. Score: 0

[Freeman and Mayraz (2018)](https://link.springer.com/article/10.1007/s10683-018-9586-z) document that the *presentation of safe-vs-risky decisions in a choice list* has a distinct effect on people's decisions, whether or not payment is based on predeterminted or randomly selected items from that list. The abstract of their paper reads:

> Choice lists with random incentives are widely used for preference
> elicitation. It is commonly assumed that subjects choose the same
> option in each question as they would have if it were the only
> question, but recent findings challenge this assumption. We conduct a
> large sample experiment varying incentives and presentation
> independently, and examine choices both near and away from certainty.
> We consistently find more risk taking when a choice between a safe
> prize and a risky lottery is embedded in a choice list than when it is
> presented on its own. This difference remains when we inform subjects
> of the paid choice in advance, implying that isolation fails not
> because of the random incentives scheme, but simply because the choice
> appears in a list together with others. We conjecture that subjects
> are uncertain about their preferences, reduce this uncertainty through
> considering the choices that confront them, and make cautious
> decisions in the interim. Other conditions and non-choice data support
> this interpretation. Our results open up the possibility that
> preferences inferred from choice lists offer a better indication of
> informed preferences than preferences inferred from single choices.

In a subsequent paper, [Freeman, Halevy and Kneeland (2019)](http://qeconomics.org/ojs/index.php/qe/article/viewFile/646/638) provided a non-EU theory that rationalizes the finding.


---

