<p>The study design sounds pretty good. Some of the good things you are proposing:</p>

<ul>
<li>Using a <strong>repeated measures design</strong> will give you more statistical power than a between subjects design, which is particularly useful when your sample size is small. </li>
<li>Randomising or <strong>counterbalancing for order</strong> should mostly control for order effects.</li>
<li><strong>Double blind</strong> will focus the test on taste rather than branding and as you say limit experimenter effects</li>
</ul>

<p>A few further suggestions:</p>

<ul>
<li>You could provide a little bread and water between the two tastings or separate the tastings by some longer period of time to limit any residual taste effects from the first taste.</li>
<li>And yes, as you say, increasing sample size will increase your statistical power. That said, if your sample is your population (e.g., you have an office of 10 people), then the smaller sample size doesn't matter.</li>
<li>You should consider how the response scale you will use for rating taste. You could simple get a binary preference. But I think it might be better to use some form of scale that rates preference in addition to an explicit question at the end about preference. </li>
</ul>

CC BY-SA 3.0
2014-05-11T04:27:50.447Z
I hadn't thought to make them clean their palate before trying the second cup.  Would salty pretzels work?  Not only do we have a near infinite supply of pretzels, but I feel the salt would contrast well with the coffee.
Soviero
CC BY-SA 3.0
2014-05-12T13:32:13.453Z
The salt might also temper the taste of the second cup, so it may be best to keep the flavours bland and neutral between tastings.
Adb
CC BY-SA 3.0
2014-05-11T01:36:53.340Z
2014-05-11T02:19:27.943Z
2014-05-11T02:19:27.943Z
Jeromy Anglim
<p>Imagine a population of people in an office environment.  There is a small war going on about whether <em>this</em> brand of coffee machine is better than <em>that</em> one.</p>

<p>How would you go about testing which coffee maker is truly most preferred?</p>

<p>My current plan is to take 10 people out of that population and make them each drink two cups of coffee, one made by each machine.  People will be given the choice of preparation styles for their individual trial, black, milk, sugar, or both.  Both cups will be made in the exact same way for that individual trial.  Whether a given cup will have coffee from machine "A" or "B" will be randomized to avoid people's preference for their first taste being taken into account.  Neither the participants, nor the administrator of the test will know which cup was made with which machine, to avoid unconscious biases.  Lastly, each participant will be independent of each other to avoid a bias forming from everyone else choosing a particular cup.  (Though the randomized nature of the study should make this a moot issue, but still.)</p>

<p>Is there anything I'm forgetting to make this a more perfect study?  I would love to have a larger sample than 10, but alas, that's not an option.  Also, I'm worried that allowing people to choose how their coffee will be prepared will skew the results, but as long as both cups within a trial are made the same way, it shouldn't be a problem, right?</p>

CC BY-SA 3.0
2014-05-11T01:38:44.010Z
Allowing people to choose their style of coffee preparation would improve generalizability but might increase variance...or it might decrease it, since people's preferences might be a little more systematic within their area of familiarity.
Nick Stauner
CC BY-SA 3.0
2014-06-07T16:14:40.457Z
Beware of the "Pepi Challenge" effect.  Preferences reported during single tastings of small quantities vary greatly from preferences reported from continuous consumption over a period of weeks.  IIRC, short taste tests tend to bias responses in favor of sweeter and more sour flavors, for instance.  You may wish to give people a cup of coffee every morning over a period of days, if such a design is feasible.
Louis Thibault
<methodology><consumer-psychology>
How to design an experiment to measure coffee preferences?
Soviero
# How to design an experiment to measure coffee preferences?

(Asked by: username_0 on 2014-05-10)

Imagine a population of people in an office environment. There is a small war going on about whether *this* brand of coffee machine is better than *that* one.

How would you go about testing which coffee maker is truly most preferred?

My current plan is to take 10 people out of that population and make them each drink two cups of coffee, one made by each machine. People will be given the choice of preparation styles for their individual trial, black, milk, sugar, or both. Both cups will be made in the exact same way for that individual trial. Whether a given cup will have coffee from machine "A" or "B" will be randomized to avoid people's preference for their first taste being taken into account. Neither the participants, nor the administrator of the test will know which cup was made with which machine, to avoid unconscious biases. Lastly, each participant will be independent of each other to avoid a bias forming from everyone else choosing a particular cup. (Though the randomized nature of the study should make this a moot issue, but still.)

Is there anything I'm forgetting to make this a more perfect study? I would love to have a larger sample than 10, but alas, that's not an option. Also, I'm worried that allowing people to choose how their coffee will be prepared will skew the results, but as long as both cups within a trial are made the same way, it shouldn't be a problem, right?

**Tags:** methodology, consumer-psychology

> **Comments:**
> **username_1**: Allowing people to choose their style of coffee preparation would improve generalizability but might increase variance...or it might decrease it, since people's preferences might be a little more systematic within their area of familiarity.  
> **username_2**: Beware of the "Pepi Challenge" effect.  Preferences reported during single tastings of small quantities vary greatly from preferences reported from continuous consumption over a period of weeks.  IIRC, short taste tests tend to bias responses in favor of sweeter and more sour flavors, for instance.  You may wish to give people a cup of coffee every morning over a period of days, if such a design is feasible.  

---

## 1 answers

### Answer by username_3 on 2014-05-11. Score: 7

The study design sounds pretty good. Some of the good things you are proposing:

* Using a **repeated measures design** will give you more statistical power than a between subjects design, which is particularly useful when your sample size is small.
* Randomising or **counterbalancing for order** should mostly control for order effects.
* **Double blind** will focus the test on taste rather than branding and as you say limit experimenter effects

A few further suggestions:

* You could provide a little bread and water between the two tastings or separate the tastings by some longer period of time to limit any residual taste effects from the first taste.
* And yes, as you say, increasing sample size will increase your statistical power. That said, if your sample is your population (e.g., you have an office of 10 people), then the smaller sample size doesn't matter.
* You should consider how the response scale you will use for rating taste. You could simple get a binary preference. But I think it might be better to use some form of scale that rates preference in addition to an explicit question at the end about preference.

> **Comments:**
> **username_0**: I hadn't thought to make them clean their palate before trying the second cup.  Would salty pretzels work?  Not only do we have a near infinite supply of pretzels, but I feel the salt would contrast well with the coffee.  
> **username_4**: The salt might also temper the taste of the second cup, so it may be best to keep the flavours bland and neutral between tastings.  

---

