<p><a href="https://en.wikipedia.org/wiki/Fisher%27s_exact_test" rel="nofollow noreferrer">Fishers's exact test</a> exists for this purpose which calculates the p-value from a contingency table. In this case the null hypothesis is that <span class="math-container">$o_{1}$</span> and <span class="math-container">$o_{2}$</span> are equally likely to be labelled with the word. As noted by @Bryan it may also be possible to use a <a href="https://en.wikipedia.org/wiki/Test_statistic#Common_test_statistics" rel="nofollow noreferrer">two-proportion z-test</a>, but this would require a larger dataset.</p>

CC BY-SA 4.0
2020-04-27T09:15:01.960Z
2020-04-27T09:15:01.960Z
A. Bollans
<p>You could possibly use a two-proportion z-test to test against the null hypothesis that <span class="math-container">$(x_{1}/n_{1}-x_{2}/n_{2})==0$</span>, though there may be better methods that consider the overall distribution of words chosen and accounting for differences in raters in terms of how many words they choose (your situation does not fit the assumptions of the test but may be 'close enough' for certain purposes).</p>

CC BY-SA 4.0
2020-04-26T13:34:29.907Z
Thanks! This looks along the right lines for what I want but I don't think I can fulfil the sampling assumptions with the data I have, as described [here](https://stattrek.com/hypothesis-test/difference-in-proportions.aspx). I think I need a way to calculate the p-value more directly
A. Bollans
CC BY-SA 4.0
2020-04-26T14:12:48.037Z
Aha I think I need to use [Fisher's exact test](https://en.wikipedia.org/wiki/Fisher%27s_exact_test), in the example replacing 'Men' and 'Women' with $o1$ and $o2$ and 'studying' and 'non-studying' to 'labelled' and 'not labelled'.
A. Bollans
CC BY-SA 4.0
2020-04-24T16:40:10.587Z
2020-04-24T16:40:10.587Z
Bryan Krause
<p>I have some data from a study where humans are shown an object and asked to select all words from a list that apply to the object. What I am trying to do is use the data to confirm that a particular object, <span class="math-container">$o_{1}$</span>, is a better fit of a particular word, <span class="math-container">$w$</span> , than another object, <span class="math-container">$o_{2}$</span>. I am trying to use hypothesis testing to confirm this, with a null hypothesis that <span class="math-container">$o_{2}$</span> is an equivalent or better fit than <span class="math-container">$o_{1}$</span> i.e. the probability of labelling <span class="math-container">$o_{2}$</span> with <span class="math-container">$w$</span> is greater than or equal to the probability of labelling <span class="math-container">$o_{1}$</span> with <span class="math-container">$w$</span>.</p>

<p>There are <span class="math-container">$n_{1}$</span> annotations for <span class="math-container">$o_{1}$</span>, in which the object is labelled <span class="math-container">$x_{1}$</span> times with <span class="math-container">$w$</span>. Similarly <span class="math-container">$o_{2}$</span> is labelled <span class="math-container">$x_{2}$</span> times in <span class="math-container">$n_{2}$</span> annotations.</p>

<p>To calculate the p-value I am working out the probability of observing the difference in the proportion of times <span class="math-container">$o_{1}$</span> and <span class="math-container">$o_{2}$</span> are labelled with <span class="math-container">$w$</span>, <span class="math-container">$P(x_{1}/n_{1}-x_{2}/n_{2}\geq d)$</span>, where d is the observed difference. This gives quite a long function, which relies on the actual underlying probability, <span class="math-container">$p_{1}$</span>, of labelling <span class="math-container">$o_{1}$</span> with <span class="math-container">$w$</span>, so now I suppose I have to find <span class="math-container">$p_{1}$</span> which maximises this function?</p>

<p>I feel uncertain that I'm tackling this in the right way, does this seem correct?</p>

<statistics><data><experiment-design><linguistics>
How to confirm that a more frequently labelled object is a better instance of the label
A. Bollans
# How to confirm that a more frequently labelled object is a better instance of the label

(Asked by: username_0 on 2020-04-24)

I have some data from a study where humans are shown an object and asked to select all words from a list that apply to the object. What I am trying to do is use the data to confirm that a particular object, $o\_{1}$, is a better fit of a particular word, $w$ , than another object, $o\_{2}$. I am trying to use hypothesis testing to confirm this, with a null hypothesis that $o\_{2}$ is an equivalent or better fit than $o\_{1}$ i.e. the probability of labelling $o\_{2}$ with $w$ is greater than or equal to the probability of labelling $o\_{1}$ with $w$.

There are $n\_{1}$ annotations for $o\_{1}$, in which the object is labelled $x\_{1}$ times with $w$. Similarly $o\_{2}$ is labelled $x\_{2}$ times in $n\_{2}$ annotations.

To calculate the p-value I am working out the probability of observing the difference in the proportion of times $o\_{1}$ and $o\_{2}$ are labelled with $w$, $P(x\_{1}/n\_{1}-x\_{2}/n\_{2}\geq d)$, where d is the observed difference. This gives quite a long function, which relies on the actual underlying probability, $p\_{1}$, of labelling $o\_{1}$ with $w$, so now I suppose I have to find $p\_{1}$ which maximises this function?

I feel uncertain that I'm tackling this in the right way, does this seem correct?

**Tags:** statistics, data, experiment-design, linguistics


---

## 2 answers

### Answer by username_0 on 2020-04-27. Score: 0 | Accepted answer

[Fishers's exact test](https://en.wikipedia.org/wiki/Fisher%27s_exact_test) exists for this purpose which calculates the p-value from a contingency table. In this case the null hypothesis is that $o\_{1}$ and $o\_{2}$ are equally likely to be labelled with the word. As noted by @Bryan it may also be possible to use a [two-proportion z-test](https://en.wikipedia.org/wiki/Test_statistic#Common_test_statistics), but this would require a larger dataset.


---

### Answer by username_1 on 2020-04-24. Score: 0

You could possibly use a two-proportion z-test to test against the null hypothesis that $(x\_{1}/n\_{1}-x\_{2}/n\_{2})==0$, though there may be better methods that consider the overall distribution of words chosen and accounting for differences in raters in terms of how many words they choose (your situation does not fit the assumptions of the test but may be 'close enough' for certain purposes).

> **Comments:**
> **username_0**: Thanks! This looks along the right lines for what I want but I don't think I can fulfil the sampling assumptions with the data I have, as described [here](https://stattrek.com/hypothesis-test/difference-in-proportions.aspx). I think I need a way to calculate the p-value more directly  
> **username_0**: Aha I think I need to use [Fisher's exact test](https://en.wikipedia.org/wiki/Fisher%27s_exact_test), in the example replacing 'Men' and 'Women' with $o1$ and $o2$ and 'studying' and 'non-studying' to 'labelled' and 'not labelled'.  

---

