<p>The biggest drawback of your procedure, also point of discussion <a href="https://cogsci.stackexchange.com/questions/17147/at-what-time-in-a-psychophysical-test-are-catch-trials-preferably-presented/17154#17154">previously here</a> is the <strong>limited number of trials</strong>. Apparently, you obtained binary data, namely one answer (yes/no) per trial, basically leaving you with a <strong>sparsely sampled, binary data set</strong>. Binary data will <strong>not allow a curve fitting</strong>, which is the conventional way to determine a threshold.</p>

<p>To make matters worse, there is a certain bias in the catch trials. But values anywhere below 25% false positive are, imo fine. So let's forget that issue, as it is likely the least of your problems.</p>

<p>Now, single-stimulus per trial data are referred to as the <strong><a href="https://psychologydictionary.org/method-of-limits/" rel="nofollow noreferrer">Method of Limits</a></strong>. Basically what you would normally do to obtain a threshold with MoL is increase the level to see where the subjects starts perceiving the stimulus (<strong>YES/NO reversal</strong>) and then flip the sequence and see where the person stops answering YES and hits NO. Generally, such a procedure is experimentally used to get a <strong>quick handle of a good starting point for the 'real deal' experiments</strong>. Or, it is useful in clinical environments. For example, <a href="https://medlineplus.gov/ency/article/003341.htm" rel="nofollow noreferrer">clinical audiometric thresholds</a> are often determined this way - quick and dirty, but accurate by about 5 dB or so.</p>

<p>The response you get in this case is a so-called type 2 response, since you never can tell whether the subject was right or not. You just evaluate when the subject reports to perceive the stimulus. The problem with the method described here is that the S may become habituated (<em>I didn't perceive it now, likely I won't in the next trial</em>) or the S makes the error of expectation (<em>It will probably become apparent now</em>). </p>

<ul>
<li>The only way in which I can see to analyze your data is <strong>determine the flipping point</strong> and take that as a measure. You can then, perhaps take the average between the two intensities where the response flips to get an estimate of the 50% level, the gold standard of the threshold in yes/no tasks.<br>
Likely you will not have an unambiguous flipping point in your data, but I'm sure you can think of something. Perhaps just take the lowest intensity where a flip occurred, or something along those lines.      </li>
<li>Alternatively, you may group your intensities into approximate levels and average out the responses to get values including rates between 0 and 100%. This procedure, however, is likely pretty much doomed to be shot to pieces by any referee or peer in your field, the more since your step sizes are unequal. </li>
</ul>

<p><sub><strong>Reference</strong><br>
<strong>-</strong> <a href="https://www.elsevier.com/books/psychophysics/kingdom/978-0-12-407156-8" rel="nofollow noreferrer">Kingdom &amp; Prins. <em>Psychophysics</em>. Elsevier 2010</a></sub> </p>

CC BY-SA 3.0
2017-08-15T13:42:55.357Z
We work around the habituation by having the levels not be monotonically decreasing such that sometimes a lower level signal is followed by a higher level signal. At the population level, when we sort by trial, we see big jumps in the probability of yes that reflect the changes in the signal level, but when we sort by signal level, we get a nice monotonic psychometric function.
StrongBad
CC BY-SA 3.0
2017-08-15T13:44:14.710Z
While threshold in the MoL is often estimated by taking the last level for a yes response, that doesn't make it the best way to do it. At the individual level, while for some subjects there is a single level above which they always responds *yes* and below which they always respond *no*, for many subjects that is not the case.
StrongBad
CC BY-SA 3.0
2017-08-15T13:54:40.580Z
@StrongBad, I understand that it may not be a sharp tipping point, as  included in my answer. My post is simply the best I can come up with after mulling it over. It's certainly not pretty, but neither is your data :)
AliceD
CC BY-SA 3.0
2017-08-15T13:56:05.007Z
Pooling the data of the population is always an option, but even less pretty imo. It defies the purpose of determining thresholds.
AliceD
CC BY-SA 3.0
2017-08-15T13:59:20.143Z
We pool the data to show differences between our two groups (control and test). We now want to look at individual subjects to see if the test has diagnostic value.
StrongBad
CC BY-SA 3.0
2017-08-15T13:11:26.690Z
2017-08-15T13:22:05.530Z
2017-08-15T13:22:05.530Z
AliceD
<p>For various reasons, I have a crappy experimental design that I am forced to deal with. I present subjects with 20 yes/no trials, out of which 5 are no-signal catch trials and 15 are signal trials. Each signal presentation has a unique level with unequal spacing (as small as one in arbitrary units and as great as 5). All subjects run the trials in the same order (i.e., the 5<sup>th</sup> trial is always at signal level, and the 11<sup>th</sup> and the 6<sup>th</sup> trial are always catch trials).</p>

<p>To make things more complicated, at the population level, and likely at the individual level, the criterion is not stable such that the probability of saying yes on a no-signal trial depends on the catch trial number (the 1<sup>st</sup> catch trial has a probability of yes of about 0.05 and the 5<sup>th</sup> catch trial has a probability of yes of about 0.2). From this mess, I need a single number representing <em>threshold</em>. I don't really care what the threshold corresponds to (<em>e.g.</em>, the signal level leading to 50% yes, 50% correct, or a d' of 1 would all be fine). Ideally, the method would provide a confidence interval on the estimate at the individual subject level.</p>

<p>I was thinking of using a <a href="http://www.cureffi.org/2015/09/20/the-math-behind-spearman-karber-analysis/" rel="nofollow noreferrer">Spearman-Karber</a> type approach, but the unequal step sizes seems problematic (along with the variable criteria). I also don't understand the history of Spearman-Karber (i.e., where was it first published and how it relates to <a href="http://onlinelibrary.wiley.com/doi/10.1111/j.2044-8295.1908.tb00176.x/abstract" rel="nofollow noreferrer">Spearman 1908</a>) and kind of hope that maybe in the past 100 years we have improved on things.</p>

CC BY-SA 3.0
2017-08-14T18:29:34.840Z
wow. Good luck :-) Are you using an adaptive algorithm? How different are the step sizes? Do you have multiple measures per step? Could you show us a data set?
AliceD
CC BY-SA 3.0
2017-08-14T18:45:04.327Z
@AliceD everyone runs the trials in the same order (i.e., not adaptive) and there is only a single trial at each signal level. The step sizes vary from 1 to 5.
StrongBad
CC BY-SA 3.0
2017-08-14T19:49:51.940Z
Then it is a MOC procedure with binary data (say 0 (no) or 1 (yes)? Can't you fit a Weibul or other psychometric function? Perhaps that is indeed difficult with binary data. Let me know. A picture of a (imaginary) data set would be nice  still to visualize your problem
AliceD
CC BY-SA 3.0
2017-08-14T19:52:44.907Z
@AliceD yes, but with only a single trial per level I am not sure that is the way to go. Additionally, I am not sure how to get a CI from the fitted function with only a single trial per level.
StrongBad
CC BY-SA 3.0
2017-08-14T19:55:36.863Z
I have always gathered enough repeats per level to estimate a correct rate. Interesting problem. I will ponder on this. +1'd already. Should be interesting for a lot of people.
AliceD
CC BY-SA 3.0
2017-08-14T22:28:56.190Z
There are several things I do not understand, if you are working in the field of the Signal Detection Theory with an nAFC n-alternative forced-choice method, there should not be in it estimation bias (memory biases may develop) you should not use Spearman-Karber, you should start with a double entry table for signal and detection with yes / no for both, develop the psychometric function and work on probability and variance, it should not matter so much the answer but the sensitivity
hexadecimal
CC BY-SA 3.0
2017-08-14T23:08:35.630Z
@hexadecimal with only one entry per level, the table is odd in that it only has entries of either 0 or 1. Fitting a psychometric function to binary data also seems problematic and with only 15 signal trials I am worried about the fit being over determined. Also, sensitivity, as I think about it, depends on a stable criteria, which I don't have.
StrongBad
CC BY-SA 3.0
2017-08-14T23:26:52.870Z
@StrongBad I put some links: http://www.cis.rit.edu/people/faculty/montag/vandplite/pages/chap_5/ch5p1.html https://weldre4.instructure.com/courses/28/pages/we-experience-our-world-through-sensation http://brain.mcmaster.ca/SDT/sdt2.html Only one entry per level? You should have the data I commented (in each trial when the signal was presented or not and that response to given person) the number of tests seems reduced but you could make an estimate.
hexadecimal
CC BY-SA 3.0
2017-08-15T15:02:10.680Z
:) Don't fully understand whats going on, but have you considered applying [IRT](https://en.wikipedia.org/wiki/Item_response_theory)?
mflo-ByeSE
CC BY-SA 3.0
2017-08-15T17:51:04.523Z
@StrongBad If you answered this question could help: Why habituation (learning) was introduced on a sensitivity estimation paradigm (perception)? It is my main doubt and it should clarify how to analyze the data
hexadecimal
<methodology><psychophysics><signal-detection><analysis>
Calculate threshold from 'binary' yes/no data
StrongBad
# Calculate threshold from 'binary' yes/no data

(Asked by: username_0 on 2017-08-14)

For various reasons, I have a crappy experimental design that I am forced to deal with. I present subjects with 20 yes/no trials, out of which 5 are no-signal catch trials and 15 are signal trials. Each signal presentation has a unique level with unequal spacing (as small as one in arbitrary units and as great as 5). All subjects run the trials in the same order (i.e., the 5th trial is always at signal level, and the 11th and the 6th trial are always catch trials).

To make things more complicated, at the population level, and likely at the individual level, the criterion is not stable such that the probability of saying yes on a no-signal trial depends on the catch trial number (the 1st catch trial has a probability of yes of about 0.05 and the 5th catch trial has a probability of yes of about 0.2). From this mess, I need a single number representing *threshold*. I don't really care what the threshold corresponds to (*e.g.*, the signal level leading to 50% yes, 50% correct, or a d' of 1 would all be fine). Ideally, the method would provide a confidence interval on the estimate at the individual subject level.

I was thinking of using a [Spearman-Karber](http://www.cureffi.org/2015/09/20/the-math-behind-spearman-karber-analysis/) type approach, but the unequal step sizes seems problematic (along with the variable criteria). I also don't understand the history of Spearman-Karber (i.e., where was it first published and how it relates to [Spearman 1908](http://onlinelibrary.wiley.com/doi/10.1111/j.2044-8295.1908.tb00176.x/abstract)) and kind of hope that maybe in the past 100 years we have improved on things.

**Tags:** methodology, psychophysics, signal-detection, analysis

> **Comments:**
> **username_1**: wow. Good luck :-) Are you using an adaptive algorithm? How different are the step sizes? Do you have multiple measures per step? Could you show us a data set?  
> **username_0**: @username_1 everyone runs the trials in the same order (i.e., not adaptive) and there is only a single trial at each signal level. The step sizes vary from 1 to 5.  
> **username_1**: Then it is a MOC procedure with binary data (say 0 (no) or 1 (yes)? Can't you fit a Weibul or other psychometric function? Perhaps that is indeed difficult with binary data. Let me know. A picture of a (imaginary) data set would be nice  still to visualize your problem  
> **username_0**: @username_1 yes, but with only a single trial per level I am not sure that is the way to go. Additionally, I am not sure how to get a CI from the fitted function with only a single trial per level.  
> **username_1**: I have always gathered enough repeats per level to estimate a correct rate. Interesting problem. I will ponder on this. +1'd already. Should be interesting for a lot of people.  
> **username_2**: There are several things I do not understand, if you are working in the field of the Signal Detection Theory with an nAFC n-alternative forced-choice method, there should not be in it estimation bias (memory biases may develop) you should not use Spearman-Karber, you should start with a double entry table for signal and detection with yes / no for both, develop the psychometric function and work on probability and variance, it should not matter so much the answer but the sensitivity  
> **username_0**: @username_2 with only one entry per level, the table is odd in that it only has entries of either 0 or 1. Fitting a psychometric function to binary data also seems problematic and with only 15 signal trials I am worried about the fit being over determined. Also, sensitivity, as I think about it, depends on a stable criteria, which I don't have.  
> **username_2**: @username_0 I put some links: http://www.cis.rit.edu/people/faculty/montag/vandplite/pages/chap_5/ch5p1.html https://weldre4.instructure.com/courses/28/pages/we-experience-our-world-through-sensation http://brain.mcmaster.ca/SDT/sdt2.html Only one entry per level? You should have the data I commented (in each trial when the signal was presented or not and that response to given person) the number of tests seems reduced but you could make an estimate.  
> **username_3**: :) Don't fully understand whats going on, but have you considered applying [IRT](https://en.wikipedia.org/wiki/Item_response_theory)?  
> **username_2**: @username_0 If you answered this question could help: Why habituation (learning) was introduced on a sensitivity estimation paradigm (perception)? It is my main doubt and it should clarify how to analyze the data  

---

## 1 answers

### Answer by username_1 on 2017-08-15. Score: 3

The biggest drawback of your procedure, also point of discussion [previously here](https://cogsci.stackexchange.com/questions/17147/at-what-time-in-a-psychophysical-test-are-catch-trials-preferably-presented/17154#17154) is the **limited number of trials**. Apparently, you obtained binary data, namely one answer (yes/no) per trial, basically leaving you with a **sparsely sampled, binary data set**. Binary data will **not allow a curve fitting**, which is the conventional way to determine a threshold.

To make matters worse, there is a certain bias in the catch trials. But values anywhere below 25% false positive are, imo fine. So let's forget that issue, as it is likely the least of your problems.

Now, single-stimulus per trial data are referred to as the **[Method of Limits](https://psychologydictionary.org/method-of-limits/)**. Basically what you would normally do to obtain a threshold with MoL is increase the level to see where the subjects starts perceiving the stimulus (**YES/NO reversal**) and then flip the sequence and see where the person stops answering YES and hits NO. Generally, such a procedure is experimentally used to get a **quick handle of a good starting point for the 'real deal' experiments**. Or, it is useful in clinical environments. For example, [clinical audiometric thresholds](https://medlineplus.gov/ency/article/003341.htm) are often determined this way - quick and dirty, but accurate by about 5 dB or so.

The response you get in this case is a so-called type 2 response, since you never can tell whether the subject was right or not. You just evaluate when the subject reports to perceive the stimulus. The problem with the method described here is that the S may become habituated (*I didn't perceive it now, likely I won't in the next trial*) or the S makes the error of expectation (*It will probably become apparent now*).

* The only way in which I can see to analyze your data is **determine the flipping point** and take that as a measure. You can then, perhaps take the average between the two intensities where the response flips to get an estimate of the 50% level, the gold standard of the threshold in yes/no tasks.  
  Likely you will not have an unambiguous flipping point in your data, but I'm sure you can think of something. Perhaps just take the lowest intensity where a flip occurred, or something along those lines.
* Alternatively, you may group your intensities into approximate levels and average out the responses to get values including rates between 0 and 100%. This procedure, however, is likely pretty much doomed to be shot to pieces by any referee or peer in your field, the more since your step sizes are unequal.

**Reference**  
**-** [Kingdom & Prins. *Psychophysics*. Elsevier 2010](https://www.elsevier.com/books/psychophysics/kingdom/978-0-12-407156-8)

> **Comments:**
> **username_0**: We work around the habituation by having the levels not be monotonically decreasing such that sometimes a lower level signal is followed by a higher level signal. At the population level, when we sort by trial, we see big jumps in the probability of yes that reflect the changes in the signal level, but when we sort by signal level, we get a nice monotonic psychometric function.  
> **username_0**: While threshold in the MoL is often estimated by taking the last level for a yes response, that doesn't make it the best way to do it. At the individual level, while for some subjects there is a single level above which they always responds *yes* and below which they always respond *no*, for many subjects that is not the case.  
> **username_1**: @username_0, I understand that it may not be a sharp tipping point, as  included in my answer. My post is simply the best I can come up with after mulling it over. It's certainly not pretty, but neither is your data :)  
> **username_1**: Pooling the data of the population is always an option, but even less pretty imo. It defies the purpose of determining thresholds.  
> **username_0**: We pool the data to show differences between our two groups (control and test). We now want to look at individual subjects to see if the test has diagnostic value.  

---

