<p>I like this idea! I think the task should be one where the data could be used by many analytic tools. Hence, the data would not only be beneficial for one particular question, for example parameter estimations, but also for other tools such as Systems Factorial Technology (see Townsend &amp; Nozawa, 1995), which provides a deeper understanding of the underlying cognitive sub-processes architecture, and other interesting information such as the process' capacity.</p>

<p>I suggest that the task at least respect a double factorial paradigm. This design is very simple: you have four conditions in a 2 x 2 design, where you vary two variables. It could be any variable, the only restriction is that you have to vary them independently one from another. In other words, each variable should reflect a cognitive sub-process. In one condition, both variables are in scenario where the sub-process should work very effectively (or optimally). In an other condition, both variables are in a scenario where the sub-process should work less effectively (or sub-optimally). And in the two remaining conditions, one variable works optimally and the other sub-optimally, and vice-versa.</p>

<p>As for the very details of the task, such as the stimuli that will be used and the specific variables, I do not have preferences. Letters are very manipulable, but they do carry semantic (more or less). Abstract stimuli could work. We could play with color, shapes, complexity, etc. At Psychonomics this year, they were many experiments which have use this design.</p>

<p>Reference for citation:
Townsend, J. T. &amp; Nozawa, G. (1995). Spatio-temporal properties of elementation perception an investigation of parallel, serial, and coactive theories. Journal of Mathematical Psychology, 39(4), 321–359. doi: 10.1006/jmps.1995.1033</p>

CC BY-SA 3.0
2017-02-08T06:40:19.133Z
Could you provide more detailed citations, preferably full APA. That will make it easier for readers to find the papers.
Robin Kramer-ten Have
CC BY-SA 3.0
2017-02-01T19:43:42.863Z
2017-02-08T16:34:42.457Z
2017-02-08T16:34:42.457Z
Marc-André Goulet
<p>Just a few comments...</p>

<p>Since one of the advantages of a large data set is high statistical power, it might be a good idea to use a task where the key effect has <em>not</em> been found in previous studies.  That would produce a guaranteed result: i.e., a high-power test of an important null result.</p>

<p>A pop-out condition in visual search would be one example (i.e., no effect of display size), but this is not ideal because it would limit the modelling end of the project.  That is, it wouldn't be too helpful to model multiple conditions with identical performance (or nearly so).</p>

<p>A better example would be an additive-factor experiment (somewhat as suggested by Marc-André Goulet).   I would suggest a simple 2x2 design for which previous studies had reported theoretically important factor additivity (i.e., non-significant interaction).  A big study would provide a high-power test of the null interaction.  And, as long as both main effects were present, there would be four distinct conditions to model.  This design is also interesting because a lot is known about conditions that should be satisfied by the RT distributions under certain models (for examples, see the references below).</p>

<p>A final point: After selecting a task, it might be very useful to test out the planned analyses on simulated data, which are considerably easier to get than real data. This would help make the project's precise goals more explicit and also make sure the goals would be achievable using the proposed data set.  It might also help to reveal what aspects of the planned task/conditions were necessary for getting informative results (e.g., minimum effect sizes).</p>

<p>References</p>

<p>Cortese, J. M. &amp; Dzhafarov, E. N. Empirical recovery of response time decomposition rules II. Discriminability of serial and parallel architectures.  Journal of Mathematical Psychology, 1996 , 40 , 203-218 </p>

<p>Dzhafarov, E. N. &amp; Cortese, J. M.  Empirical recovery of response time decomposition rules I. Sample-level decomposition tests. Journal of Mathematical Psychology, 1996 , 40 , 185-202</p>

<p>Dzhafarov, E. N. &amp; Schweickert, R.  Decompositions of response times: An almost general theory.  Journal of Mathematical Psychology, 1995 , 39 , 285-314</p>

<p>Roberts, S. &amp; Sternberg, S. The meaning of additive reaction-time effects: Tests of three alternatives.  In Meyer, D. E. &amp; Kornblum, S. (Eds.) Attention and performance XIV. Synergies in experimental psychology, artificial intelligence, and cognitive neuroscience., MIT Press, 1992 , 611-653 </p>

CC BY-SA 3.0
2017-02-12T00:22:37.263Z
2017-02-12T00:22:37.263Z
Jeff Miller
<p>I agree with Marc-André, the selected stimuli should have no semantic association (or as little as possible) which is why I would encourage the use of geometric shapes that vary in colour. Additionally, I believe that the task should be as simple as possible in order to encourage the overall cleanliness of the data. </p>

<p>As you suggested, one such task is the redundant attribute target detection task (Miller, 1982). In this task, participants are asked to keep in mind target dimensions (e.g. blue and square) and as soon as one of the dimensions is presented on screen, the participants respond as quickly and as accurately as possible. Results are typically fast, accurate, and slightly skewed – appropriate for this initiative! We could select a few colours and shapes (if we do go forward with this task) as well as what colour harmony should be selected. </p>

<p>Additionally, having colours that vary in “obviousness” can potentially give us a clearer understanding of the parameters within certain models. For example, an obvious blue should have quicker RT than a muted blue and the rate parameter of each individual channel should reflect that (assuming, of course, that the rate parameter is associated with "obviousness" at all). I am particularly interested in this aspect.</p>

<p>While a go/no would be great for this task, I also strongly believe that a 2AFC task would yield much more interesting results and the 1million RT initiative could help us really understand the “race” in “race models”. This could be as simple as modifying the task as one with “target present” or “target absent” decisions. With clearer trends in RT we could hopefully gather information on the independence of dimensions and whether there is crosstalk between detectors.</p>

<p>Additionally, having such data would not only benefit model fitting but also more solid architecture analyses. This would, in turn, help us get a more concrete understanding of coactive architectures (the architecture that Miller, 1982 suggested was at play in this type of task) which haven’t been as rigorously defined like other architectures (as noted by Houpt &amp; Townsend, 2011).</p>

<p>EDIT:</p>

<p>Houpt, J.W., &amp; Townsend, J.T. (2011). An extension of SIC predictions to the Wiener coactive model. Journal of Mathematical Psychology, 55, 267-270</p>

<p>Miller J. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14,247–279</p>

CC BY-SA 3.0
2017-02-08T06:39:55.060Z
Could you provide more detailed citations, preferably full APA. That will make it easier for readers to find the papers.
Robin Kramer-ten Have
CC BY-SA 3.0
2017-02-02T21:02:44.593Z
2017-02-14T14:59:50.893Z
2017-02-14T14:59:50.893Z
user14603
<p>Given many individuals (mostly men) have anomalies of colour perception, you might avoid colour; you also want a simple task with 'easy' stimuli, to enable short trial durations.  Visual search paradigms are good for this.
I echo the call for a project in which the experimental design would address something of particular interest, and ideally that would be difficult to address with a 'standard' design (e.g., with 10,000 data points, rather than 1,000,000).  Estimating the distribution of processing times for a process inferred from indirect measures may be an interesting problem domain.</p>

<p>PJ</p>

CC BY-SA 3.0
2017-02-08T06:36:35.400Z
Could you please add references to your statements. At CogSci aim to have well researched and well referenced questions and answers.
Robin Kramer-ten Have
CC BY-SA 3.0
2017-02-07T19:49:47.613Z
2017-02-07T19:49:47.613Z
Pierre Jolicoeur
<p>I vote for a simple random dot motion task. This has the advantages that:
  1. It's easy to program. It can be delivered over the web via javascript, which makes the "many labs" aspect easy.
  2. It's very well studied and well understood.
  3. It is easy to set 5-ish coherence levels that will sweep out almost the full range of accuracy, from chance level to 100%. This is important for distinguishing between the models.</p>

CC BY-SA 3.0
2017-02-08T06:35:36.700Z
Please add some references to your answer, with examples of or the development of this task. CogSci is a scientific stack, not a discussion board.
Robin Kramer-ten Have
CC BY-SA 3.0
2017-02-07T21:04:09.807Z
2017-02-07T21:04:09.807Z
Scott Brown
<p>May I suggest collecting data online? </p>

<p>With the latest browser technologies, RTs are extremely accurate, in the order of 5 microseconds (<a href="https://greensock.com/forums/topic/15548-timelinelite-timing-accuracy-usable-in-behavioural-reaction-time-based-research/" rel="nofollow noreferrer">discussion</a>). Even with slightly older browser tech, collected RTs can be accurate (<a href="https://peerj.com/articles/1058/#p-60" rel="nofollow noreferrer">ref</a>).</p>

<p>There are several sites where people freely volunteer to do research (<a href="https://peerj.com/articles/1058/#p-47" rel="nofollow noreferrer">discussion</a>). </p>

<p>Collecting data over multiple sites is most certainly easier when there is setup time per location.</p>

CC BY-SA 3.0
2017-02-13T12:39:18.680Z
This does not answer the OP's question. He is asking for RT tasks/paradigms. Although the info is interesting, it should have been posted as a comment, or in a new question altogether ( e.g. http://cogsci.stackexchange.com/q/9584/11318 )
Robin Kramer-ten Have
CC BY-SA 3.0
2017-02-13T10:02:54.540Z
2017-02-13T10:02:54.540Z
andyw
<p>May I ask, have you thought about using existing large datasets, such as those provided by <a href="https://www.projectimplicit.net/index.html" rel="nofollow noreferrer">Project Implicit</a>? <a href="https://osf.io/52qxl/" rel="nofollow noreferrer">Dataset</a>. </p>

<p>(sadly, it seems Project Implicit has not stored individual data points but rather a computed 'effect' per participant).</p>

CC BY-SA 3.0
2017-02-13T12:39:38.340Z
Thanks @andyw for your input. However, we really need raw data. Many models are best fit using likelihood methods. Also, some effects are visible at the extreme tails of the distributions so that summary statistics (mean, standard deviation, even skewness) are not enough information.
Denis Cousineau
CC BY-SA 3.0
2017-05-11T07:52:03.723Z
Just a followup. It is likely we run a 10 country, 800 participant per country replication of the gender/career IAT run in project implicit. We would save ALL the data. Perhaps 200 trials per participant (1.6 million RT scores). Of any use? https://uwahomewardbound.wordpress.com/2016/01/20/do-you-have-a-gender-career-bias/
andyw
CC BY-SA 3.0
2017-02-13T09:31:08.137Z
2017-02-13T09:55:30.540Z
2017-02-13T09:55:30.540Z
andyw
<p>A million trials isn't really that many. There are lots of tasks where you can collect 1000 trials an hour.  With a ten person subject panel where each person works 2 hours a day (10 hours a week) you could collect a million trials in 10 weeks. Assuming you are paying 15 USD an hour you can collect all the data for 15,000 USD which is a small grant. To run a ten person panel you would ideally want 3 testing stations and an RA to run things. </p>

CC BY-SA 3.0
2017-02-14T02:00:12.030Z
You do not provide an answer to the question.  It is asking for RT tasks/paradigms.
Denis Cousineau
CC BY-SA 3.0
2017-02-14T02:13:46.203Z
@DenisCousineau my answer is that your premise that collecting a million trials is difficult is wrong. It does not really qualify as a large N study that needs additional scrutiny or to be crowd sourced.
StrongBad
CC BY-SA 3.0
2017-02-14T07:36:13.393Z
@StrongBad such a remark would probably best fit as a comment on the question. Answers are reserved to actually answering the questions with statements based on scientific literature.
Robin Kramer-ten Have
CC BY-SA 3.0
2017-02-14T01:22:50.530Z
2017-02-14T01:28:50.400Z
2017-02-14T01:28:50.400Z
StrongBad
<p>Please allow me to question your setup:</p>

<p>Your assertion that you need a million data points to distinguish between models raises the question whether your're still doing valid science, I think. Are you sure that the differences between the models are scientifically sufficiently interesting to warrant a monster project to try and distinguish between them? Especially, if this is all based on a single task. I am sure, your quest will result in a "winning model". But I am not sure that it will result a generalizable scientific insight.</p>

CC BY-SA 3.0
2017-01-31T02:08:51.373Z
in a brief note, it is difficult to enumerate all the benefits of such a dataset. Examples: Are parameters describing performances normally distributed across subjects? Does training effect --resulting in faster RT-- imply that the parameters are all evolving in one direction? Does Bayesian estimation more apt than Maximum Likelihood at characterizing the parameters? How can we estimate non-decision times? What is the shape of the RT distributions and how training effect alter its shape? In Bayes again, after how many trials does priors have no longer an impact on the estimates? etc
Denis Cousineau
CC BY-SA 3.0
2017-01-31T02:01:24.143Z
2017-01-31T02:01:24.143Z
ElenorDavey
<p>I don't like the idea that the task should be low error, errors are needed for modeling choice. I don't like random dot motion because there are big individual difference making it hard to find a common calibration point over people. I prefer instead a numerosity judgement, e.g., are more of less than 50% of a square array of pixels (usually a large number) blue or orange? You can make the array big enough to get fine grained control of difficulty, and randomly choose which pixels to have in each color (or brightness) on each frame if you want to make particular strategies hard to use.  </p>

CC BY-SA 3.0
2017-04-07T00:22:00.923Z
Thank you. The proposed task has desired properties: fast responses; difficulty level easily adjustable; training effects are likely under extended practice; only two response alternatives. The experiment will begin May 1st, 2017!
Denis Cousineau
CC BY-SA 3.0
2017-04-14T11:18:39.993Z
@DenisCousineau Your comment sounds like a task has been chosen. What task was chosen? Is it still possible for researchers to contribute data?
matus
CC BY-SA 3.0
2017-04-23T23:45:02.903Z
@matus: The task chosen is a discrimination task: squares of two possible colors are shown dynamically, and the decision is to decide which of the two colors are the most common. Yes, you can indeed contribute data to this research. I plan on starting in September 2017. I will contact you at that moment. Thanks.
Denis Cousineau
CC BY-SA 3.0
2017-02-08T04:33:43.463Z
2017-04-13T06:40:16.360Z
2017-04-13T06:40:16.360Z
Andrew Heathcote
<p><strong>Context</strong>: Models of cognitive processes require very large datasets to be fitted. Sadly, it is difficult for a single laboratory to achieve this alone. I propose that we work collectively and collaboratively in realizing a 1 million RT experiment. This initiative is described on my ResearchGate page, <a href="https://www.researchgate.net/project/The-1-Million-RT-initiative" rel="nofollow noreferrer">www.researchgate.net/profile/Denis_Cousineau</a>. </p>

<p>Beyond model fitting, such a dataset will be beneficial for other purposes. Examples: Are parameters describing performances normally distributed across subjects? Does training effect --resulting in faster RT-- imply that the parameters are all evolving in one direction? Does Bayesian estimation more apt than Maximum Likelihood at characterizing the parameters? How can we estimate non-decision times? What is the shape of the RT distributions and how training effect alter its shape? In Bayes again, after how many trials does priors have no longer an impact on the estimates? etc.</p>

<p><strong>Task Requirements:</strong>
The task must be behavioral only (response times and response choices) as it is not possible to move EEG or fMRI material all around the world. The precise task used is not a critical constraint because the current models of cognition (sampling models such as LBA or Diffusion) are very flexible; they aim at predicting performance in a wide variety of situations.  </p>

<p>The tasks proposed will be evaluated based on the following criteria criteria: </p>

<ul>
<li>small number of response alternatives</li>
<li>few conditions</li>
<li>low error rates</li>
<li>stimuli that can be characterized for modeling purposes</li>
<li>possibility of extended training on that task. </li>
</ul>

<p>Possible candidate tasks could include: A redundant target detection task (Miller, 1982) or a same-different task (e.g., Bamber, 1969). These tasks are simple and accuracy is typically high (above 90% correct); they are possibly performed using some form of parallel processing; response times are generally well below 700 ms.</p>

<p>So in summary:</p>

<p><strong>What task and experimental design would be most suitable given the above constraints?</strong></p>

<p><em>Please let's see what are the possible tasks, comment on others' proposals, and if you find the perfect task, upvote it. Have your colleagues make comments or propose new tasks.</em></p>

CC BY-SA 3.0
2017-01-30T19:02:33.570Z
I like the idea, but asking for "the best" is rather opinion based, which is a reason for closure of the question. Could you explain a little bit more about what you are researching and why you (and others) want such a large amount of data? Don't cognitive models, and thus the data used, depend on the specific research questions?
Robin Kramer-ten Have
CC BY-SA 3.0
2017-01-30T19:11:39.600Z
@RobinKramer: The current models of cognitions (sampling models such as LBA or Diffusion) are very flexible; they aim at predicting performance in a wide variety of situations. Hence, the exact task used is not a critical constraint when testing these models. For example, they can be tested on lexical decision tasks, dot motion tasks, numerosity tasks, matching tasks, etc., all tasks that are very different but make similar assumptions about the underlying processes (accumulation of evidence+thresholds). However, what is critical is the sample size; with small $n$, all models offer good fits.
Denis Cousineau
CC BY-SA 3.0
2017-01-30T19:13:24.783Z
As of the label "best", that would be with regard to those criteria: small number of response alternatives, few conditions, low error rates, stimuli that can be characterized for modeling purposes, possibility of extended training on that task. Hence,with these objective criteria in mind, the question is no longer opinion-based.
Denis Cousineau
CC BY-SA 3.0
2017-01-30T19:15:39.293Z
Good point. I think it would be interesting to have a lot of different paradigms gathered here. As for the discussion part, that should be started in a chat, as the comments/answers are no place for discussions.
Robin Kramer-ten Have
CC BY-SA 3.0
2017-01-30T22:28:55.067Z
This is a post begging for some kind of pob competition. This is definitely not the site to start out some kind of in silico award for the best psychophysical task. -1 & vote-to-close & flagging too on top of it.
AliceD
CC BY-SA 3.0
2017-01-30T23:28:15.270Z
It is not a competition, it is a collaborative project. Getting a lot of researchers in experimental psychology to brainstorm alternatives is definitely the best way to find the optimal task.
Denis Cousineau
CC BY-SA 3.0
2017-01-31T08:32:59.563Z
might be but you may benefit from reading the help center on asking proper questions. The idea can be OK, the platform you've chosen to execute it is not.
AliceD
CC BY-SA 3.0
2017-01-31T16:00:52.687Z
See meta [What is cognitive sciences about](http://meta.cogsci.stackexchange.com/questions/2172/what-is-cognitive-science-about-and-what-characterize-cognitive-science-research) for my view on this.
Denis Cousineau
CC BY-SA 3.0
2017-02-01T09:50:24.073Z
See the [help-center](http://cogsci.stackexchange.com/help/dont-ask): [... *open-ended questions diminish the usefulness of our site and push other questions off the front page* ].Go to the chat room for chatty conversations. Go to the meta site to find out whether a type of question is appropriate.
AliceD
CC BY-SA 3.0
2017-02-03T01:12:17.167Z
I think this is a very different kind of question. It does go against some of the rules of the site. It could be adapted further to the norms of the site. That said, Denis (as a senior academic), I would like to welcome you to the site. My preference would be to give this question some latitude, given that we are keen to engage more academics and active researchers on the site. Perhaps we could propose and make edits to the question to bring the question more in line with expectations.
Jeromy Anglim
CC BY-SA 3.0
2017-02-03T01:26:17.597Z
I've given it a tiny edit for structure. I think the final bit "Please let's see what are..." would normally not be required on this site. But it appears Denis is inviting new researchers to the site to contribute (which is fantastic), so I think these notes are still useful.
Jeromy Anglim
CC BY-SA 3.0
2017-02-03T10:08:49.870Z
@JeromyAnglim - Thanks for looking into my flags. I'll leave my comments here, but if you support this question, feel free to remove my (kind of negative) stuff here if you consider it nothing but clutter. Cheers.
AliceD
CC BY-SA 3.0
2017-02-04T02:59:50.163Z
@AliceD Thanks for the openness.
Jeromy Anglim
<cognitive-modeling><decision-making><reaction-time>
What experiment in a simple decision task should we run to obtain one million trials?
Denis Cousineau
# What experiment in a simple decision task should we run to obtain one million trials?

(Asked by: username_0 on 2017-01-30)

**Context**: Models of cognitive processes require very large datasets to be fitted. Sadly, it is difficult for a single laboratory to achieve this alone. I propose that we work collectively and collaboratively in realizing a 1 million RT experiment. This initiative is described on my ResearchGate page, [www.researchgate.net/profile/Denis\_Cousineau](https://www.researchgate.net/project/The-1-Million-RT-initiative).

Beyond model fitting, such a dataset will be beneficial for other purposes. Examples: Are parameters describing performances normally distributed across subjects? Does training effect --resulting in faster RT-- imply that the parameters are all evolving in one direction? Does Bayesian estimation more apt than Maximum Likelihood at characterizing the parameters? How can we estimate non-decision times? What is the shape of the RT distributions and how training effect alter its shape? In Bayes again, after how many trials does priors have no longer an impact on the estimates? etc.

**Task Requirements:**
The task must be behavioral only (response times and response choices) as it is not possible to move EEG or fMRI material all around the world. The precise task used is not a critical constraint because the current models of cognition (sampling models such as LBA or Diffusion) are very flexible; they aim at predicting performance in a wide variety of situations.

The tasks proposed will be evaluated based on the following criteria criteria:

* small number of response alternatives
* few conditions
* low error rates
* stimuli that can be characterized for modeling purposes
* possibility of extended training on that task.

Possible candidate tasks could include: A redundant target detection task (Miller, 1982) or a same-different task (e.g., Bamber, 1969). These tasks are simple and accuracy is typically high (above 90% correct); they are possibly performed using some form of parallel processing; response times are generally well below 700 ms.

So in summary:

**What task and experimental design would be most suitable given the above constraints?**

*Please let's see what are the possible tasks, comment on others' proposals, and if you find the perfect task, upvote it. Have your colleagues make comments or propose new tasks.*

**Tags:** cognitive-modeling, decision-making, reaction-time

> **Comments:**
> **username_1**: I like the idea, but asking for "the best" is rather opinion based, which is a reason for closure of the question. Could you explain a little bit more about what you are researching and why you (and others) want such a large amount of data? Don't cognitive models, and thus the data used, depend on the specific research questions?  
> **username_0**: @RobinKramer: The current models of cognitions (sampling models such as LBA or Diffusion) are very flexible; they aim at predicting performance in a wide variety of situations. Hence, the exact task used is not a critical constraint when testing these models. For example, they can be tested on lexical decision tasks, dot motion tasks, numerosity tasks, matching tasks, etc., all tasks that are very different but make similar assumptions about the underlying processes (accumulation of evidence+thresholds). However, what is critical is the sample size; with small $n$, all models offer good fits.  
> **username_0**: As of the label "best", that would be with regard to those criteria: small number of response alternatives, few conditions, low error rates, stimuli that can be characterized for modeling purposes, possibility of extended training on that task. Hence,with these objective criteria in mind, the question is no longer opinion-based.  
> **username_1**: Good point. I think it would be interesting to have a lot of different paradigms gathered here. As for the discussion part, that should be started in a chat, as the comments/answers are no place for discussions.  
> **username_2**: This is a post begging for some kind of pob competition. This is definitely not the site to start out some kind of in silico award for the best psychophysical task. -1 & vote-to-close & flagging too on top of it.  
> **username_0**: It is not a competition, it is a collaborative project. Getting a lot of researchers in experimental psychology to brainstorm alternatives is definitely the best way to find the optimal task.  
> **username_2**: might be but you may benefit from reading the help center on asking proper questions. The idea can be OK, the platform you've chosen to execute it is not.  
> **username_0**: See meta [What is cognitive sciences about](http://meta.cogsci.stackexchange.com/questions/2172/what-is-cognitive-science-about-and-what-characterize-cognitive-science-research) for my view on this.  
> **username_2**: See the [help-center](http://cogsci.stackexchange.com/help/dont-ask): [... *open-ended questions diminish the usefulness of our site and push other questions off the front page* ].Go to the chat room for chatty conversations. Go to the meta site to find out whether a type of question is appropriate.  
> **username_3**: I think this is a very different kind of question. It does go against some of the rules of the site. It could be adapted further to the norms of the site. That said, Denis (as a senior academic), I would like to welcome you to the site. My preference would be to give this question some latitude, given that we are keen to engage more academics and active researchers on the site. Perhaps we could propose and make edits to the question to bring the question more in line with expectations.  
> **username_3**: I've given it a tiny edit for structure. I think the final bit "Please let's see what are..." would normally not be required on this site. But it appears Denis is inviting new researchers to the site to contribute (which is fantastic), so I think these notes are still useful.  
> **username_2**: @JeromyAnglim - Thanks for looking into my flags. I'll leave my comments here, but if you support this question, feel free to remove my (kind of negative) stuff here if you consider it nothing but clutter. Cheers.  
> **username_3**: @username_2 Thanks for the openness.  

---

## 10 answers

### Answer by username_12 on 2017-02-08. Score: 4 | Accepted answer

I don't like the idea that the task should be low error, errors are needed for modeling choice. I don't like random dot motion because there are big individual difference making it hard to find a common calibration point over people. I prefer instead a numerosity judgement, e.g., are more of less than 50% of a square array of pixels (usually a large number) blue or orange? You can make the array big enough to get fine grained control of difficulty, and randomly choose which pixels to have in each color (or brightness) on each frame if you want to make particular strategies hard to use.

> **Comments:**
> **username_0**: Thank you. The proposed task has desired properties: fast responses; difficulty level easily adjustable; training effects are likely under extended practice; only two response alternatives. The experiment will begin May 1st, 2017!  
> **username_13**: @DenisCousineau Your comment sounds like a task has been chosen. What task was chosen? Is it still possible for researchers to contribute data?  
> **username_0**: @username_13: The task chosen is a discrimination task: squares of two possible colors are shown dynamically, and the decision is to decide which of the two colors are the most common. Yes, you can indeed contribute data to this research. I plan on starting in September 2017. I will contact you at that moment. Thanks.  

---

### Answer by username_4 on 2017-02-01. Score: 4

I like this idea! I think the task should be one where the data could be used by many analytic tools. Hence, the data would not only be beneficial for one particular question, for example parameter estimations, but also for other tools such as Systems Factorial Technology (see Townsend & Nozawa, 1995), which provides a deeper understanding of the underlying cognitive sub-processes architecture, and other interesting information such as the process' capacity.

I suggest that the task at least respect a double factorial paradigm. This design is very simple: you have four conditions in a 2 x 2 design, where you vary two variables. It could be any variable, the only restriction is that you have to vary them independently one from another. In other words, each variable should reflect a cognitive sub-process. In one condition, both variables are in scenario where the sub-process should work very effectively (or optimally). In an other condition, both variables are in a scenario where the sub-process should work less effectively (or sub-optimally). And in the two remaining conditions, one variable works optimally and the other sub-optimally, and vice-versa.

As for the very details of the task, such as the stimuli that will be used and the specific variables, I do not have preferences. Letters are very manipulable, but they do carry semantic (more or less). Abstract stimuli could work. We could play with color, shapes, complexity, etc. At Psychonomics this year, they were many experiments which have use this design.

Reference for citation:
Townsend, J. T. & Nozawa, G. (1995). Spatio-temporal properties of elementation perception an investigation of parallel, serial, and coactive theories. Journal of Mathematical Psychology, 39(4), 321–359. doi: 10.1006/jmps.1995.1033

> **Comments:**
> **username_1**: Could you provide more detailed citations, preferably full APA. That will make it easier for readers to find the papers.  

---

### Answer by username_5 on 2017-02-12. Score: 4

Just a few comments...

Since one of the advantages of a large data set is high statistical power, it might be a good idea to use a task where the key effect has *not* been found in previous studies. That would produce a guaranteed result: i.e., a high-power test of an important null result.

A pop-out condition in visual search would be one example (i.e., no effect of display size), but this is not ideal because it would limit the modelling end of the project. That is, it wouldn't be too helpful to model multiple conditions with identical performance (or nearly so).

A better example would be an additive-factor experiment (somewhat as suggested by username_4). I would suggest a simple 2x2 design for which previous studies had reported theoretically important factor additivity (i.e., non-significant interaction). A big study would provide a high-power test of the null interaction. And, as long as both main effects were present, there would be four distinct conditions to model. This design is also interesting because a lot is known about conditions that should be satisfied by the RT distributions under certain models (for examples, see the references below).

A final point: After selecting a task, it might be very useful to test out the planned analyses on simulated data, which are considerably easier to get than real data. This would help make the project's precise goals more explicit and also make sure the goals would be achievable using the proposed data set. It might also help to reveal what aspects of the planned task/conditions were necessary for getting informative results (e.g., minimum effect sizes).

References

Cortese, J. M. & Dzhafarov, E. N. Empirical recovery of response time decomposition rules II. Discriminability of serial and parallel architectures. Journal of Mathematical Psychology, 1996 , 40 , 203-218

Dzhafarov, E. N. & Cortese, J. M. Empirical recovery of response time decomposition rules I. Sample-level decomposition tests. Journal of Mathematical Psychology, 1996 , 40 , 185-202

Dzhafarov, E. N. & Schweickert, R. Decompositions of response times: An almost general theory. Journal of Mathematical Psychology, 1995 , 39 , 285-314

Roberts, S. & Sternberg, S. The meaning of additive reaction-time effects: Tests of three alternatives. In Meyer, D. E. & Kornblum, S. (Eds.) Attention and performance XIV. Synergies in experimental psychology, artificial intelligence, and cognitive neuroscience., MIT Press, 1992 , 611-653


---

### Answer by username_6 on 2017-02-02. Score: 4

I agree with Marc-André, the selected stimuli should have no semantic association (or as little as possible) which is why I would encourage the use of geometric shapes that vary in colour. Additionally, I believe that the task should be as simple as possible in order to encourage the overall cleanliness of the data.

As you suggested, one such task is the redundant attribute target detection task (Miller, 1982). In this task, participants are asked to keep in mind target dimensions (e.g. blue and square) and as soon as one of the dimensions is presented on screen, the participants respond as quickly and as accurately as possible. Results are typically fast, accurate, and slightly skewed – appropriate for this initiative! We could select a few colours and shapes (if we do go forward with this task) as well as what colour harmony should be selected.

Additionally, having colours that vary in “obviousness” can potentially give us a clearer understanding of the parameters within certain models. For example, an obvious blue should have quicker RT than a muted blue and the rate parameter of each individual channel should reflect that (assuming, of course, that the rate parameter is associated with "obviousness" at all). I am particularly interested in this aspect.

While a go/no would be great for this task, I also strongly believe that a 2AFC task would yield much more interesting results and the 1million RT initiative could help us really understand the “race” in “race models”. This could be as simple as modifying the task as one with “target present” or “target absent” decisions. With clearer trends in RT we could hopefully gather information on the independence of dimensions and whether there is crosstalk between detectors.

Additionally, having such data would not only benefit model fitting but also more solid architecture analyses. This would, in turn, help us get a more concrete understanding of coactive architectures (the architecture that Miller, 1982 suggested was at play in this type of task) which haven’t been as rigorously defined like other architectures (as noted by Houpt & Townsend, 2011).

EDIT:

Houpt, J.W., & Townsend, J.T. (2011). An extension of SIC predictions to the Wiener coactive model. Journal of Mathematical Psychology, 55, 267-270

Miller J. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14,247–279

> **Comments:**
> **username_1**: Could you provide more detailed citations, preferably full APA. That will make it easier for readers to find the papers.  

---

### Answer by username_7 on 2017-02-07. Score: 2

Given many individuals (mostly men) have anomalies of colour perception, you might avoid colour; you also want a simple task with 'easy' stimuli, to enable short trial durations. Visual search paradigms are good for this.
I echo the call for a project in which the experimental design would address something of particular interest, and ideally that would be difficult to address with a 'standard' design (e.g., with 10,000 data points, rather than 1,000,000). Estimating the distribution of processing times for a process inferred from indirect measures may be an interesting problem domain.

PJ

> **Comments:**
> **username_1**: Could you please add references to your statements. At CogSci aim to have well researched and well referenced questions and answers.  

---

### Answer by username_8 on 2017-02-07. Score: 2

I vote for a simple random dot motion task. This has the advantages that:
1. It's easy to program. It can be delivered over the web via javascript, which makes the "many labs" aspect easy.
2. It's very well studied and well understood.
3. It is easy to set 5-ish coherence levels that will sweep out almost the full range of accuracy, from chance level to 100%. This is important for distinguishing between the models.

> **Comments:**
> **username_1**: Please add some references to your answer, with examples of or the development of this task. CogSci is a scientific stack, not a discussion board.  

---

### Answer by username_9 on 2017-02-13. Score: 1

May I suggest collecting data online?

With the latest browser technologies, RTs are extremely accurate, in the order of 5 microseconds ([discussion](https://greensock.com/forums/topic/15548-timelinelite-timing-accuracy-usable-in-behavioural-reaction-time-based-research/)). Even with slightly older browser tech, collected RTs can be accurate ([ref](https://peerj.com/articles/1058/#p-60)).

There are several sites where people freely volunteer to do research ([discussion](https://peerj.com/articles/1058/#p-47)).

Collecting data over multiple sites is most certainly easier when there is setup time per location.

> **Comments:**
> **username_1**: This does not answer the OP's question. He is asking for RT tasks/paradigms. Although the info is interesting, it should have been posted as a comment, or in a new question altogether ( e.g. http://cogsci.stackexchange.com/q/9584/11318 )  

---

### Answer by username_9 on 2017-02-13. Score: 1

May I ask, have you thought about using existing large datasets, such as those provided by [Project Implicit](https://www.projectimplicit.net/index.html)? [Dataset](https://osf.io/52qxl/).

(sadly, it seems Project Implicit has not stored individual data points but rather a computed 'effect' per participant).

> **Comments:**
> **username_0**: Thanks @username_9 for your input. However, we really need raw data. Many models are best fit using likelihood methods. Also, some effects are visible at the extreme tails of the distributions so that summary statistics (mean, standard deviation, even skewness) are not enough information.  
> **username_9**: Just a followup. It is likely we run a 10 country, 800 participant per country replication of the gender/career IAT run in project implicit. We would save ALL the data. Perhaps 200 trials per participant (1.6 million RT scores). Of any use? https://uwahomewardbound.wordpress.com/2016/01/20/do-you-have-a-gender-career-bias/  

---

### Answer by username_10 on 2017-02-14. Score: -1

A million trials isn't really that many. There are lots of tasks where you can collect 1000 trials an hour. With a ten person subject panel where each person works 2 hours a day (10 hours a week) you could collect a million trials in 10 weeks. Assuming you are paying 15 USD an hour you can collect all the data for 15,000 USD which is a small grant. To run a ten person panel you would ideally want 3 testing stations and an RA to run things.

> **Comments:**
> **username_0**: You do not provide an answer to the question.  It is asking for RT tasks/paradigms.  
> **username_10**: @DenisCousineau my answer is that your premise that collecting a million trials is difficult is wrong. It does not really qualify as a large N study that needs additional scrutiny or to be crowd sourced.  
> **username_1**: @username_10 such a remark would probably best fit as a comment on the question. Answers are reserved to actually answering the questions with statements based on scientific literature.  

---

### Answer by username_11 on 2017-01-31. Score: -1

Please allow me to question your setup:

Your assertion that you need a million data points to distinguish between models raises the question whether your're still doing valid science, I think. Are you sure that the differences between the models are scientifically sufficiently interesting to warrant a monster project to try and distinguish between them? Especially, if this is all based on a single task. I am sure, your quest will result in a "winning model". But I am not sure that it will result a generalizable scientific insight.

> **Comments:**
> **username_0**: in a brief note, it is difficult to enumerate all the benefits of such a dataset. Examples: Are parameters describing performances normally distributed across subjects? Does training effect --resulting in faster RT-- imply that the parameters are all evolving in one direction? Does Bayesian estimation more apt than Maximum Likelihood at characterizing the parameters? How can we estimate non-decision times? What is the shape of the RT distributions and how training effect alter its shape? In Bayes again, after how many trials does priors have no longer an impact on the estimates? etc  

---

