<p>If the magnitude of reaction times in the millisecond range are necessary, the options to do decent experiments are very much limited. However, often reaction times are compared between situations and then <em>relative</em> reaction times are sufficient, which greatly reduces the requirement of <em>accurate</em> measurements, although the <em>precision</em> still needs to be good. The latter, however, is much easier to ascertain even with mediocre systems.</p>
<p>Focusing on the reaction time problem here; the problem is that many platforms based on Windows are not suitable for accurate measurements in the millisecond range, despite processors operating in the Gigahertz range. This is because Windows has its own, illogical, pesky and utterly inefficient internal queue of processes waiting in line to be executed, including executing utterly irrelevant updates while you are <strike>doing</strike> attempting to do your reaction time experiments. Button presses may be coming in delayed or execution of the next part of your script may be postponed because of some other background process irrelevant to your experiment. MacOS and especially Linux are much better platforms in that regard as they tend to have less of a mind of their own.</p>
<p>Then the programming environment. To be frank, I only have experience with MATLAB, which is not designed for reaction time experiments, although it can be fast. However, running on a Windows machine it will be massively sluggish. To circumvent these issues I have used the (open access) <a href="http://psychtoolbox.org/" rel="nofollow noreferrer"><strong>Psychophysics3 toolbox for MATLAB</strong></a>. This toolbox can prioritize the script to put it somewhat on top of the queue to enhance timing. However, it is still far from perfect and in my hands the delay was nonetheless about 40 ms between command execution and (sound) output as measured from the soundcard. So that's not great. It seems that other systems are able to reduce this latency to the single digit millisecond range. I can recommend sifting through the <a href="http://peterscarfe.com/ptbtutorials.html" rel="nofollow noreferrer"><strong>introduction of the PTB3 toolbox</strong></a> and take the time to look at the excellent <strong><a href="http://psychtoolbox.org/forum.html" rel="nofollow noreferrer">PTB3 forum</a></strong> which has many excellent posts on reaction time experiments in using PTB3.</p>
<p>From what I understand, C / C++ are great environments for speedy and reliable timing. In addition, letting dedicated hardware take over the job of keyboard or mouse button presses accelerates system delays (USB polling times, especially sluggish with keyboards), such as dedicated <a href="https://pstnet.com/products/chronos/" rel="nofollow noreferrer"><strong>response boxes</strong></a>.</p>

CC BY-SA 4.0
2021-06-22T09:02:45.200Z
Thanks for your very nice introduction! I got it! In terms of accurate timing, the better practice may be implementing offline experiments on computers with Linux-based OS and timing function optimized software or packages, but the choice finally depends on the requirement of precision. Is that true?
C.K.
CC BY-SA 4.0
2021-06-22T09:08:15.153Z
@C.K. that sums it up pretty nicely. I would seriously suggest sifting through the forum questions, there are excellent and detailed posts of Mario Kleiner in there.
AliceD
CC BY-SA 4.0
2021-06-22T15:15:25.940Z
Generally I'd recommend going with dedicated hardware if millisecond-precision RTs are needed.
Bryan Krause
CC BY-SA 4.0
2021-06-22T08:15:39.993Z
2021-06-23T06:47:44.097Z
2021-06-23T06:47:44.097Z
AliceD
<p>All of these are pretty great options, to the point where the answer is probably just 'use the one you're most comfortable with'. (Although open source is a huge plus)</p>
<p>Javascript <em>can</em> do pretty decent timing, (for example here: <a href="https://link.springer.com/article/10.3758/s13428-021-01573-x" rel="nofollow noreferrer">https://link.springer.com/article/10.3758/s13428-021-01573-x</a>). There's still &quot;losing control of the rest of the environment&quot; if your study is online, but the js itself isn't a strongly limiting factor.</p>
<p>You might enjoy the timing mega-study by Bridges &amp; friends, <a href="https://peerj.com/articles/9414/" rel="nofollow noreferrer">https://peerj.com/articles/9414/</a> which goes into this stuff in detail. It has a nice up-to-date collection of presentation platforms, and does the digging into all their timing properties so that you don't have to.</p>

CC BY-SA 4.0
2021-06-22T00:42:20.290Z
Very much thanks! The papers are really helpful for me! According to the latter paper, it seems javascript can get a pretty good timing despite a relatively large lag at the milliseconds level (less than 50ms). So, relative timing comparison may be good with the same environment whereas absolute timing should be measured in lab-based e-prime, psychtoolbox, psychopy, etc.
C.K.
CC BY-SA 4.0
2021-06-21T21:57:15.193Z
2021-06-21T21:57:15.193Z
steveLangsford
<p>In terms of presenting and implementing experiments, which programming language is better?</p>
<p>Note: What we are talking about is <strong>not data processing</strong> but experiment presentation.</p>
<p>What I need:</p>
<ul>
<li>Experiments can be implemented on many computers at the same time, since the number of participants may be sometimes large.</li>
<li>Sometimes I need accurate timing to get parameters such as the reaction time in the millisecond range. Both the magnitude of the reaction time, as well as timing differences between conditions are of interest.</li>
<li>I prefer open-source programming environments, yet I have access to MATLAB for instance and finds are available.</li>
</ul>
<p>What I've considered so far:</p>
<ul>
<li><p><strong>Python</strong>. I've only tried <code>psychopy</code>.</p>
<ul>
<li>pros: Accurate Timing. It is claimed that the timing function is nice.</li>
<li>cons: Robustness. The <code>psychopy</code> programs are not robust enough and sometimes I encounter annoying problems and the community is also, although good, but not good enough in my opinion.</li>
</ul>
</li>
<li><p><strong>Html/JS + Backend Server</strong>. I've tried the scheme to write html/css/js to present my experiment and send data to the local server which I wrote with <code>flask</code> in Python since js cannot read and write files through the browser.</p>
<ul>
<li>pros:
<ul>
<li>Flexible. With this scheme, I can do almost all I want to do and even more simply with the web app development technique.</li>
<li>Apparence. The front-end, i.e. the UI of the experiment can be as pretty as possible since there are many tools to enhance the performance of html/css/js.</li>
<li>Online Data Collection.</li>
</ul>
</li>
<li>cons: Accurate Timing. ​I guess timing is a disadvantage for this scheme. I don't know how better is the optimization of <code>psychopy</code> or <code>E-prime</code> than simply using javascript, but I guess the measurement of parameters such as reaction time can be more inaccurate. If javascript is enough to do this, please tell me!</li>
</ul>
</li>
<li><p><strong><code>C#</code>/<code>QT</code>(c++ or python)/<code>VB.NET</code></strong> etc. These are similar to the previous html scheme while the presentation and data collection can be done in a desktop app together, but I have no idea about the timing problem as well.</p>
</li>
<li><p><strong>E-prime</strong>. I'm not very familiar with E-prime so if there are any mistakes in the description please point them out, thanks!</p>
<ul>
<li>pros:
<ul>
<li>Accurate Timing.</li>
<li>Robustness. As a well-known commercial experiment software, I believe e-prime is, to some extent, better than <code>psychopy</code>.</li>
</ul>
</li>
<li>cons:
<ul>
<li>Closed Source. I do have money to pay for it and if it's indeed irreplaceable and I'll use it, but if I have other choices, I prefer open-source :)</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>So, any suggestion on this question? or is there some recommendation? Very much thanks!</p>

CC BY-SA 4.0
2021-06-20T18:40:13.643Z
Hey C.K., are you interested in general pros/cons or a recommendation for your particular situation? Also are you interested in languages specifically or are you thinking about software programs?  If you want a specific recommendation, more information would be helpful. For example, do you need to collect data online? Do you need precision timing? Prefer open source?  Do you have your own server space, or money to pay for services? PS. I think e-prime tasks should run without the dongle. (someone please correct me if that's not the case)
pep
CC BY-SA 4.0
2021-06-21T03:31:26.763Z
Welcome to Psychology.SE. I feel this question could lead to an answer which could fill a textbook. Could you please separate your question into several different ones (what are the pros and cons of Python, what are the pros and cons of C#....) This way it provides more focused questions and answers.
Chris Rogers
CC BY-SA 4.0
2021-06-21T03:39:30.677Z
@P.P. Very much thanks! Sometimes I do need precision timing and indeed I prefer open-source therefore I tried `psychopy`. I'd directly play with html/js if no precision timing is needed :), and I have money to pay for services, that's no problem. What I'm considering about it only precision timing with JS. However, I found that `psychopy` is not that robust for me so I'd reconsider this question. As for e-prime, I'm no very familiar with that, so there may be some mistakes in my description and thanks for pointing out them.
C.K.
<experimental-psychology>
What are the pros and cons of different programming languages for offline psychology experiments?
C.K.
# What are the pros and cons of different programming languages for offline psychology experiments?

(Asked by: username_0 on 2021-06-20)

In terms of presenting and implementing experiments, which programming language is better?

Note: What we are talking about is **not data processing** but experiment presentation.

What I need:

* Experiments can be implemented on many computers at the same time, since the number of participants may be sometimes large.
* Sometimes I need accurate timing to get parameters such as the reaction time in the millisecond range. Both the magnitude of the reaction time, as well as timing differences between conditions are of interest.
* I prefer open-source programming environments, yet I have access to MATLAB for instance and finds are available.

What I've considered so far:

* **Python**. I've only tried `psychopy`.

  + pros: Accurate Timing. It is claimed that the timing function is nice.
  + cons: Robustness. The `psychopy` programs are not robust enough and sometimes I encounter annoying problems and the community is also, although good, but not good enough in my opinion.
* **Html/JS + Backend Server**. I've tried the scheme to write html/css/js to present my experiment and send data to the local server which I wrote with `flask` in Python since js cannot read and write files through the browser.

  + pros:
    - Flexible. With this scheme, I can do almost all I want to do and even more simply with the web app development technique.
    - Apparence. The front-end, i.e. the UI of the experiment can be as pretty as possible since there are many tools to enhance the performance of html/css/js.
    - Online Data Collection.
  + cons: Accurate Timing. ​I guess timing is a disadvantage for this scheme. I don't know how better is the optimization of `psychopy` or `E-prime` than simply using javascript, but I guess the measurement of parameters such as reaction time can be more inaccurate. If javascript is enough to do this, please tell me!
* **`C#`/`QT`(c++ or python)/`VB.NET`** etc. These are similar to the previous html scheme while the presentation and data collection can be done in a desktop app together, but I have no idea about the timing problem as well.
* **E-prime**. I'm not very familiar with E-prime so if there are any mistakes in the description please point them out, thanks!

  + pros:
    - Accurate Timing.
    - Robustness. As a well-known commercial experiment software, I believe e-prime is, to some extent, better than `psychopy`.
  + cons:
    - Closed Source. I do have money to pay for it and if it's indeed irreplaceable and I'll use it, but if I have other choices, I prefer open-source :)

So, any suggestion on this question? or is there some recommendation? Very much thanks!

**Tags:** experimental-psychology

> **Comments:**
> **username_1**: Hey username_0, are you interested in general pros/cons or a recommendation for your particular situation? Also are you interested in languages specifically or are you thinking about software programs?  If you want a specific recommendation, more information would be helpful. For example, do you need to collect data online? Do you need precision timing? Prefer open source?  Do you have your own server space, or money to pay for services? PS. I think e-prime tasks should run without the dongle. (someone please correct me if that's not the case)  
> **username_2**: Welcome to Psychology.SE. I feel this question could lead to an answer which could fill a textbook. Could you please separate your question into several different ones (what are the pros and cons of Python, what are the pros and cons of C#....) This way it provides more focused questions and answers.  
> **username_0**: @P.P. Very much thanks! Sometimes I do need precision timing and indeed I prefer open-source therefore I tried `psychopy`. I'd directly play with html/js if no precision timing is needed :), and I have money to pay for services, that's no problem. What I'm considering about it only precision timing with JS. However, I found that `psychopy` is not that robust for me so I'd reconsider this question. As for e-prime, I'm no very familiar with that, so there may be some mistakes in my description and thanks for pointing out them.  

---

## 2 answers

### Answer by username_3 on 2021-06-22. Score: 3

If the magnitude of reaction times in the millisecond range are necessary, the options to do decent experiments are very much limited. However, often reaction times are compared between situations and then *relative* reaction times are sufficient, which greatly reduces the requirement of *accurate* measurements, although the *precision* still needs to be good. The latter, however, is much easier to ascertain even with mediocre systems.

Focusing on the reaction time problem here; the problem is that many platforms based on Windows are not suitable for accurate measurements in the millisecond range, despite processors operating in the Gigahertz range. This is because Windows has its own, illogical, pesky and utterly inefficient internal queue of processes waiting in line to be executed, including executing utterly irrelevant updates while you are doing attempting to do your reaction time experiments. Button presses may be coming in delayed or execution of the next part of your script may be postponed because of some other background process irrelevant to your experiment. MacOS and especially Linux are much better platforms in that regard as they tend to have less of a mind of their own.

Then the programming environment. To be frank, I only have experience with MATLAB, which is not designed for reaction time experiments, although it can be fast. However, running on a Windows machine it will be massively sluggish. To circumvent these issues I have used the (open access) [**Psychophysics3 toolbox for MATLAB**](http://psychtoolbox.org/). This toolbox can prioritize the script to put it somewhat on top of the queue to enhance timing. However, it is still far from perfect and in my hands the delay was nonetheless about 40 ms between command execution and (sound) output as measured from the soundcard. So that's not great. It seems that other systems are able to reduce this latency to the single digit millisecond range. I can recommend sifting through the [**introduction of the PTB3 toolbox**](http://peterscarfe.com/ptbtutorials.html) and take the time to look at the excellent **[PTB3 forum](http://psychtoolbox.org/forum.html)** which has many excellent posts on reaction time experiments in using PTB3.

From what I understand, C / C++ are great environments for speedy and reliable timing. In addition, letting dedicated hardware take over the job of keyboard or mouse button presses accelerates system delays (USB polling times, especially sluggish with keyboards), such as dedicated [**response boxes**](https://pstnet.com/products/chronos/).

> **Comments:**
> **username_0**: Thanks for your very nice introduction! I got it! In terms of accurate timing, the better practice may be implementing offline experiments on computers with Linux-based OS and timing function optimized software or packages, but the choice finally depends on the requirement of precision. Is that true?  
> **username_3**: @username_0 that sums it up pretty nicely. I would seriously suggest sifting through the forum questions, there are excellent and detailed posts of Mario Kleiner in there.  
> **username_4**: Generally I'd recommend going with dedicated hardware if millisecond-precision RTs are needed.  

---

### Answer by username_5 on 2021-06-21. Score: 2

All of these are pretty great options, to the point where the answer is probably just 'use the one you're most comfortable with'. (Although open source is a huge plus)

Javascript *can* do pretty decent timing, (for example here: <https://link.springer.com/article/10.3758/s13428-021-01573-x>). There's still "losing control of the rest of the environment" if your study is online, but the js itself isn't a strongly limiting factor.

You might enjoy the timing mega-study by Bridges & friends, <https://peerj.com/articles/9414/> which goes into this stuff in detail. It has a nice up-to-date collection of presentation platforms, and does the digging into all their timing properties so that you don't have to.

> **Comments:**
> **username_0**: Very much thanks! The papers are really helpful for me! According to the latter paper, it seems javascript can get a pretty good timing despite a relatively large lag at the milliseconds level (less than 50ms). So, relative timing comparison may be good with the same environment whereas absolute timing should be measured in lab-based e-prime, psychtoolbox, psychopy, etc.  

---

