<p><strong>When your goals is to measure one option over the other</strong></p>

<p>As part of a Human-Computer Interaction (HCI) study I also needed to measure cognitive load in order to assess whether a particular user interface could reduce perceived cognitive load (as compared to a control group). Because of practical reasons (and not to intrude too much in 'ordinary' computer usage) I decided to stick to the commonly used NASA TLX scale (<a href="http://dx.doi.org/10.1177/154193120605000909" rel="noreferrer">Hart, 2006</a>). However, for small-scale studies (mine involved just 16 participants) I have the impression you are just as well off merely asking for qualitative feedback. Of course you can't run statistics on these, but interpreting effect size on such small sample sizes is arguably meaningless either way. You could thus simply ask participants which option is preferred in regards to perceived cognitive load and report on this as a 'significant' result. Of course I can not guarantee whether reviewers would agree. :) In the case you have suitable existing NASA TLX measures to compare against this still remains a worthwhile option.</p>

<p>However, as you state a big downside to such a scale-based test is that it can only be administered at a few points throughout the study, since filling out scales is time intensive.</p>

<p><strong>For large scale or continuous quantitative studies</strong></p>

<p>Skin conductance, blood volume, and pulse rate seem quite promising and are being used in HCI (<a href="http://dx.doi.org/10.1016/j.ijhcs.2015.03.002" rel="noreferrer">Janssen et al., 2015</a>; <a href="http://dx.doi.org/10.1145/2207676.2207754" rel="noreferrer">Mark et al., 2012</a>), which are indicative of <em>arousal</em>. These nowadays can more easily be measured in an easy form factor (e.g., a watch). However, it is important to mention that arousal is not indicative of <em>valence</em>—the
pleasantness or unpleasantness of stimuli. Neither is using such equipment straightforward. Establishing a baseline for participants takes a considerable amount of work and care needs to be taken to do it properly (if one wants to compare results with other participants). Measurements on the wrist (as in measured by a watch) are weaker than on palms. For some people, no suitable signal is measured at all. Environmental factors like temperature changes and movement (in regards to skin conductance) and changes in lighting (pupil dilation) need to be controlled for.</p>

<p>To answer one of your containing questions, I do not believe there is an <em>established</em> opinion or a preferred method and it still relies heavily on what can be measured in the given context and what is to be expected (e.g., can it be presumed there is no positive valence?). Depending on the study one could still be preferred over the other (does the task involve movement?), and I definitely would not rule out qualitative measures.</p>

<p><strong>Studies comparing different measures</strong></p>

<p>When I was looking for information I did run into some sources which compare different measures. However, I can not summarize them by heart, but I hope the current mentioned issues and sources can already guide you in the right direction.</p>

<ul>
<li>A master thesis by Forne (2012): <a href="http://kiosk.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2012/rapporter12/forne_malin_12082.pdf" rel="noreferrer">Physiology as a Tool for UX and Usability Testing.</a></li>
<li>Haapalainen et al. (2010) <a href="http://www.cs.cmu.edu/~sjunikim/publications/UBICOMP2010_Cognitive_Load.pdf" rel="noreferrer">Psycho-physiological measures for assessing cognitive load.</a></li>
</ul>

<p><sub>
Hart, S. G. (2006, October). <a href="http://dx.doi.org/10.1177/154193120605000909" rel="noreferrer">NASA-task load index (NASA-TLX); 20 years later.</a> In Proceedings of the human factors and ergonomics society annual meeting (Vol. 50, No. 9, pp. 904-908). Sage Publications.<br>
Janssen, C. P., Gould, S. J., Li, S. Y., Brumby, D. P., &amp; Cox, A. L. (2015). <a href="http://dx.doi.org/10.1016/j.ijhcs.2015.03.002" rel="noreferrer">Integrating knowledge of multitasking and Interruptions across different Perspectives and research methods.</a> International Journal of Human-Computer Studies, 79, 1-5.<br>
Mark, G., Voida, S., &amp; Cardello, A. (2012, May). <a href="http://dx.doi.org/10.1145/2207676.2207754" rel="noreferrer">A pace not dictated by electrons: an empirical study of work without email.</a> In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 555-564). ACM.<br>
Forne, M. (2012). <a href="http://kiosk.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2012/rapporter12/forne_malin_12082.pdf" rel="noreferrer">Physiology as a Tool for UX and Usability Testing.</a> School of Computer Science and Communication, Master. Royal Institute of Technology, Stockholm.<br>
Haapalainen, E., Kim, S., Forlizzi, J. F., &amp; Dey, A. K. (2010, September). <a href="http://www.cs.cmu.edu/~sjunikim/publications/UBICOMP2010_Cognitive_Load.pdf" rel="noreferrer">Psycho-physiological measures for assessing cognitive load.</a> In Proceedings of the 12th ACM international conference on Ubiquitous computing (pp. 301-310). ACM.
</sub></p>

CC BY-SA 3.0
2016-05-19T22:49:29.093Z
Do not know whether this is relevant: small secondary tasks which are executed along a primary task can also be used to measure cognitive load (as in decreased performance). E.g., a stroop test: 
[Using stroop task to assess cognitive load](http://dl.acm.org/citation.cfm?id=1962345).
Steven Jeuris
CC BY-SA 3.0
2016-05-19T22:51:03.407Z
Wow thanks for this +1 - I merely skimmed over it and will do a more thorough read after a good nite's rest. Thanks again.
AliceD
CC BY-SA 3.0
2016-05-19T23:01:38.450Z
@Christiaan. You are welcome. Perhaps a bit of context in regards to this answer. I read up and experimented a bit with this since I was also looking for a more 'objective' and temporally sensitive measure of cognitive load. However, I ended up not doing anything with the recorded data due to the difficulties of interpreting it. :) It is however definitely worthwhile pursuing if it forms one of the main measures you are after, which in my case was not the main intent of my study.
Steven Jeuris
CC BY-SA 3.0
2016-05-20T11:56:46.517Z
Would you happen to know whether the Haapalainen et al. (2010) is a reliable source? I'm unfamiliar with Ubicomp and the authors. That kind of research is exactly what I was looking for! Thanks again.
AliceD
CC BY-SA 3.0
2016-05-20T12:06:59.443Z
@Christiaan, UbiComp is [a top tier (peer reviewed) conference in HCI](http://academic.research.microsoft.com/RankList?entitytype=3&topDomainID=2&subDomainID=12) (and conferences compared to other fields are as important as journals, if not more). However, it is a multidisciplinary field that is merely interested in measuring cognitive load as part of its broader research agenda, thus I would value papers from a more focused field more as I'd expect them to have more experience on the topic. In other words, it should be good quality, but I would also look at the past experience of the authors.
Steven Jeuris
CC BY-SA 3.0
2016-05-19T22:38:45.647Z
2016-05-20T11:26:24.630Z
2016-05-20T11:26:24.630Z
Steven Jeuris
<p>I am wondering what the most straightforward way is to measure stress levels objectively and quantitatively *, and specifically, <strong><a href="https://en.wikipedia.org/wiki/Cognitive_load">cognitive load</a></strong>. I am open to suggestions, but my understanding is that <a href="http://www.calmclinic.com/anxiety/symptoms/pupil"><strong>pupil dilation</strong></a> and increased <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3386730/"><strong>skin conductivity</strong></a> are good objective and quantifiable  measures of stress. </p>

<p>My question now is - <strong>is there an established opinion which of the two is the preferred method?</strong> </p>

<p>The thing is that skin conductance is relatively straightforward. An electrode or two suffices - too easy. Pupillometry, on the other hand, requires either cumbersome glasses or a head-mounted device, or, alternatively, a table-top device necessitating the subject to focus on a restricted field of view (otherwise the camera looses the pupil).</p>

<p>Given we are considering measuring children, an electrode would intuitively be better, as it can be secured firmly to a finger, it allows for easy instructions - 'don't touch it', and it is relatively robust to artifacts, given that puil recordings suffer from eye blinks. </p>

<p><strong>Should we consider practical considerations only and choose skin conductance, or are there compelling reasons to use an eye tracker instead</strong> **? </p>

<p><sub>*: we wish to avoid questionnaires due to time constraints<br>
**: we don't have money for both :-)</sub>  </p>

CC BY-SA 3.0
2016-05-26T11:56:30.307Z
At the CHI conference this year there was a session on [Usability and User Burden](https://chi2016.acm.org/wp/paper-proceedings/) with some papers which might be worthwhile looking into: _"Developing and Validating the User Burden Scale: A Tool
for Assessing User Burden in Computing Systems"_, and _"Understanding the Relationship between Frustration
and the Severity of Usability Problems:
What can Psychophysiological Data (Not) Tell Us?"_ specifically seem promising.
Steven Jeuris
CC BY-SA 3.0
2016-05-26T13:27:10.817Z
I just quickly skimmed _"[Understanding the Relationship between Frustration and the Severity of Usability Problems: What can Psychophysiological Data (Not) Tell Us?](http://dl.acm.org/citation.cfm?id=2858511&CFID=601330845&CFTOKEN=79906773)"_ and believe it is highly relevant for this question. I don't have time to report on it thoroughly, but I believe a summary write-up of that paper to be a suitable answer to this question.
Steven Jeuris
CC BY-SA 3.0
2016-05-26T19:21:24.617Z
@StevenJeuris - again, many thanks - I will check it out!
AliceD
<measurement><stress><experiment-design>
How to measure stress quantitatively?
AliceD
# How to measure stress quantitatively?

(Asked by: username_0 on 2016-05-19)

I am wondering what the most straightforward way is to measure stress levels objectively and quantitatively \*, and specifically, **[cognitive load](https://en.wikipedia.org/wiki/Cognitive_load)**. I am open to suggestions, but my understanding is that [**pupil dilation**](http://www.calmclinic.com/anxiety/symptoms/pupil) and increased [**skin conductivity**](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3386730/) are good objective and quantifiable measures of stress.

My question now is - **is there an established opinion which of the two is the preferred method?**

The thing is that skin conductance is relatively straightforward. An electrode or two suffices - too easy. Pupillometry, on the other hand, requires either cumbersome glasses or a head-mounted device, or, alternatively, a table-top device necessitating the subject to focus on a restricted field of view (otherwise the camera looses the pupil).

Given we are considering measuring children, an electrode would intuitively be better, as it can be secured firmly to a finger, it allows for easy instructions - 'don't touch it', and it is relatively robust to artifacts, given that puil recordings suffer from eye blinks.

**Should we consider practical considerations only and choose skin conductance, or are there compelling reasons to use an eye tracker instead** \*\*?

\*: we wish to avoid questionnaires due to time constraints  
\*\*: we don't have money for both :-)

**Tags:** measurement, stress, experiment-design

> **Comments:**
> **username_1**: At the CHI conference this year there was a session on [Usability and User Burden](https://chi2016.acm.org/wp/paper-proceedings/) with some papers which might be worthwhile looking into: _"Developing and Validating the User Burden Scale: A Tool for Assessing User Burden in Computing Systems"_, and _"Understanding the Relationship between Frustration and the Severity of Usability Problems: What can Psychophysiological Data (Not) Tell Us?"_ specifically seem promising.  
> **username_1**: I just quickly skimmed _"[Understanding the Relationship between Frustration and the Severity of Usability Problems: What can Psychophysiological Data (Not) Tell Us?](http://dl.acm.org/citation.cfm?id=2858511&CFID=601330845&CFTOKEN=79906773)"_ and believe it is highly relevant for this question. I don't have time to report on it thoroughly, but I believe a summary write-up of that paper to be a suitable answer to this question.  
> **username_0**: @StevenJeuris - again, many thanks - I will check it out!  

---

## 1 answers

### Answer by username_1 on 2016-05-19. Score: 5 | Accepted answer

**When your goals is to measure one option over the other**

As part of a Human-Computer Interaction (HCI) study I also needed to measure cognitive load in order to assess whether a particular user interface could reduce perceived cognitive load (as compared to a control group). Because of practical reasons (and not to intrude too much in 'ordinary' computer usage) I decided to stick to the commonly used NASA TLX scale ([Hart, 2006](http://dx.doi.org/10.1177/154193120605000909)). However, for small-scale studies (mine involved just 16 participants) I have the impression you are just as well off merely asking for qualitative feedback. Of course you can't run statistics on these, but interpreting effect size on such small sample sizes is arguably meaningless either way. You could thus simply ask participants which option is preferred in regards to perceived cognitive load and report on this as a 'significant' result. Of course I can not guarantee whether reviewers would agree. :) In the case you have suitable existing NASA TLX measures to compare against this still remains a worthwhile option.

However, as you state a big downside to such a scale-based test is that it can only be administered at a few points throughout the study, since filling out scales is time intensive.

**For large scale or continuous quantitative studies**

Skin conductance, blood volume, and pulse rate seem quite promising and are being used in HCI ([Janssen et al., 2015](http://dx.doi.org/10.1016/j.ijhcs.2015.03.002); [Mark et al., 2012](http://dx.doi.org/10.1145/2207676.2207754)), which are indicative of *arousal*. These nowadays can more easily be measured in an easy form factor (e.g., a watch). However, it is important to mention that arousal is not indicative of *valence*—the
pleasantness or unpleasantness of stimuli. Neither is using such equipment straightforward. Establishing a baseline for participants takes a considerable amount of work and care needs to be taken to do it properly (if one wants to compare results with other participants). Measurements on the wrist (as in measured by a watch) are weaker than on palms. For some people, no suitable signal is measured at all. Environmental factors like temperature changes and movement (in regards to skin conductance) and changes in lighting (pupil dilation) need to be controlled for.

To answer one of your containing questions, I do not believe there is an *established* opinion or a preferred method and it still relies heavily on what can be measured in the given context and what is to be expected (e.g., can it be presumed there is no positive valence?). Depending on the study one could still be preferred over the other (does the task involve movement?), and I definitely would not rule out qualitative measures.

**Studies comparing different measures**

When I was looking for information I did run into some sources which compare different measures. However, I can not summarize them by heart, but I hope the current mentioned issues and sources can already guide you in the right direction.

* A master thesis by Forne (2012): [Physiology as a Tool for UX and Usability Testing.](http://kiosk.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2012/rapporter12/forne_malin_12082.pdf)
* Haapalainen et al. (2010) [Psycho-physiological measures for assessing cognitive load.](http://www.cs.cmu.edu/~sjunikim/publications/UBICOMP2010_Cognitive_Load.pdf)

Hart, S. G. (2006, October). [NASA-task load index (NASA-TLX); 20 years later.](http://dx.doi.org/10.1177/154193120605000909) In Proceedings of the human factors and ergonomics society annual meeting (Vol. 50, No. 9, pp. 904-908). Sage Publications.  
Janssen, C. P., Gould, S. J., Li, S. Y., Brumby, D. P., & Cox, A. L. (2015). [Integrating knowledge of multitasking and Interruptions across different Perspectives and research methods.](http://dx.doi.org/10.1016/j.ijhcs.2015.03.002) International Journal of Human-Computer Studies, 79, 1-5.  
Mark, G., Voida, S., & Cardello, A. (2012, May). [A pace not dictated by electrons: an empirical study of work without email.](http://dx.doi.org/10.1145/2207676.2207754) In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 555-564). ACM.  
Forne, M. (2012). [Physiology as a Tool for UX and Usability Testing.](http://kiosk.nada.kth.se/utbildning/grukth/exjobb/rapportlistor/2012/rapporter12/forne_malin_12082.pdf) School of Computer Science and Communication, Master. Royal Institute of Technology, Stockholm.  
Haapalainen, E., Kim, S., Forlizzi, J. F., & Dey, A. K. (2010, September). [Psycho-physiological measures for assessing cognitive load.](http://www.cs.cmu.edu/~sjunikim/publications/UBICOMP2010_Cognitive_Load.pdf) In Proceedings of the 12th ACM international conference on Ubiquitous computing (pp. 301-310). ACM.

> **Comments:**
> **username_1**: Do not know whether this is relevant: small secondary tasks which are executed along a primary task can also be used to measure cognitive load (as in decreased performance). E.g., a stroop test:  [Using stroop task to assess cognitive load](http://dl.acm.org/citation.cfm?id=1962345).  
> **username_0**: Wow thanks for this +1 - I merely skimmed over it and will do a more thorough read after a good nite's rest. Thanks again.  
> **username_1**: @Christiaan. You are welcome. Perhaps a bit of context in regards to this answer. I read up and experimented a bit with this since I was also looking for a more 'objective' and temporally sensitive measure of cognitive load. However, I ended up not doing anything with the recorded data due to the difficulties of interpreting it. :) It is however definitely worthwhile pursuing if it forms one of the main measures you are after, which in my case was not the main intent of my study.  
> **username_0**: Would you happen to know whether the Haapalainen et al. (2010) is a reliable source? I'm unfamiliar with Ubicomp and the authors. That kind of research is exactly what I was looking for! Thanks again.  
> **username_1**: @Christiaan, UbiComp is [a top tier (peer reviewed) conference in HCI](http://academic.research.microsoft.com/RankList?entitytype=3&topDomainID=2&subDomainID=12) (and conferences compared to other fields are as important as journals, if not more). However, it is a multidisciplinary field that is merely interested in measuring cognitive load as part of its broader research agenda, thus I would value papers from a more focused field more as I'd expect them to have more experience on the topic. In other words, it should be good quality, but I would also look at the past experience of the authors.  

---

