<p>You will probably need to do pilot testing to ascertain the difficulty of a given correlational task. </p>

<p>From pilot testing, you would need to quantify the difficulty of the task. One option would be the mean discrepancy between actual and predicted correlation. However, there would be other metrics of both degree of error and difficulty.</p>

<p>I would hypothesise that the easiest correlations to guess based on scatterplots would be strong linear relationships involving bivariate normal distributions and lots of data (e.g., n > 1000).</p>

<p>There are a whole bunch of things that you could try to make the task more difficult. Here's what comes to my mind:</p>

<ul>
<li>Outliers (both outliers that increase the correlation and outliers that decrease the correlations); vary the number of outliers in a particular region; combine outliers that increase and decrease the correlation; make outliers even more extreme</li>
<li>Non-linear relationships (e.g., mixtures of linear and another function such as quadratic, cyclical, stepwise; power functions, logistic fucntions, circular, etc.)</li>
<li>bimodal distributions on one or both variables</li>
<li>Highly skewed distributions on one or both variables</li>
<li>Fewer data points</li>
<li>data that yield correlations only a little bit above or below zero</li>
<li>where there is a function like a quadratic, make the x depend on y rather than y depend on x. </li>
</ul>

<p>More generally, I think that practice and feedback effects will also be relevant. I.e., it may be relevant to ascertain whether the differences in difficulty with novices also correspond to relative differences in difficulty with participants who have been exposed to the full range of items you have generated.</p>

CC BY-SA 3.0
2016-10-24T22:52:16.293Z
2016-10-24T23:21:18.823Z
2016-10-24T23:21:18.823Z
Jeromy Anglim
<p>I am trying to generate correlations between random variables (two dimensional) with a defined linear relationship (in the $r$ sense), but with different visual patterns when plotted. I am trying to create a 'guess the correlation' task where I can systematically manipulate the difficulty for an observer to guess the linear relationship. </p>

<p>What I am doing now is given a correlation $r$ I generate the first and second values, $X_1$ and $X_2$, with $n$ samples from the standard normal distribution. Then from there I make $X_3$ a linear combination of the two $X_3 = r X_1 + \sqrt{1-r^2}\,X_2$</p>

<p>Then:  $Y_1 = \mu_1  + \sigma_1 X_1, \quad Y_2 = \mu_2 + \sigma_2 X_3$</p>

<p>And now $Y_1$ and $Y_2$ have a correlation $r$.</p>

<p>For manipulating the difficulty I've been playing with the parameters of the distribution and $n$, however, I am not satisfied with the results. </p>

<p>Any idea on how to systematically increase the difficulty of the task? (i.e., adding outliers, for instance etc).</p>

<p>Note: Difficulty is a cognitive/psychology question rather than a statistical one. I intend to test the notion of difficulty empirically (i.e., under a specific parameter combinations, people tend to do worse). The idea is to generate plots with varying parameters for a given correlation value (i.e., changing the number of points, the variance, outlier, functional form? etc). What are the parameters and what would be a systematic way to manipulate them.</p>

CC BY-SA 3.0
2016-10-24T16:55:14.520Z
Even though intentions are cognitive, and I really like the question, I do believe you have the best chance at finding an answer at [stats.se]. I'll mark the question and see whether moderators can migrate it. The question over there will then still be linked to this website, making it easier to find.
Robin Kramer-ten Have
CC BY-SA 3.0
2016-10-24T22:39:42.577Z
@Robin I disagree to some extent. The question is about how to make the cognitive task of guessing a correlation from a scatter plot more difficult. Perhaps stats people would have insights, but I imagine its also more of a psychological question.
Jeromy Anglim
CC BY-SA 3.0
2016-10-25T10:28:47.837Z
I assume you know about this? http://guessthecorrelation.com
splint
CC BY-SA 3.0
2016-10-25T15:08:35.017Z
@JeromyAnglim You are correct. I read the question as "how do you simulate correlations?", thinking it was more like a coding-question.
Robin Kramer-ten Have
<cognitive-psychology><measurement><experimental-psychology>
Manipulating the difficulty of guessing a correlation
amaatouq
# Manipulating the difficulty of guessing a correlation

(Asked by: username_0 on 2016-10-24)

I am trying to generate correlations between random variables (two dimensional) with a defined linear relationship (in the $r$ sense), but with different visual patterns when plotted. I am trying to create a 'guess the correlation' task where I can systematically manipulate the difficulty for an observer to guess the linear relationship.

What I am doing now is given a correlation $r$ I generate the first and second values, $X\_1$ and $X\_2$, with $n$ samples from the standard normal distribution. Then from there I make $X\_3$ a linear combination of the two $X\_3 = r X\_1 + \sqrt{1-r^2}\,X\_2$

Then: $Y\_1 = \mu\_1 + \sigma\_1 X\_1, \quad Y\_2 = \mu\_2 + \sigma\_2 X\_3$

And now $Y\_1$ and $Y\_2$ have a correlation $r$.

For manipulating the difficulty I've been playing with the parameters of the distribution and $n$, however, I am not satisfied with the results.

Any idea on how to systematically increase the difficulty of the task? (i.e., adding outliers, for instance etc).

Note: Difficulty is a cognitive/psychology question rather than a statistical one. I intend to test the notion of difficulty empirically (i.e., under a specific parameter combinations, people tend to do worse). The idea is to generate plots with varying parameters for a given correlation value (i.e., changing the number of points, the variance, outlier, functional form? etc). What are the parameters and what would be a systematic way to manipulate them.

**Tags:** cognitive-psychology, measurement, experimental-psychology

> **Comments:**
> **username_1**: Even though intentions are cognitive, and I really like the question, I do believe you have the best chance at finding an answer at [stats.se]. I'll mark the question and see whether moderators can migrate it. The question over there will then still be linked to this website, making it easier to find.  
> **username_2**: @Robin I disagree to some extent. The question is about how to make the cognitive task of guessing a correlation from a scatter plot more difficult. Perhaps stats people would have insights, but I imagine its also more of a psychological question.  
> **username_3**: I assume you know about this? http://guessthecorrelation.com  
> **username_1**: @JeromyAnglim You are correct. I read the question as "how do you simulate correlations?", thinking it was more like a coding-question.  

---

## 1 answers

### Answer by username_2 on 2016-10-24. Score: 2 | Accepted answer

You will probably need to do pilot testing to ascertain the difficulty of a given correlational task.

From pilot testing, you would need to quantify the difficulty of the task. One option would be the mean discrepancy between actual and predicted correlation. However, there would be other metrics of both degree of error and difficulty.

I would hypothesise that the easiest correlations to guess based on scatterplots would be strong linear relationships involving bivariate normal distributions and lots of data (e.g., n > 1000).

There are a whole bunch of things that you could try to make the task more difficult. Here's what comes to my mind:

* Outliers (both outliers that increase the correlation and outliers that decrease the correlations); vary the number of outliers in a particular region; combine outliers that increase and decrease the correlation; make outliers even more extreme
* Non-linear relationships (e.g., mixtures of linear and another function such as quadratic, cyclical, stepwise; power functions, logistic fucntions, circular, etc.)
* bimodal distributions on one or both variables
* Highly skewed distributions on one or both variables
* Fewer data points
* data that yield correlations only a little bit above or below zero
* where there is a function like a quadratic, make the x depend on y rather than y depend on x.

More generally, I think that practice and feedback effects will also be relevant. I.e., it may be relevant to ascertain whether the differences in difficulty with novices also correspond to relative differences in difficulty with participants who have been exposed to the full range of items you have generated.


---

