<p>This idea of score is interesting, but it's painful to assess if the two problems you raised are important or not. For the re-use of the "meaure group" I think it would be careful to not do it. I took much thinking over it and I still don't know what to think. Unfortunately, I have no solution, but if I share my different thoughts process here maybe it would help.</p>

<p><strong>1]</strong> I first thought that you should better build a rank of your different categories than to build a measure to avoid the problems you raised. </p>

<p>It would need not much subjects to get a reliable result. If you build a rank, you can defend the idea of using the same group for the two phases because it would have been extremely unlucky to get a different rank using a different sample for the same population. It can be assessed statistically how unlucky it would have been.</p>

<p>I must point out a problem that you have to deal with.
Let's imagine two different populations one has this rank $A&gt;B&gt;C&gt;D$ the other this one $A&lt;B&lt;C&lt;D$. Unless you have some a priori to the population, you can't judge than one is more "creative" than another. I known that "creative" is my words not yours but one must be careful about the interpretation of the different distribution you will have. If you wan't to avoid this, no need to build a rank for this reason, and just go for an independance test as mfloren suggested.</p>

<p>Let's say that you still conserve this rank.</p>

<p>To test your hypothesis, I think the simplest way is to compare distributions of creations between your two groups. <strong>ordered</strong> categories with a <strong>1-tailed kolmogorov-smirnov test</strong> because it's directly related to the hypothesis of you want to test. It is based on the biggest difference in the cumulative distribution of your two groups across you ordered category.</p>

<p>But there is a big problem (even for the case of independance test) : One could object that the i.i.d assumption is not met because subjects give many answers and some might be more creative than others. I don't know any between+within subjects alternative to kolmogorov smirnov. The simplest way to avoid this problem is to get back to 1 subject=1 data point...and this is where I am stuck because I don't see a nice way to pick one category. I must say that with your measure, this problem do not occur.<br>
<strong>2]</strong> I am not sure that avoiding the scoring solution was a good idea because of this pesty i.i.d problem.</p>

CC BY-SA 3.0
2015-06-19T20:08:53.407Z
-- I want to thank you for such deep consideration of my question. I'm still wrestling with how to class the designs. I'd defined 6 categories, but there are some designs that will not easily fit. Also, I like your question regarding what qualifies as a "creative" design. I assumed that designs of lesser frequency (i.e. rare) would be more creative; however, there are strategies which subjects may use to capitalize on the number of designs at the expense their novelty, which is something I've not yet considered. I think the use of a non-parametric test is a good idea as well.
credenzamatic
CC BY-SA 3.0
2015-06-20T08:48:27.000Z
But if two populations have reverted frequency... One would be less creative to the respect of the other - it just depends on the population you use to estimate the rank or the frequency. For this reason and for the i.i.d problem (which is important, also in the mfloren answser), I ll produce another answer to let you know what I would do there which fix theses two problems.
brumar
CC BY-SA 3.0
2015-06-17T20:49:00.527Z
2015-06-17T21:06:06.967Z
2015-06-17T21:06:06.967Z
brumar
<p>This is a follow-up after my first answer, this is based on the suggestion of @mfloren in the comments part of his answer.</p>

<p>You can't really use an independence test or a kolmogorov smirnov test because your data are not i.i.d. (your subjects give more than one answer). Your scoring idea would fix this, because you reestablish 1data=1subject but this has two problems :</p>

<ul>
<li>Subject-consuming (one group to establish the measure, and 2 groups to compare). The idea to use the same subjects for the measure and for the control group is generally a thing to avoid even if I am not 100% sure that it causes big trouble here.    </li>
<li>If your design, under certain conditions (reverted rank for example) AD could be shown as "more creative" than healthy subject, but the contrary could also be true if you use AD patients to build your scoring system. Using healthy people has a reference point is maybe not a big deal, but this is an argument which deserves to be dealt with.</li>
</ul>

<p>A simpler measure which would avoid these problems would be to consider the number of different design produced. From 0 to 6 if you have 6 different designs then. This is the proposal of @mfloren in the comments. I think it deserves a proper answer.</p>

<p>As you have two conditions, you can use a Friedman test which is a sort of non parametric ANOVA and you may use wilcoxon tests to study each condition with a multiple test correction like Bonferroni-Holmes.</p>

CC BY-SA 3.0
2015-06-20T09:21:24.230Z
2015-06-20T09:21:24.230Z
brumar
<p>To measure the frequencies of different patterns (do some patterns occur more frequently that others based on group) I see this as a chi-squared test of independence. If you are unfamiliar with the test, a quick example is <a href="http://www.biostathandbook.com/chiind.html" rel="nofollow">here</a>. For your situation, all participants would get the same placements of dots, and you would count how often each possible pattern is drawn. In this design you would have two variables: 1 being control/AD, and the other being "design" with levels of every design drawn (if there are low frequencies on the designs, check out <a href="https://en.wikipedia.org/wiki/Fisher&#39;s_exact_test" rel="nofollow">Fisher's Exact Test</a>).</p>

<p>As you mention, the only way you could get enough data for RM ANOVA is to add up all of the frequencies for all of the graphs. Not knowing the theoretical backing for this measure, are all of them designed to be equivalent? At the very least it seems that the 5 dot and the 10 dot should be separated...</p>

<p>All my 2 cents, but on to <strong>your real questions:</strong></p>

<p>1) Your "control sample" would be, in essence, a normative group (you would be determining what the "normal" person's frequencies were). These samples tend to be quite large and drawn with very rigorous designs. Check out your standard errors of your measures and see if they are acceptable (if they are, then the representativeness of your population may be your large concern with a small normative sample).</p>

<p>2) Double dipping is almost always a big "no-no." Idiosyncrasies in your initial "control sample" would drop down to your "clinical sample," casting doubt on the internal validity of the study. If the classifications are <em>made</em> for/from one group it usually follows that, in comparing to a second group, the second group with <em>almost always</em> not meet perfectly with the classification. In regression, this concept is that of "overfitting the model." One way you can get around this is create your classifications with, say, half of the control group, and use the other half as the clinical group. This has two problems, though: you further reduce your sample size; and I'd cast questions on the validity of "making classifications" (especially with so small a sample). Perhaps look at making all possible connections their own level of a categorical variable (combining only reflections and rotations).</p>

CC BY-SA 3.0
2015-06-17T20:53:17.053Z
Concerning your proposal of an independence test, In my pseudo-answer I raised the problem of multiple observations by subjects due to the fact that they draw as many designs as possible.If you don't think it's problematic or if you have a solution for that, I'd be glad to know it.
brumar
CC BY-SA 3.0
2015-06-17T21:00:11.407Z
I was thinking the presence or absence of the drawing being the "occurrence", though on reflection the participation in multiple "groups" by an individual may not be perfect... Perhaps a solution would be in the DISTINCT patterns by person over observation...
mflo-ByeSE
CC BY-SA 3.0
2015-06-17T21:03:22.873Z
You mean counting the number of distinct patterns for each subject ? I like this idea a lot and it seems to fit OP needs.
brumar
CC BY-SA 3.0
2015-06-19T20:19:05.497Z
@mfloren - Thanks for your answer. I had a feeling "double-dipping" was a bad idea, so this confirms it. Would you mind explaining how to check the standard error and what might be an acceptable level?
credenzamatic
CC BY-SA 3.0
2015-06-17T17:17:18.207Z
2015-06-17T17:17:18.207Z
mflo-ByeSE
<p>Before I can ask my questions, I have to give a little background information.  I'm involved in a study using a measure of design fluency.  Subjects are shown a page full of squares, like the one shown below.  Each square contains 5 dots.  Subjects are given 60 seconds to generate as many different designs as fast as they can by connecting dots in each square.  The criteria for correct designs are that the designs must contain 4 lines and that all the lines in the design must be continuous.  There are 2 trials in the test: one with 5 black dots and one with 5 black dots and 5 white dots.  In the second trial, the extra dots are included as form of interference. </p>

<p><img src="https://i.sstatic.net/WCJts.png" alt="two dots boxes"></p>

<p>Here are examples of correct designs produced in the first trial of the design fluency task.  Similar designs would be likely produced under the second trial.</p>

<p><img src="https://i.sstatic.net/oM0dk.jpg" alt="three correct designs"></p>

<p>The way this measure is usually quantified is by counting the <em>number</em> of correct designs produced.  However, I'm interested in looking at this task a little differently.  I want to know if members of a clinical group, say Alzheimer's disease (AD), will produce a <em>quality</em> of designs that is different than normal controls.</p>

<p>One way I thought about doing this was by looking at the frequency of designs produced.  This has been observed for tasks of verbal fluency, with <a href="http://www.tandfonline.com/doi/abs/10.1080/13803399508406584" rel="nofollow noreferrer">patients with AD producing words of a greater frequency than controls</a>.  The problem is that while counts of word frequency are readily available, there is no equivalent source for production on the design fluency task that I know of.</p>

<p>Here's how I thought about approaching this.  I'd start by taking the responses produced by a large number of controls and coming up with a way of classifying them, say 4 or 5 <em>types</em> of designs.  Then I'd count the number of times each of these design types occurred in the control sample and apply a weight to them so that design types with lesser frequency would have a greater weight (think <a href="http://nlp.stanford.edu/IR-book/html/htmledition/inverse-document-frequency-1.html" rel="nofollow noreferrer">inverse document frequency</a>).  I'd then look at a smaller AD sample and a equivalent control sample matched on demographic variables.  I'd count the number of each design types produced by each subject in both groups, and multiply the number of each design type by the weight assigned to it.  The average frequency would be calculated for each subject for each trial type (the 5 dot trial and the 10 dot interference trial).  Group performance would then be compared in a repeated measures ANOVA.</p>

<p>There are the two questions that I have. The control sample that I have to determine the frequency of the design types is small: less than 70 subjects.  Is this too small for determining the frequency and weights of the design types?  Secondly, when comparing the clinical sample to a control sample, would it be possible to draw the small control sample from this larger control sample used to derive the frequency and the weights, or is there some reason that I should not do this?</p>

<p>Thanks in advance for your attention and response.  Any input would be greatly  appreciated.  I realize this question may be off topic for this site.  If you feel that's the case, would you mind suggesting a more appropriate forum.  I've previously <a href="https://stats.stackexchange.com/questions/157221/use-of-a-sample-to-define-a-criterion-and-to-examine-difference-with-clinical-sa">posted this question on Cross Validated</a>, with no replies, so I'm hopeful for better reception here.</p>

CC BY-SA 3.0
2015-06-17T14:48:38.450Z
This is a great question for the site. It would help me understand the task a bit better if you could show some examples of designs that subjects might produce.
Josh
CC BY-SA 3.0
2015-06-17T15:02:21.990Z
@Josh -- Sure.  I just added them to the original post.  These are fairly representative of the kinds of designs produced.
credenzamatic
CC BY-SA 3.0
2015-06-17T16:25:42.797Z
I read carefully your question when you asked it on cross validated but was not confident enough to come up with an answer. I will try an answer here though. I did not understand the part with the "larger control sample", what is this sample ? The clinical one ?
brumar
CC BY-SA 3.0
2015-06-17T16:37:23.583Z
@brumar -- Let me rephrase the second question: I have a normal control sample of roughly 65 participants.  I'd like to generate frequency counts for the types of designs they generated, much like generating word frequencies.  Would it then be OK to use members of this larger control sample to compare to a clinical sample, or is this "double-dipping" for lack of a better term?  Thank you.
credenzamatic
CC BY-SA 3.0
2015-06-17T16:43:19.250Z
Ah.To be sure : Do you mean you have 3 samples, one to build your measure (65 subjects) and two others (control vs clinical) and you ask if the "measure group" can be reused in the experiment as control group ?
brumar
CC BY-SA 3.0
2015-06-17T17:00:22.097Z
Sorry, another question : The purpose of this score is just to test the hypothesis that AD patients are (in some sense) less creative, or you need this score for an ulterior use (like diagnosis) ?
brumar
CC BY-SA 3.0
2015-06-17T17:05:28.930Z
@brumar -- Right.  I only have the sample of 65 controls.  Can I use members of the that group twice, both to determine the frequency of the designs and then to compare to the AD sample?
credenzamatic
CC BY-SA 3.0
2015-06-17T17:07:47.630Z
@brumar -- I hadn't thought of it that way, but I guess that is the question I'm asking: Are AD patients less creative than controls?  The AD diagnosis would be confirmed through other means (diagnosis by specialist).
credenzamatic
<test><clinical-psychology><statistics><linguistics>
Qualitative scoring for responses to design fluency task
credenzamatic
# Qualitative scoring for responses to design fluency task

(Asked by: username_0 on 2015-06-17)

Before I can ask my questions, I have to give a little background information. I'm involved in a study using a measure of design fluency. Subjects are shown a page full of squares, like the one shown below. Each square contains 5 dots. Subjects are given 60 seconds to generate as many different designs as fast as they can by connecting dots in each square. The criteria for correct designs are that the designs must contain 4 lines and that all the lines in the design must be continuous. There are 2 trials in the test: one with 5 black dots and one with 5 black dots and 5 white dots. In the second trial, the extra dots are included as form of interference.

![two dots boxes](https://i.sstatic.net/WCJts.png)

Here are examples of correct designs produced in the first trial of the design fluency task. Similar designs would be likely produced under the second trial.

![three correct designs](https://i.sstatic.net/oM0dk.jpg)

The way this measure is usually quantified is by counting the *number* of correct designs produced. However, I'm interested in looking at this task a little differently. I want to know if members of a clinical group, say Alzheimer's disease (AD), will produce a *quality* of designs that is different than normal controls.

One way I thought about doing this was by looking at the frequency of designs produced. This has been observed for tasks of verbal fluency, with [patients with AD producing words of a greater frequency than controls](http://www.tandfonline.com/doi/abs/10.1080/13803399508406584). The problem is that while counts of word frequency are readily available, there is no equivalent source for production on the design fluency task that I know of.

Here's how I thought about approaching this. I'd start by taking the responses produced by a large number of controls and coming up with a way of classifying them, say 4 or 5 *types* of designs. Then I'd count the number of times each of these design types occurred in the control sample and apply a weight to them so that design types with lesser frequency would have a greater weight (think [inverse document frequency](http://nlp.stanford.edu/IR-book/html/htmledition/inverse-document-frequency-1.html)). I'd then look at a smaller AD sample and a equivalent control sample matched on demographic variables. I'd count the number of each design types produced by each subject in both groups, and multiply the number of each design type by the weight assigned to it. The average frequency would be calculated for each subject for each trial type (the 5 dot trial and the 10 dot interference trial). Group performance would then be compared in a repeated measures ANOVA.

There are the two questions that I have. The control sample that I have to determine the frequency of the design types is small: less than 70 subjects. Is this too small for determining the frequency and weights of the design types? Secondly, when comparing the clinical sample to a control sample, would it be possible to draw the small control sample from this larger control sample used to derive the frequency and the weights, or is there some reason that I should not do this?

Thanks in advance for your attention and response. Any input would be greatly appreciated. I realize this question may be off topic for this site. If you feel that's the case, would you mind suggesting a more appropriate forum. I've previously [posted this question on Cross Validated](https://stats.stackexchange.com/questions/157221/use-of-a-sample-to-define-a-criterion-and-to-examine-difference-with-clinical-sa), with no replies, so I'm hopeful for better reception here.

**Tags:** test, clinical-psychology, statistics, linguistics

> **Comments:**
> **username_1**: This is a great question for the site. It would help me understand the task a bit better if you could show some examples of designs that subjects might produce.  
> **username_0**: @username_1 -- Sure.  I just added them to the original post.  These are fairly representative of the kinds of designs produced.  
> **username_2**: I read carefully your question when you asked it on cross validated but was not confident enough to come up with an answer. I will try an answer here though. I did not understand the part with the "larger control sample", what is this sample ? The clinical one ?  
> **username_0**: @username_2 -- Let me rephrase the second question: I have a normal control sample of roughly 65 participants.  I'd like to generate frequency counts for the types of designs they generated, much like generating word frequencies.  Would it then be OK to use members of this larger control sample to compare to a clinical sample, or is this "double-dipping" for lack of a better term?  Thank you.  
> **username_2**: Ah.To be sure : Do you mean you have 3 samples, one to build your measure (65 subjects) and two others (control vs clinical) and you ask if the "measure group" can be reused in the experiment as control group ?  
> **username_2**: Sorry, another question : The purpose of this score is just to test the hypothesis that AD patients are (in some sense) less creative, or you need this score for an ulterior use (like diagnosis) ?  
> **username_0**: @username_2 -- Right.  I only have the sample of 65 controls.  Can I use members of the that group twice, both to determine the frequency of the designs and then to compare to the AD sample?  
> **username_0**: @username_2 -- I hadn't thought of it that way, but I guess that is the question I'm asking: Are AD patients less creative than controls?  The AD diagnosis would be confirmed through other means (diagnosis by specialist).  

---

## 3 answers

### Answer by username_3 on 2015-06-17. Score: 2 | Accepted answer

To measure the frequencies of different patterns (do some patterns occur more frequently that others based on group) I see this as a chi-squared test of independence. If you are unfamiliar with the test, a quick example is [here](http://www.biostathandbook.com/chiind.html). For your situation, all participants would get the same placements of dots, and you would count how often each possible pattern is drawn. In this design you would have two variables: 1 being control/AD, and the other being "design" with levels of every design drawn (if there are low frequencies on the designs, check out [Fisher's Exact Test](https://en.wikipedia.org/wiki/Fisher's_exact_test)).

As you mention, the only way you could get enough data for RM ANOVA is to add up all of the frequencies for all of the graphs. Not knowing the theoretical backing for this measure, are all of them designed to be equivalent? At the very least it seems that the 5 dot and the 10 dot should be separated...

All my 2 cents, but on to **your real questions:**

1) Your "control sample" would be, in essence, a normative group (you would be determining what the "normal" person's frequencies were). These samples tend to be quite large and drawn with very rigorous designs. Check out your standard errors of your measures and see if they are acceptable (if they are, then the representativeness of your population may be your large concern with a small normative sample).

2) Double dipping is almost always a big "no-no." Idiosyncrasies in your initial "control sample" would drop down to your "clinical sample," casting doubt on the internal validity of the study. If the classifications are *made* for/from one group it usually follows that, in comparing to a second group, the second group with *almost always* not meet perfectly with the classification. In regression, this concept is that of "overfitting the model." One way you can get around this is create your classifications with, say, half of the control group, and use the other half as the clinical group. This has two problems, though: you further reduce your sample size; and I'd cast questions on the validity of "making classifications" (especially with so small a sample). Perhaps look at making all possible connections their own level of a categorical variable (combining only reflections and rotations).

> **Comments:**
> **username_2**: Concerning your proposal of an independence test, In my pseudo-answer I raised the problem of multiple observations by subjects due to the fact that they draw as many designs as possible.If you don't think it's problematic or if you have a solution for that, I'd be glad to know it.  
> **username_3**: I was thinking the presence or absence of the drawing being the "occurrence", though on reflection the participation in multiple "groups" by an individual may not be perfect... Perhaps a solution would be in the DISTINCT patterns by person over observation...  
> **username_2**: You mean counting the number of distinct patterns for each subject ? I like this idea a lot and it seems to fit OP needs.  
> **username_0**: @mfloren - Thanks for your answer. I had a feeling "double-dipping" was a bad idea, so this confirms it. Would you mind explaining how to check the standard error and what might be an acceptable level?  

---

### Answer by username_2 on 2015-06-17. Score: 1

This idea of score is interesting, but it's painful to assess if the two problems you raised are important or not. For the re-use of the "meaure group" I think it would be careful to not do it. I took much thinking over it and I still don't know what to think. Unfortunately, I have no solution, but if I share my different thoughts process here maybe it would help.

**1]** I first thought that you should better build a rank of your different categories than to build a measure to avoid the problems you raised.

It would need not much subjects to get a reliable result. If you build a rank, you can defend the idea of using the same group for the two phases because it would have been extremely unlucky to get a different rank using a different sample for the same population. It can be assessed statistically how unlucky it would have been.

I must point out a problem that you have to deal with.
Let's imagine two different populations one has this rank $A>B>C>D$ the other this one $A

Let's say that you still conserve this rank.

To test your hypothesis, I think the simplest way is to compare distributions of creations between your two groups. **ordered** categories with a **1-tailed kolmogorov-smirnov test** because it's directly related to the hypothesis of you want to test. It is based on the biggest difference in the cumulative distribution of your two groups across you ordered category.

But there is a big problem (even for the case of independance test) : One could object that the i.i.d assumption is not met because subjects give many answers and some might be more creative than others. I don't know any between+within subjects alternative to kolmogorov smirnov. The simplest way to avoid this problem is to get back to 1 subject=1 data point...and this is where I am stuck because I don't see a nice way to pick one category. I must say that with your measure, this problem do not occur.  
**2]** I am not sure that avoiding the scoring solution was a good idea because of this pesty i.i.d problem.

> **Comments:**
> **username_0**: -- I want to thank you for such deep consideration of my question. I'm still wrestling with how to class the designs. I'd defined 6 categories, but there are some designs that will not easily fit. Also, I like your question regarding what qualifies as a "creative" design. I assumed that designs of lesser frequency (i.e. rare) would be more creative; however, there are strategies which subjects may use to capitalize on the number of designs at the expense their novelty, which is something I've not yet considered. I think the use of a non-parametric test is a good idea as well.  
> **username_2**: But if two populations have reverted frequency... One would be less creative to the respect of the other - it just depends on the population you use to estimate the rank or the frequency. For this reason and for the i.i.d problem (which is important, also in the mfloren answser), I ll produce another answer to let you know what I would do there which fix theses two problems.  

---

### Answer by username_2 on 2015-06-20. Score: 0

This is a follow-up after my first answer, this is based on the suggestion of @mfloren in the comments part of his answer.

You can't really use an independence test or a kolmogorov smirnov test because your data are not i.i.d. (your subjects give more than one answer). Your scoring idea would fix this, because you reestablish 1data=1subject but this has two problems :

* Subject-consuming (one group to establish the measure, and 2 groups to compare). The idea to use the same subjects for the measure and for the control group is generally a thing to avoid even if I am not 100% sure that it causes big trouble here.
* If your design, under certain conditions (reverted rank for example) AD could be shown as "more creative" than healthy subject, but the contrary could also be true if you use AD patients to build your scoring system. Using healthy people has a reference point is maybe not a big deal, but this is an argument which deserves to be dealt with.

A simpler measure which would avoid these problems would be to consider the number of different design produced. From 0 to 6 if you have 6 different designs then. This is the proposal of @mfloren in the comments. I think it deserves a proper answer.

As you have two conditions, you can use a Friedman test which is a sort of non parametric ANOVA and you may use wilcoxon tests to study each condition with a multiple test correction like Bonferroni-Holmes.


---

