<p>You <em>could</em> use a one-sample <a href="https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test" rel="nofollow noreferrer">Wilcoxon signed-rank test</a> to ask whether the median of the human participant responses is different from the ChatGPT response. That's only appropriate for a single sentence, though; if you have a question about the sentences <em>as a group</em> you'll need to come up with an appropriate hypothesis that applies to the whole group before you bother testing anything at all. Hypothesis first!!</p>
<p>However, even for the one question this tests a pretty narrow hypothesis (whether the human responses are on average greater or less than the ChatGPT answer) that may not actually be relevant to your research question. From your description, you already know that the response patterns are not similar if you only have one ChatGPT response that is fixed and human responses that vary: those are completely different distributions. (You might also check whether this assumption of yours is actually correct or if it's possible to get different ChatGPT answers) So, I would be careful about what your actual hypothesis is and make sure that descriptive statistics aren't a better fit. Not every question is appropriate for null hypothesis significance testing. If you have a large enough human sample, you will <em>always</em> find statistical significance eventually, so it's probably more important that you characterize <em>how</em> different the responses are.</p>

CC BY-SA 4.0
2024-07-15T13:20:34.620Z
2024-07-15T13:20:34.620Z
Bryan Krause
<p>I am doing a study in which I intend to study if the response of chatGPT corresponds to responses by human participants. I will ask human participants (n = 50) to rate the correctness of some sentences on a scale of 1-10 and also ask the same questions to chatGPT. If I were to compare these two responses (humans vs chatGPT), what kind of statistics should I use since there is only one response from chatGPT whereas 50 responses from human participants? This is done to see if the response patterns are similar.</p>

<cognitive-psychology><social-psychology><decision-making>
How to compare ChatGPT response with human participant responses?
Christina
# How to compare ChatGPT response with human participant responses?

(Asked by: username_0 on 2024-07-15)

I am doing a study in which I intend to study if the response of chatGPT corresponds to responses by human participants. I will ask human participants (n = 50) to rate the correctness of some sentences on a scale of 1-10 and also ask the same questions to chatGPT. If I were to compare these two responses (humans vs chatGPT), what kind of statistics should I use since there is only one response from chatGPT whereas 50 responses from human participants? This is done to see if the response patterns are similar.

**Tags:** cognitive-psychology, social-psychology, decision-making


---

## 1 answers

### Answer by username_1 on 2024-07-15. Score: 0

You *could* use a one-sample [Wilcoxon signed-rank test](https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test) to ask whether the median of the human participant responses is different from the ChatGPT response. That's only appropriate for a single sentence, though; if you have a question about the sentences *as a group* you'll need to come up with an appropriate hypothesis that applies to the whole group before you bother testing anything at all. Hypothesis first!!

However, even for the one question this tests a pretty narrow hypothesis (whether the human responses are on average greater or less than the ChatGPT answer) that may not actually be relevant to your research question. From your description, you already know that the response patterns are not similar if you only have one ChatGPT response that is fixed and human responses that vary: those are completely different distributions. (You might also check whether this assumption of yours is actually correct or if it's possible to get different ChatGPT answers) So, I would be careful about what your actual hypothesis is and make sure that descriptive statistics aren't a better fit. Not every question is appropriate for null hypothesis significance testing. If you have a large enough human sample, you will *always* find statistical significance eventually, so it's probably more important that you characterize *how* different the responses are.


---

