<p>There is no overtraining here, it isn't deep learning and there appears no parameterisation. If parameterisation was occurring on the training set that can and does cause over-tightening.</p>
<p>Essentially, you haven't included a data split for parameterisation, i.e. there is no parameterisation. The theory of a parameterisation is it must be distinct from training (and obviously) testing. It allows parameter tuned without overtightening. Lasso regression will require parameterisation.</p>
<p>I don't know about the survival regressions for machine learning, but lasso/ridge regression are standard approaches.</p>
<hr />
<p>So just to explain, what you do is have a split for parameters. Thus instead of 80:20 split it could be 70:20:10 split where the 10% is used for parameter optimisation. Bad code risks leakage from the parameter split and that can result in over-tightening.</p>

CC BY-SA 4.0
2022-07-26T15:17:07.250Z
2022-07-26T20:01:37.923Z
2022-07-26T20:01:37.923Z
M__
<ol>
<li>I got the data of stomach cancer from TCGA, I found some different
expression genes between cancer and normal sample.</li>
<li>I put aside my normal sample, and divide my cancer sample into
train(80%) and test(20%) groups.</li>
<li>I did Unicox (with survival package), lasso (with glmnet package)
and multicox ( with survival package) on train group, to find out
which genes are more related to survival.</li>
<li>I use coefficients of each genes (that I got from multicox), to
built my model and calculate the risk score. Like that: (expression
of gene1×coefficient gene1 + expression of gene2×coefficient gene2
+...).</li>
<li>I divided my sample in to high risk and low risk groups, to compare
survival between them, using Kaplan-Meier.</li>
</ol>
<p>In training group, Kaplan-Meier shows significant difference between high and low risk group. However when I use my model on my test group, it is not significant. Why this problem occurs? I assumed my model is overfit of my train group, but how can I fix it? Why does this happen even when I use lasso?</p>

CC BY-SA 4.0
2022-07-25T21:01:44.170Z
Do you have sufficient power? Are your hazard rates/ratios very different between train and test? It would be possible to have a model that performs exactly as well as it did in the training data, but that shows up with no significant difference simply due to having a smaller sample size in the test data.
Nuclear Hoagie
CC BY-SA 4.0
2022-07-25T21:10:34.833Z
Hi, thank u for your answer, yes at first I thought it is because of small sample size in test group, so I tried diving samples, 70-30, 60-40, and even 50-50. In all of the them pvalue in Kaplan meier was <0.001 in train group, and in non of the it was significant in test group
Farideh Jafari
CC BY-SA 4.0
2022-07-25T21:15:19.453Z
Hi @FaridehJafari could you kindly format your question to make it clear?
M__
<r>
My model doesn't work in test dataset
# My model doesn't work in test dataset

(Asked by: username_0 on 2022-07-25)

1. I got the data of stomach cancer from TCGA, I found some different
   expression genes between cancer and normal sample.
2. I put aside my normal sample, and divide my cancer sample into
   train(80%) and test(20%) groups.
3. I did Unicox (with survival package), lasso (with glmnet package)
   and multicox ( with survival package) on train group, to find out
   which genes are more related to survival.
4. I use coefficients of each genes (that I got from multicox), to
   built my model and calculate the risk score. Like that: (expression
   of gene1×coefficient gene1 + expression of gene2×coefficient gene2
   +...).
5. I divided my sample in to high risk and low risk groups, to compare
   survival between them, using Kaplan-Meier.

In training group, Kaplan-Meier shows significant difference between high and low risk group. However when I use my model on my test group, it is not significant. Why this problem occurs? I assumed my model is overfit of my train group, but how can I fix it? Why does this happen even when I use lasso?

**Tags:** r
> **Comments:**
> **username_1**: Do you have sufficient power? Are your hazard rates/ratios very different between train and test? It would be possible to have a model that performs exactly as well as it did in the training data, but that shows up with no significant difference simply due to having a smaller sample size in the test data.  
> **username_0**: Hi, thank u for your answer, yes at first I thought it is because of small sample size in test group, so I tried diving samples, 70-30, 60-40, and even 50-50. In all of the them pvalue in Kaplan meier was <0.001 in train group, and in non of the it was significant in test group  
> **username_2**: Hi @FaridehJafari could you kindly format your question to make it clear?  

---

## 1 answers

### Answer by username_2 on 2022-07-26. Score: 0

There is no overtraining here, it isn't deep learning and there appears no parameterisation. If parameterisation was occurring on the training set that can and does cause over-tightening.

Essentially, you haven't included a data split for parameterisation, i.e. there is no parameterisation. The theory of a parameterisation is it must be distinct from training (and obviously) testing. It allows parameter tuned without overtightening. Lasso regression will require parameterisation.

I don't know about the survival regressions for machine learning, but lasso/ridge regression are standard approaches.

---

So just to explain, what you do is have a split for parameters. Thus instead of 80:20 split it could be 70:20:10 split where the 10% is used for parameter optimisation. Bad code risks leakage from the parameter split and that can result in over-tightening.


---

