<p>The concern with your approach is called 'leakage' because you are parameterising the same data set that you are training. This can easily lead to overtraining is in ANN (artificial neural networks) and is not cool. Overtraining is where the ANN very tightly wraps around the training data set that it performs poorly on any other data set. In other words it is overtightened.</p>
<p>The way they get around it is to split the data set usually 60:20:20 into a training split, parameterisation (validation) split and test (testing) split. If the parameterisation split gets mixed with the training set - leakage happens and overtraining results. Normally the data is trained, and the trained model parameters are assessed/optimised in the parameter split. The trained data and optimised parameters are then tested in the test split.</p>
<p>As long as you are aware of the approach you are using, that I guess that is ok. If you separate your training and parameterisation (validation) data that would be cool.</p>

CC BY-SA 4.0
2020-06-23T01:09:06.760Z
2020-06-23T19:08:05.120Z
2020-06-23T19:08:05.120Z
M__
<p>I have a script in R/RStudio which creates random datasets of binomial variables, feeds them through a neural network, and calculates their likelihood ratio statistic and deviance. I'd like the script to loop with the <code>seed</code> incremented by 1, and collect the value from each run into a data frame, from which the values get averaged. For example, if the script returns <code>5, 10, and 15</code> on the first, second, and third iterations of the loop, I'd like the script to create a data frame with those values and then calculate their mean.</p>
<p>Here is my script, annotated:</p>
<pre><code>library(neuralnet)

x=1021

#change nrow to number of times code is looped
result &lt;- data.frame(matrix(nrow = 1000, ncol = 2)) 
colnames(result) &lt;- c(&quot;LogL&quot;, &quot;D&quot;)

repeat{

set.seed(x)

#2 inputs, max number of terms 3
x1=rbinom(2000,1,0.5)
x2=rbinom(2000,1,0.5)
y=rbinom(2000,1,0.5)

data=data.frame(x1,x2,y)

head(data)

#hidden units = 2
nnet=neuralnet(y~x1+x2,data,hidden=2,err.fct = &quot;ce&quot;,linear.output = F,likelihood = T)

predictions=compute(nnet,data[,1:2])

predictions$net.result

#likelihood ratio stat
argument=y*log(predictions<span class="math-container">$net.result)+(1-y)*log(1-predictions$</span>net.result)

head(argument)

#sum of likelihood ratio stat
LogL=sum(argument)

LogL

D=-2*LogL

D

result[x-1022, 1] &lt;- LogL
result[x-1022, 2] &lt;- D

result

#results &lt;- data.frame(LogL, D)

x=x+1
if (x==2021) {
  break
}

print(x)


}

result
</code></pre>
<p>My repeat loop works, but a thousand iterations, for example, takes multiple minutes. Is there a more efficient way to code such a loop?</p>

CC BY-SA 4.0
2020-06-22T23:01:23.177Z
Your `repeat` termination conditions will never be satisfied. Every round, x will be set to 1021, incremented by 1 and compared to 1023, which will never be true. How will the repeat ever stop?
Ram RS
CC BY-SA 4.0
2020-06-22T23:26:41.513Z
There are other approaches to parameterisation in ANN. @RamRS question on terminating recursion is definitely a good question.
M__
CC BY-SA 4.0
2020-06-22T23:44:49.673Z
Thanks, edited that and it works now, but it's super slow. I'll update the question!
Equinox
<r><machine-learning><loop>
How do I set a neural network to loop multiple times and average the resulting values?
# How do I set a neural network to loop multiple times and average the resulting values?

(Asked by: username_0 on 2020-06-22)

I have a script in R/RStudio which creates random datasets of binomial variables, feeds them through a neural network, and calculates their likelihood ratio statistic and deviance. I'd like the script to loop with the seed incremented by 1, and collect the value from each run into a data frame, from which the values get averaged. For example, if the script returns 5, 10, and 15 on the first, second, and third iterations of the loop, I'd like the script to create a data frame with those values and then calculate their mean.
Here is my script, annotated:
library(neuralnet)

x=1021

#change nrow to number of times code is looped
result <- data.frame(matrix(nrow = 1000, ncol = 2)) 
colnames(result) <- c("LogL", "D")

repeat{

set.seed(x)

#2 inputs, max number of terms 3
x1=rbinom(2000,1,0.5)
x2=rbinom(2000,1,0.5)
y=rbinom(2000,1,0.5)

data=data.frame(x1,x2,y)

head(data)

#hidden units = 2
nnet=neuralnet(y~x1+x2,data,hidden=2,err.fct = "ce",linear.output = F,likelihood = T)

predictions=compute(nnet,data[,1:2])

predictions$net.result

#likelihood ratio stat
argument=y*log(predictions$net.result)+(1-y)*log(1-predictions$net.result)

head(argument)

#sum of likelihood ratio stat
LogL=sum(argument)

LogL

D=-2*LogL

D

result[x-1022, 1] <- LogL
result[x-1022, 2] <- D

result

#results <- data.frame(LogL, D)

x=x+1
if (x==2021) {
  break
}

print(x)


}

result

My repeat loop works, but a thousand iterations, for example, takes multiple minutes. Is there a more efficient way to code such a loop?

**Tags:** r, machine-learning, loop
> **Comments:**
> **username_1**: Your `repeat` termination conditions will never be satisfied. Every round, x will be set to 1021, incremented by 1 and compared to 1023, which will never be true. How will the repeat ever stop?  
> **username_2**: There are other approaches to parameterisation in ANN. @RamRS question on terminating recursion is definitely a good question.  
> **username_0**: Thanks, edited that and it works now, but it's super slow. I'll update the question!  

---

## 1 answers

### Answer by username_2 on 2020-06-23. Score: 1

The concern with your approach is called 'leakage' because you are parameterising the same data set that you are training. This can easily lead to overtraining is in ANN (artificial neural networks) and is not cool. Overtraining is where the ANN very tightly wraps around the training data set that it performs poorly on any other data set. In other words it is overtightened.

The way they get around it is to split the data set usually 60:20:20 into a training split, parameterisation (validation) split and test (testing) split. If the parameterisation split gets mixed with the training set - leakage happens and overtraining results. Normally the data is trained, and the trained model parameters are assessed/optimised in the parameter split. The trained data and optimised parameters are then tested in the test split.

As long as you are aware of the approach you are using, that I guess that is ok. If you separate your training and parameterisation (validation) data that would be cool.


---

