  Often the question arises whether $Y$ can be predicted based on $X$ using a
certain model. Especially for highly flexible models such as neural networks
one may ask whether a seemingly good prediction is actually better than fitting
pure noise or whether it has to be attributed to the flexibility of the model.
This paper proposes a rigorous permutation test to assess whether the
prediction is better than the prediction of pure noise. The test avoids any
sample splitting and is based instead on generating new pairings of
$(X_i,Y_j)$. It introduces a new formulation of the null hypothesis and
rigorous justification for the test, which distinguishes it from previous
literature. The theoretical findings are applied both to simulated data and to
sensor data of tennis serves in an experimental context. The simulation study
underscores how the available information affects the test. It shows that the
less informative the predictors, the lower the probability of rejecting the
null hypothesis of fitting pure noise and emphasizes that detecting weaker
dependence between variables requires a sufficient sample size.
