  This study introduces a novel approach to ensure the existence and uniqueness
of optimal parameters in neural networks. The paper details how a recurrent
neural networks (RNN) can be transformed into a contraction in a domain where
its parameters are linear. It then demonstrates that a prediction problem
modeled through an RNN, with a specific regularization term in the loss
function, can have its first-order conditions expressed analytically. This
system of equations is reduced to two matrix equations involving Sylvester
equations, which can be partially solved. We establish that, if certain
conditions are met, optimal parameters exist, are unique, and can be found
through a straightforward algorithm to any desired precision. Also, as the
number of neurons grows the conditions of convergence become easier to fulfill.
Feedforward neural networks (FNNs) are also explored by including linear
constraints on parameters. According to our model, incorporating loops (with
fixed or variable weights) will produce loss functions that train easier,
because it assures the existence of a region where an iterative method
converges.
