  Feedforward neural networks offer a promising approach for solving
differential equations. However, the reliability and accuracy of the
approximation still represent delicate issues that are not fully resolved in
the current literature. Computational approaches are in general highly
dependent on a variety of computational parameters as well as on the choice of
optimisation methods, a point that has to be seen together with the structure
of the cost function. The intention of this paper is to make a step towards
resolving these open issues. To this end we study here the solution of a simple
but fundamental stiff ordinary differential equation modelling a damped system.
We consider two computational approaches for solving differential equations by
neural forms. These are the classic but still actual method of trial solutions
defining the cost function, and a recent direct construction of the cost
function related to the trial solution method. Let us note that the settings we
study can easily be applied more generally, including solution of partial
differential equations. By a very detailed computational study we show that it
is possible to identify preferable choices to be made for parameters and
methods. We also illuminate some interesting effects that are observable in the
neural network simulations. Overall we extend the current literature in the
field by showing what can be done in order to obtain reliable and accurate
results by the neural network approach. By doing this we illustrate the
importance of a careful choice of the computational setup.
