  We compare the Finite Element Method (FEM) simulation of a standard Partial
Differential Equation thermal problem of a plate with a hole with a Neural
Network (NN) simulation. The largest deviation from the true solution obtained
from FEM ($0.015$ for a solution on the order of unity) is easily achieved with
NN too without much tuning of the hyperparameters. Accuracies below $0.01$
instead require refinement with an alternative optimizer to reach a similar
performance with NN. A rough comparison between the Floating Point Operations
values, as a machine-independent quantification of the computational
performance, suggests a significant difference between FEM and NN in favour of
the former. This also strongly holds for computation time: for an accuracy on
the order of $10^{-5}$, FEM and NN require $54$ and $1100$ seconds,
respectively. A detailed analysis of the effect of varying different
hyperparameters shows that accuracy and computational time only weakly depend
on the major part of them. Accuracies below $0.01$ cannot be achieved with the
"adam'' optimizers and it looks as though accuracies below $10^{-5}$ cannot be
achieved at all. In conclusion, the present work shows that for the concrete
case of solving a steady-state 2D heat equation, the performance of a FEM
algorithm is significantly better than the solution via networks.
