  We present the hidden-layer concatenated physics informed neural network
(HLConcPINN) method, which combines hidden-layer concatenated feed-forward
neural networks, a modified block time marching strategy, and a physics
informed approach for approximating partial differential equations (PDEs). We
analyze the convergence properties and establish the error bounds of this
method for two types of PDEs: parabolic (exemplified by the heat and Burgers'
equations) and hyperbolic (exemplified by the wave and nonlinear Klein-Gordon
equations). We show that its approximation error of the solution can be
effectively controlled by the training loss for dynamic simulations with long
time horizons. The HLConcPINN method in principle allows an arbitrary number of
hidden layers not smaller than two and any of the commonly-used smooth
activation functions for the hidden layers beyond the first two, with
theoretical guarantees. This generalizes several recent neural-network
techniques, which have theoretical guarantees but are confined to two hidden
layers in the network architecture and the $\tanh$ activation function. Our
theoretical analyses subsequently inform the formulation of appropriate
training loss functions for these PDEs, leading to physics informed neural
network (PINN) type computational algorithms that differ from the standard PINN
formulation. Ample numerical experiments are presented based on the proposed
algorithm to validate the effectiveness of this method and confirm aspects of
the theoretical analyses.
