  We develop a distributed framework for the physics-informed neural networks
(PINNs) based on two recent extensions, namely conservative PINNs (cPINNs) and
extended PINNs (XPINNs), which employ domain decomposition in space and in
time-space, respectively. This domain decomposition endows cPINNs and XPINNs
with several advantages over the vanilla PINNs, such as parallelization
capacity, large representation capacity, efficient hyperparameter tuning, and
is particularly effective for multi-scale and multi-physics problems. Here, we
present a parallel algorithm for cPINNs and XPINNs constructed with a hybrid
programming model described by MPI $+$ X, where X $\in
\{\text{CPUs},~\text{GPUs}\}$. The main advantage of cPINN and XPINN over the
more classical data and model parallel approaches is the flexibility of
optimizing all hyperparameters of each neural network separately in each
subdomain. We compare the performance of distributed cPINNs and XPINNs for
various forward problems, using both weak and strong scalings. Our results
indicate that for space domain decomposition, cPINNs are more efficient in
terms of communication cost but XPINNs provide greater flexibility as they can
also handle time-domain decomposition for any differential equations, and can
deal with any arbitrarily shaped complex subdomains. To this end, we also
present an application of the parallel XPINN method for solving an inverse
diffusion problem with variable conductivity on the United States map, using
ten regions as subdomains.
