  Probabilistic programming languages (PPLs) are receiving widespread attention
for performing Bayesian inference in complex generative models. However,
applications to science remain limited because of the impracticability of
rewriting complex scientific simulators in a PPL, the computational cost of
inference, and the lack of scalable implementations. To address these, we
present a novel PPL framework that couples directly to existing scientific
simulators through a cross-platform probabilistic execution protocol and
provides Markov chain Monte Carlo (MCMC) and deep-learning-based inference
compilation (IC) engines for tractable inference. To guide IC inference, we
perform distributed training of a dynamic 3DCNN--LSTM architecture with a
PyTorch-MPI-based framework on 1,024 32-core CPU nodes of the Cori
supercomputer with a global minibatch size of 128k: achieving a performance of
450 Tflop/s through enhancements to PyTorch. We demonstrate a Large Hadron
Collider (LHC) use-case with the C++ Sherpa simulator and achieve the
largest-scale posterior inference in a Turing-complete PPL.
