  In this paper, we present Partially Stochastic Infinitely Deep Bayesian
Neural Networks, a novel family of architectures that integrates partial
stochasticity into the framework of infinitely deep neural networks. Our new
class of architectures is designed to improve the computational efficiency of
existing architectures at training and inference time. To do this, we leverage
the advantages of partial stochasticity in the infinite-depth limit which
include the benefits of full stochasticity e.g. robustness, uncertainty
quantification, and memory efficiency, whilst improving their limitations
around computational complexity. We present a variety of architectural
configurations, offering flexibility in network design including different
methods for weight partition. We also provide mathematical guarantees on the
expressivity of our models by establishing that our network family qualifies as
Universal Conditional Distribution Approximators. Lastly, empirical evaluations
across multiple tasks show that our proposed architectures achieve better
downstream task performance and uncertainty quantification than their
counterparts while being significantly more efficient. The code can be found at
\url{https://github.com/Sergio20f/part_stoch_inf_deep}
