  Two novel deep hybrid architectures, the Deep Hybrid Boltzmann Machine and
the Deep Hybrid Denoising Auto-encoder, are proposed for handling
semi-supervised learning problems. The models combine experts that model
relevant distributions at different levels of abstraction to improve overall
predictive performance on discriminative tasks. Theoretical motivations and
algorithms for joint learning for each are presented. We apply the new models
to the domain of data-streams in work towards life-long learning. The proposed
architectures show improved performance compared to a pseudo-labeled, drop-out
rectifier network.
