  Learning invariant representations that remain useful for a downstream task
is still a key challenge in machine learning. We investigate a set of related
information funnels and bottleneck problems that claim to learn invariant
representations from the data. We also propose a new element to this family of
information-theoretic objectives: The Conditional Privacy Funnel with Side
Information, which we investigate in fully and semi-supervised settings. Given
the generally intractable objectives, we derive tractable approximations using
amortized variational inference parameterized by neural networks and study the
intrinsic trade-offs of these objectives. We describe empirically the proposed
approach and show that with a few labels it is possible to learn fair
classifiers and generate useful representations approximately invariant to
unwanted sources of variation. Furthermore, we provide insights about the
applicability of these methods in real-world scenarios with ordinary tabular
datasets when the data is scarce.
