  Compression is at the heart of effective representation learning. However,
lossy compression is typically achieved through simple parametric models like
Gaussian noise to preserve analytic tractability, and the limitations this
imposes on learning are largely unexplored. Further, the Gaussian prior
assumptions in models such as variational autoencoders (VAEs) provide only an
upper bound on the compression rate in general. We introduce a new noise
channel, \emph{Echo noise}, that admits a simple, exact expression for mutual
information for arbitrary input distributions. The noise is constructed in a
data-driven fashion that does not require restrictive distributional
assumptions. With its complex encoding mechanism and exact rate regularization,
Echo leads to improved bounds on log-likelihood and dominates $\beta$-VAEs
across the achievable range of rate-distortion trade-offs. Further, we show
that Echo noise can outperform flow-based methods without the need to train
additional distributional transformations.
