  Finding disentangled representation plays a predominant role in the success
of modern deep learning applications, but the results lack a straightforward
explanation. Here we apply the information bottleneck method and its
$\beta$-VAE implementation to find the disentangled low-dimensional
representation of classical models. For the Ising model, our results reveal a
deep connection between the disentangled features and the physical order
parameters, and the widely-used Bernoulli decoder is found to be learning a
mean-field Hamiltonian at fixed temperature. This analogy motivates us to
extend the application of $\beta$-VAE to more complex classical models with
non-binary variables using different decoder neural network and propose a
modified architecture $\beta^2$-VAE to enforce thermal fluctuations in
generated samples. Our work provides a way to design novel physics-informed
algorithm that can yield learned features in potential correspondence with real
physical properties.
