  Understanding physical phenomena is a key competence that enables humans and
animals to act and interact under uncertain perception in previously unseen
environments containing novel objects and their configurations. Developmental
psychology has shown that such skills are acquired by infants from observations
at a very early stage.
  In this paper, we contrast a more traditional approach of taking a
model-based route with explicit 3D representations and physical simulation by
an {\em end-to-end} approach that directly predicts stability from appearance.
We ask the question if and to what extent and quality such a skill can directly
be acquired in a data-driven way---bypassing the need for an explicit
simulation at run-time.
  We present a learning-based approach based on simulated data that predicts
stability of towers comprised of wooden blocks under different conditions and
quantities related to the potential fall of the towers. We first evaluate the
approach on synthetic data and compared the results to human judgments on the
same stimuli. Further, we extend this approach to reason about future states of
such towers that in turn enables successful stacking.
