  In Augmented Reality (AR) environment, realistic interactions between the
virtual and real objects play a crucial role in user experience. Much of recent
advances in AR has been largely focused on developing geometry-aware
environment, but little has been done in dealing with interactions at the
semantic level. High-level scene understanding and semantic descriptions in AR
would allow effective design of complex applications and enhanced user
experience. In this paper, we present a novel approach and a prototype system
that enables the deeper understanding of semantic properties of the real world
environment, so that realistic physical interactions between the real and the
virtual objects can be generated. A material-aware AR environment has been
created based on the deep material learning using a fully convolutional network
(FCN). The state-of-the-art dense Simultaneous Localisation and Mapping (SLAM)
has been used for the semantic mapping. Together with efficient accelerated 3D
ray casting, natural and realistic physical interactions are generated for
interactive AR games. Our approach has significant impact on the future
development of advanced AR systems and applications.
