  With the technological advancements of aerial imagery and accurate 3d
reconstruction of urban environments, more and more attention has been paid to
the automated analyses of urban areas. In our work, we examine two important
aspects that allow live analysis of building structures in city models given
oblique aerial imagery, namely automatic building extraction with convolutional
neural networks (CNNs) and selective real-time depth estimation from aerial
imagery. We use transfer learning to train the Faster R-CNN method for
real-time deep object detection, by combining a large ground-based dataset for
urban scene understanding with a smaller number of images from an aerial
dataset. We achieve an average precision (AP) of about 80% for the task of
building extraction on a selected evaluation dataset. Our evaluation focuses on
both dataset-specific learning and transfer learning. Furthermore, we present
an algorithm that allows for multi-view depth estimation from aerial imagery in
real-time. We adopt the semi-global matching (SGM) optimization strategy to
preserve sharp edges at object boundaries. In combination with the Faster
R-CNN, it allows a selective reconstruction of buildings, identified with
regions of interest (RoIs), from oblique aerial imagery.
