  Deep Convolutional Neural Networks (CNNs) have recently evinced immense
success for various image recognition tasks. However, a question of paramount
importance is somewhat unanswered in deep learning research - is the selected
CNN optimal for the dataset in terms of accuracy and model size? In this paper,
we intend to answer this question and introduce a novel strategy that alters
the architecture of a given CNN for a specified dataset, to potentially enhance
the original accuracy while possibly reducing the model size. We use two
operations for architecture refinement, viz. stretching and symmetrical
splitting. Our procedure starts with a pre-trained CNN for a given dataset, and
optimally decides the stretch and split factors across the network to refine
the architecture. We empirically demonstrate the necessity of the two
operations. We evaluate our approach on two natural scenes attributes datasets,
SUN Attributes and CAMIT-NSAD, with architectures of GoogleNet and VGG-11, that
are quite contrasting in their construction. We justify our choice of datasets,
and show that they are interestingly distinct from each other, and together
pose a challenge to our architectural refinement algorithm. Our results
substantiate the usefulness of the proposed method.
