  Aerial scene classification remains challenging as: 1) the size of key
objects in determining the scene scheme varies greatly; 2) many objects
irrelevant to the scene scheme are often flooded in the image. Hence, how to
effectively perceive the region of interests (RoIs) from a variety of sizes and
build more discriminative representation from such complicated object
distribution is vital to understand an aerial scene. In this paper, we propose
a novel all grains, one scheme (AGOS) framework to tackle these challenges. To
the best of our knowledge, it is the first work to extend the classic multiple
instance learning into multi-grain formulation. Specially, it consists of a
multi-grain perception module (MGP), a multi-branch multi-instance
representation module (MBMIR) and a self-aligned semantic fusion (SSF) module.
Firstly, our MGP preserves the differential dilated convolutional features from
the backbone, which magnifies the discriminative information from multi-grains.
Then, our MBMIR highlights the key instances in the multi-grain representation
under the MIL formulation. Finally, our SSF allows our framework to learn the
same scene scheme from multi-grain instance representations and fuses them, so
that the entire framework is optimized as a whole. Notably, our AGOS is
flexible and can be easily adapted to existing CNNs in a plug-and-play manner.
Extensive experiments on UCM, AID and NWPU benchmarks demonstrate that our AGOS
achieves a comparable performance against the state-of-the-art methods.
