  The advancement of deep learning technology has enabled us to develop systems
that outperform any other classification technique. However, success of any
empirical system depends on the quality and diversity of the data available to
train the proposed system. In this research, we have carefully accumulated a
relatively challenging dataset that contains images collected from various
sources for three different disasters: fire, water and land. Besides this, we
have also collected images for various damaged infrastructure due to natural or
man made calamities and damaged human due to war or accidents. We have also
accumulated image data for a class named non-damage that contains images with
no such disaster or sign of damage in them. There are 13,720 manually annotated
images in this dataset, each image is annotated by three individuals. We are
also providing discriminating image class information annotated manually with
bounding box for a set of 200 test images. Images are collected from different
news portals, social media, and standard datasets made available by other
researchers. A three layer attention model (TLAM) is trained and average five
fold validation accuracy of 95.88% is achieved. Moreover, on the 200 unseen
test images this accuracy is 96.48%. We also generate and compare attention
maps for these test images to determine the characteristics of the trained
attention model. Our dataset is available at
https://niloy193.github.io/Disaster-Dataset
