  In this work, we study a new image annotation task named Extractive Tags
Summarization (ETS). The goal is to extract important tags from the context
lying in an image and its corresponding tags. We adjust some state-of-the-art
deep learning models to utilize both visual and textual information. Our
proposed solution consists of different widely used blocks like convolutional
and self-attention layers, together with a novel idea of combining auxiliary
loss functions and the gating mechanism to glue and elevate these fundamental
components and form a unified architecture. Besides, we introduce a loss
function that aims to reduce the imbalance of the training data and a simple
but effective data augmentation technique dedicated to alleviates the effect of
outliers on the final results. Last but not least, we explore an unsupervised
pre-training strategy to further boost the performance of the model by making
use of the abundant amount of available unlabeled data. Our model shows the
good results as 90% $F_\text{1}$ score on the public NUS-WIDE benchmark, and
50% $F_\text{1}$ score on a noisy large-scale real-world private dataset.
Source code for reproducing the experiments is publicly available at:
https://github.com/pixta-dev/labteam
