  The transformer structure employed in large language models (LLMs), as a
specialized category of deep neural networks (DNNs) featuring attention
mechanisms, stands out for their ability to identify and highlight the most
relevant aspects of input data. Such a capability is particularly beneficial in
addressing a variety of communication challenges, notably in the realm of
semantic communication where proper encoding of the relevant data is critical
especially in systems with limited bandwidth. In this work, we employ vision
transformers specifically for the purpose of compression and compact
representation of the input image, with the goal of preserving semantic
information throughout the transmission process. Through the use of the
attention mechanism inherent in transformers, we create an attention mask. This
mask effectively prioritizes critical segments of images for transmission,
ensuring that the reconstruction phase focuses on key objects highlighted by
the mask. Our methodology significantly improves the quality of semantic
communication and optimizes bandwidth usage by encoding different parts of the
data in accordance with their semantic information content, thus enhancing
overall efficiency. We evaluate the effectiveness of our proposed framework
using the TinyImageNet dataset, focusing on both reconstruction quality and
accuracy. Our evaluation results demonstrate that our framework successfully
preserves semantic information, even when only a fraction of the encoded data
is transmitted, according to the intended compression rates.
