  Building multi-modal language models has been a trend in the recent years,
where additional modalities such as image, video, speech, etc. are jointly
learned along with natural languages (i.e., textual information). Despite the
success of these multi-modal language models with different modalities, there
is no existing solution for neural network architectures and natural languages.
Providing neural architectural information as a new modality allows us to
provide fast architecture-2-text and text-2-architecture retrieval/generation
services on the cloud with a single inference. Such solution is valuable in
terms of helping beginner and intermediate ML users to come up with better
neural architectures or AutoML approaches with a simple text query. In this
paper, we propose ArchBERT, a bi-modal model for joint learning and
understanding of neural architectures and natural languages, which opens up new
avenues for research in this area. We also introduce a pre-training strategy
named Masked Architecture Modeling (MAM) for a more generalized joint learning.
Moreover, we introduce and publicly release two new bi-modal datasets for
training and validating our methods. The ArchBERT's performance is verified
through a set of numerical experiments on different downstream tasks such as
architecture-oriented reasoning, question answering, and captioning
(summarization). Datasets, codes, and demos are available supplementary
materials.
