  Foundation Models (FMs) have demonstrated unprecedented capabilities
including zero-shot learning, high fidelity data synthesis, and out of domain
generalization. However, as we show in this paper, FMs still have poor
out-of-the-box performance on expert tasks (e.g. retrieval of car manuals
technical illustrations from language queries), data for which is either unseen
or belonging to a long-tail part of the data distribution of the huge datasets
used for FM pre-training. This underlines the necessity to explicitly evaluate
and finetune FMs on such expert tasks, arguably ones that appear the most in
practical real-world applications. In this paper, we propose a first of its
kind FETA benchmark built around the task of teaching FMs to understand
technical documentation, via learning to match their graphical illustrations to
corresponding language descriptions. Our FETA benchmark focuses on
text-to-image and image-to-text retrieval in public car manuals and sales
catalogue brochures. FETA is equipped with a procedure for completely automatic
annotation extraction (code would be released upon acceptance), allowing easy
extension of FETA to more documentation types and application domains in the
future. Our automatic annotation leads to an automated performance metric shown
to be consistent with metrics computed on human-curated annotations (also
released). We provide multiple baselines and analysis of popular FMs on FETA
leading to several interesting findings that we believe would be very valuable
to the FM community, paving the way towards real-world application of FMs for
practical expert tasks currently 'overlooked' by standard benchmarks focusing
on common objects.
