  Foundation models possess strong capabilities in reasoning and memorizing
across modalities. To further unleash the power of foundation models, we
present FIND, a generalized interface for aligning foundation models'
embeddings with unified image and dataset-level understanding spanning modality
and granularity. As shown in the teaser figure, a lightweight transformer
interface without tuning any foundation model weights is enough for
segmentation, grounding, and retrieval in an interleaved manner. The proposed
interface has the following favorable attributes: (1) Generalizable. It applies
to various tasks spanning retrieval, segmentation, etc., under the same
architecture and weights. (2) Interleavable. With the benefit of multi-task
multi-modal training, the proposed interface creates an interleaved shared
embedding space. (3) Extendable. The proposed interface is adaptive to new
tasks, and new models. In light of the interleaved embedding space, we
introduce FIND-Bench, which introduces new training and evaluation annotations
to the COCO dataset for interleaved segmentation and retrieval. We are the
first work aligning foundations models' embeddings for interleave
understanding. Meanwhile, our approach achieves state-of-the-art performance on
FIND-Bench and competitive performance on standard retrieval and segmentation
settings.
