  Visual-Semantic Embedding (VSE) networks can help search engines better
understand the meaning behind visual content and associate it with relevant
textual information, leading to more accurate search results. VSE networks can
be used in cross-modal search engines to embed image and textual descriptions
in a shared space, enabling image-to-text and text-to-image retrieval tasks.
However, the full potential of VSE networks for search engines has yet to be
fully explored. This paper presents Boon, a novel cross-modal search engine
that combines two state-of-the-art networks: the GPT-3.5-turbo large language
model, and the VSE network VITR (VIsion Transformers with Relation-focused
learning) to enhance the engine's capabilities in extracting and reasoning with
regional relationships in images. VITR employs encoders from CLIP that were
trained with 400 million image-description pairs and it was fine-turned on the
RefCOCOg dataset. Boon's neural-based components serve as its main
functionalities: 1) a 'cross-modal search engine' that enables end-users to
perform image-to-text and text-to-image retrieval. 2) a 'multi-lingual
conversational AI' component that enables the end-user to converse about one or
more images selected by the end-user. Such a feature makes the search engine
accessible to a wide audience, including those with visual impairments. 3) Boon
is multi-lingual and can take queries and handle conversations about images in
multiple languages. Boon was implemented using the Django and PyTorch
frameworks. The interface and capabilities of the Boon search engine are
demonstrated using the RefCOCOg dataset, and the engine's ability to search for
multimedia through the web is facilitated by Google's API.
