BRIT: Bidirectional Retrieval over Unified Image-Text Graph

Enterprise documents containing proprietary terminology and company-specific names often cause traditional embedding-based RAG systems to fail due to poor similarity matching with pre-trained models. BRIT addresses this challenge through a novel multi-modal RAG framework that constructs unified text-image graphs from documents and retrieves query-relevant subgraphs without requiring fine-tuning. By capturing both semantic and spatial relationships between textual and visual elements, BRIT enables effective cross-modal retrieval on enterprise-specific content. We introduce the MM-RAG test set to evaluate multi-modal question answering capabilities that require understanding complex text-image relationships.

We introduce a MM-RAG benchmark comprising 500 complex questions that necessitate cross-modal, multi-hop retrieval to identify the key information. The benchmark contains three types of questions:

• Text-Image questions: Questions requiring a relevant image for answering but that image cannot be directly identified by the question
• Image-Text questions: Questions requiring a relevant texts for answering but texts cannot be directly identified by the question
• Image-Image questions: Simple baseline questions which can be answered directly based on the images

BRIT outperforms the baselines in overall QA accuracy on complex question:

• Image-Image questions are the simple question which does not require the crossmodal multi-hop retrieval. Thus, the baseline methods with the simple retrieval of the top-2 most relevant images achieves the highest accuracy, while BRIT only retrieve 1 image with the similarity.
• The baseline struggles with Image-Text questions, resulting in a large drop in performance compared to other cases.