Finding Needles in Images: Can Multi-modal LLMs Locate Fine Details?

Multi-modal Large Language Models (MLLMs) can read entire documents, but can they find hidden, precise details buried in complex layouts? NiM-Benchmark introduces tough new VQA challenges: fine-grained, localized visual reasoning, like spotting menu prices or a tiny table insight on a research paper. Spot-IT delivers an intuitive approach that helps MLLMs "zoom in" — intelligently picking image patches and applying a dynamic Gaussian focus guided by the user's query. Large-scale evaluation sets new accuracy records and reveals just how far these models still lag behind human readers, especially for tiny, context-heavy details.

Vision-Language Models are bridging the gap between seeing and understanding, but their real-world utility hinges on precision: can they find just the answer—no more, no less—when it occupies only a sliver of a dense document? Most DocVQA benchmarks miss this, focusing on broad summaries or generic extraction, not the real-world pain-point of hidden, context-rich queries.

NiM-Benchmark is crafted for precision stress-testing, with 2,970 images and 1,180 question-answer pairs, from everyday sources:

• Restaurant menus (prices, dish attributes)
• Academic papers (tiny figure/table details, technical params)
• Magazines, newspapers
• Website & lecture screenshots

Each question targets a region covering <5% of the visual area. Domain variety and question complexity is ensured (see paper Table 9).

Spot-IT was tested on open-source and leading commercial MLLMs. Highlights:

• 15-21% accuracy improvement on NiM-Benchmark and ArxiVQA, over basic MLLM/OCR DocVQA pipelines.
• Even GPT-4o achieves only 38% EM (humans get 63%), showing how tough fine-grained questions are.
• Most failures: model hallucination, missing tiny evidence, or reasoning breakdown.
• Latency only +4s—acceptable for detailed search.

• Dataset : Hugging Face (see top links).
• Spot-IT is a plug-in step—no need to retrain! Works across modern MLLMs.
• Experiments done on NVIDIA A30 GPUs and with all major API-access models.

@inproceedings{thakkar-etal-2025-finding,
    title = "Finding Needles in Images: Can Multi-modal {LLM}s Locate Fine Details?",
    author = "Thakkar, Parth  and
      Agarwal, Ankush  and
      Kasu, Prasad  and
      Bansal, Pulkit  and
      Devaguptapu, Chaitanya",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.acl-long.1152/",
    pages = "23626--23648",
    ISBN = "979-8-89176-251-0"
}