Helpful or Harmful? Re-Evaluating Frugality in Retrieval-Augmented Generation for Medical Question Answering

Medical question answering (QA) systems and conversational agents have attracted growing interest as tools that can assist clinicians, support medical students, and help patients navigate complex information sources. However, existing evaluations of retrieval strategies largely overlook the cost–benefit balance—here referred to as frugality, under realistic computational constraints. This work introduces a frugality-based evaluation framework that jointly assesses accuracy improvements and computational cost to determine when retrieval-augmented generation is beneficial in medical question answering, rather than evaluating retrieval effectiveness through accuracy alone. This study addresses these gaps through a systematic comparative framework that evaluates retrieval relevance, computational efficiency, and knowledge base composition across multiple biomedical QA tasks. We employ open-source LLMs (LlaMA-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and DeepSeek-7B-Chat) across three benchmark medical QA datasets, including MedMCQA, MedQA-USMLE, and PubMedQA. In addition to that, we evaluate a dataset with larger contexts to simulate model distraction across the CliniQG4QA dataset using additional models (Meditron-7B, Qwen2.5-7B-Medical, Medgemma-4B, Phi-3-mini-4k-Instruct, and GPT4o-Mini). We examine how retrieval design choices alter the accuracy–latency trade-off, examining how relevance, corpus design, and hardware constraints interact in medical retrieval-augmented generation (RAG) systems. Our comprehensive results demonstrate when retrieval is genuinely beneficial versus when it imposes unnecessary computational costs, highlighting interactions between relevance and corpus designs in QA.