A clinical decision enhancement framework for large language models guided by medical knowledge graph: Improving diagnostic accuracy and rationality

Background With the breakthrough progress of large language models, they have shown tremendous application potential in medical scenarios such as health consulting, medical education, medical research and clinical decision support. However, large language models often suffer from low accuracy and are prone to hallucinations when making clinical decisions especially in real-world clinical settings. It is important to develop a framework that can mitigate hallucinations in clinical decision support scenarios. Methods To solve this problem, the study focuses on scenarios of diagnostic support and proposes a framework to enhance medical knowledge of large language models which is used in clinical decision support. The framework contains three modules: knowledge graph (KG) building module, which integrates data from the unified medical language system (UMLS), SemMedDB and electronic health record; knowledge retrieval module, which implements a mixed retrieval strategy based on KG inference and text embedding; knowledge utilization enhancement module, which uses an instruction fine-tuned dataset and LoRA technique to improve the knowledge utilization ability of large language models. By leveraging retrieved and graph-grounded medical knowledge during instruction fine-tuning, the framework enables large language models to learn more effective knowledge utilization under limited clinical labels, thereby improving diagnostic accuracy and mitigating hallucination tendencies. Results To validate the proposed method, experiments were conducted using data on acute abdominal diseases from the Medical Information Mart for Intensive Care (MIMIC) dataset, including all necessary auxiliary examinations recommended by the guidelines for the primary diagnosis of the corresponding diseases, involving 2,400 patients. And data on chronic kidney disease in diabetic patients which includes data of 1088 inpatients with kidney disease who have both type 2 diabetes and moderate proteinuria. Experimental results demonstrate that the proposed method outperforms baseline strategies, including zero-shot inference, chain-of-thought (CoT), and retrieval-augmented generation (RAG), across both diagnostic scenarios. For acute abdominal disease diagnosis, our method achieves average accuracies of 94.81%, 92.94%, and 95.12% on Qwen-7B, Qwen-14B, and GLM-4, respectively, compared with corresponding zero-shot baseline performances of 75.35%, 78.74%, and 69.24%. This corresponds to absolute accuracy improvements of 19.46%, 14.20%, and 25.88%, respectively, and also exceeds CoT and RAG baselines by 26.57% and 14.61% on average. For chronic kidney disease diagnosis in diabetic patients, the proposed framework also demonstrates consistent performance gains. Specifically, the micro-F1 scores of our method reach 50.01%, 46.09%, and 42.82% for Qwen-7B, Qwen-14B, and GLM-4, respectively, compared with corresponding zero-shot baseline performances of 35.42%, 36.89%, and 34.72%. Additionally, trustworthy medical knowledge is incorporated during recommendation decision-making, enhancing the credibility of the generated diagnostic results. Conclusion These results are helpful to improve the value of large language models in clinical practice, while providing a probable method for building clinical decision support system integrating multi-source knowledge and large language models.