Vision-Language Transformers for Medical Report Synthesis: A Multimodal Approach

The efficacy of large language models has revolutionized the healthcare domain by improving medical records documentation, report generation, question answering, and retrieval-augmented generation. Traditional models are equipped to process text-based input, limiting their effectiveness in the presence of multimodal data such as healthcare images, unstructured reports, and prompts. This research proposes a multimodal large language model augmented using RAG for automated report generation in healthcare. A BLIP model integrated with the LoRA fine-tuned T5 transformer has been developed to interpret user-driven prompts and X-ray images to synthesize accurate, and context-aware clinical reports. The model shows high report generation accuracy using various evaluation metrics (Bleu - 0.11, Meteor - 0.39, Rouge-L - 0.58). By leveraging visionlanguage models and transformer architectures,the proposed MLLM integrates textual, and visual data to enhance diagnostic accuracy and reduce physician workload.