Automated Transformation of Unstructured Cardiovascular Diagnostic Reports into Structured Datasets Using Sequentially Deployed Large Language Models

Abstract Background Rich data in cardiovascular diagnostic testing are often sequestered in unstructured reports, limiting their use. Methods We sequentially deployed generative and interpretative open-source large language models (LLMs; Llama2-70b, Llama2-13b). Using Llama2-70b, we generated varying formats of transthoracic echocardiogram (TTE) reports from 3000 real-world reports with paired structured elements. Using prompt-based supervised training, we fine-tuned Llama2-13b using sequentially larger batches of generated TTE reports as inputs, to extract data across 18 clinically-relevant echocardiographic fields. We evaluated the fine-tuned model, HeartDX-LM, on distinct datasets: (i) different time periods and formats at Yale New Haven Health System (YNHHS), (ii) Medical Information Mart for Intensive Care (MIMIC) III, and (iii) MIMIC IV. We used accuracy and Cohen's Kappa as evaluation metrics and have publicly released the HeartDX-LM model. Results HeartDX-LM was trained on 2,000 synthetic reports with varying formats and paired structured labels. We identified a lower threshold of 500 unstructured reports – structured data pairs required for fine-tuning to achieve consistent performance. At YNHHS, HeartDX-LM accurately extracted 69,144 of 70,032 values (98.7%) across 18 fields in the contemporary test set where paired structured data were available. In 100 older YNHHS reports, HeartDX-LM achieved 87.1% accuracy against expert annotations. In external validation sets from MIMIC-III and MIMIC-IV, HeartDX-LM correctly extracted 615 of 707 available values (87.9%) and 201 of 220 available values (91.3%), from 100 random, expert-annotated reports from each set. Conclusion We developed and validated a novel approach using paired large and moderate-sized LLMs to transform free-text echocardiographic reports into tabular datasets.