Reading List for Multimodal Learning in Healthcare

Table of Contents

• Reading List for Multimodal Learning in Healthcare
  • Table of Contents
  • Papers
    • Survey papers
    • EHR + images
    • EHR + notes
    • EHR + signals
    • More than two
  • Framework
  • Tutorials
  • Courses
  • Workshops
  • Labs

Papers

Survey papers

• Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. npj Digit. Med. 3, 136 (2020). html
• Behrad, F. & Saniee Abadeh, M. An overview of deep learning methods for multimodal medical data mining. Expert Systems with Applications 200, 117006 (2022). html

EHR + images

• Khader, F. et al. Multimodal Deep Learning for Integrating Chest Radiographs and  Clinical Parameters: A Case for Transformers. Radiology 309, e230806 (2023).html
• Barros, V. et al. Virtual Biopsy by Using Artificial Intelligence-based Multimodal Modeling of Binational Mammography Data. Radiology 220027 (2022) pdf
• Qiu, S. et al. Multimodal deep learning for Alzheimer’s disease dementia assessment. Nature Communications 17 (2022). pdf
• Xu, M. et al. Accurately Differentiating Between Patients With COVID-19, Patients With Other Viral Infections, and Healthy Individuals: Multimodal Late Fusion Learning Approach. J Med Internet Res 23, e25535 (2021). pdf
• Mei, X. et al. Artificial intelligence–enabled rapid diagnosis of patients with COVID-19. Nat Med 26, 1224–1228 (2020).html

EHR + notes

• Silva, J. F. & Matos, S. Modelling patient trajectories using multimodal information. J Biomed Inform 134, 104195 (2022).html
• Liu, S. et al. Multimodal Data Matters: Language Model Pre-Training Over Structured and Unstructured Electronic Health Records. IEEE J Biomed Health Inform PP, (2022).pdf
• Darabi, S., Kachuee, M., Fazeli, S. & Sarrafzadeh, M. TAPER: Time-aware patient EHR representation. IEEE Journal of Biomedical and Health Informatics 24, 3268–3275 (2020).pdf

EHR + signals

• Sundrani, S. et al. Predicting patient decompensation from continuous physiologic monitoring in the emergency department. Npj Digit Med 6, 1–10 (2023).html
• Kim, H. B. et al. Computational signatures for post-cardiac arrest trajectory prediction: Importance of early physiological time series. Anaesth Crit Care Pa 41, 101015 (2022).html
• Chen, H., Lundberg, S. M., Erion, G., Kim, J. H. & Lee, S.-I. Forecasting adverse surgical events using self-supervised transfer learning for physiological signals. npj Digit. Med. 4, 1–13 (2021).html
• Feng, Y. et al. DCMN: Double Core Memory Network for Patient Outcome Prediction with Multimodal Data. IEEE International Conference on Data Mining (ICDM) 10 (2019).html
• Xu, Y., Biswal, S., Deshpande, S. R., Maher, K. O. & Sun, J. RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data. in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 2565–2573 (ACM, 2018).html

More than two

• Soenksen, L. R. et al. Integrated multimodal artificial intelligence framework for healthcare applications. npj Digit. Med. 5, 149 (2022).html
• Boehm, K. M. Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer. Nature Cancer 3, 24 (2022).html
• Golovanevsky, M., Eickhoff, C. & Singh, R. Multimodal attention-based deep learning for Alzheimer’s disease diagnosis. J Am Med Inform Assn ocac168 (2022) pdf

Framework

• HAIM 2022
  • GitHub - lrsoenksen/HAIM: This repository contains the code to replicate the data processing, modeling and reporting of our Holistic AI in Medicine (HAIM) Publication in Nature Machine Intelligence (Soenksen LR, Ma Y, Zeng C et al. 2022).
• Multimodal Alzheimer’s Disease Diagnosis framework(MADDi) 2022
  • GitHub - rsinghlab/MADDi: This repository is for the Multimodal Alzheimer’s Disease Diagnosis framework (MADDi).

Tutorials

• MML TutorialCMU

Courses

• MultiModal Machine LearningCMU

Workshops

• Machine Learning for Multimodal Healthcare DataICML 2023

Labs

• CMU MultiComp lab