Skip to content

Codes for "Prescribing Large Language Models for Perioperative Care: What’s The Right Dose for Pretrained Models?"

Notifications You must be signed in to change notification settings

cja5553/LLMs_in_perioperative_care

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

27 Commits
 
 
 
 

Repository files navigation

Prescribing Large Language Models (LLMs) for Perioperative Care: What’s The Right Dose for Pretrained Models?

Our best performing finetuned models are available at 🤗 Huggingface

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("cja5553/BJH-perioperative-notes-bioClinicalBERT")
model = AutoModel.from_pretrained("cja5553/BJH-perioperative-notes-bioClinicalBERT")
from transformers import BioGptTokenizer, AutoModelForCausalLM
model=AutoModelForCausalLM.from_pretrained("cja5553/BJH-perioperative-notes-bioGPT")
tokenizer = BioGptTokenizer.from_pretrained("microsoft/biogpt")

Goal:

  • Experiment the use of pretrained LLMs across different fine-tuning strategies in surgical outcomes of Perioperative Care.

  • The following strategies were experimented:

    1. using pretrained models alone
    2. applying finetuning
    3. applying semi-supervised fine-tuning with the labels
    4. foundational model where a multi-task learning strategy was employed.
  • 3 primary models were used for prediction

    1. bioGPT
    2. ClinicalBERT
    3. bioclinicalBERT.

Dataset:

  • We used 84,875 clinical notes from patients spanning the Barnes Jewish Center Hospital (BJC) hospital system in St Louis, MO.

    • The following outcomes were used:
      1. Death in 30 days
      2. Deep vein thrombosis (DVT)
      3. pulmonary embolism (PE)
      4. Pneumonia
      5. Acute Knee Injury
      6. delirium
  • Characteristics:

    • vocabulary size 3203
    • averaging 8.9 words per case,
    • all single sentenced clinical notes

To use:

  • You should be able to run the codes as it is on the Jupyter notebook files provided (of course with your own dataset)
  • For the semi-supervised and foundational version, you may need to clone the transformers package from huggingface's github profile and slot the relevant files in the same folders of which they appear in the local_transformers folders of this github repo.

Questions?

Contact me at alba@wustl.edu

About

Codes for "Prescribing Large Language Models for Perioperative Care: What’s The Right Dose for Pretrained Models?"

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published