This repository contains code and results for RaLEs: Radiology Language Evaluations (manuscript under consideration). Check out the leaderboard!
The repository is organized in 5 modules, each contained within its own directory:
| Module | Purpose |
|---|---|
| datasets | Provides instructions for RaLEs dataset download and preprocessing |
| fine_tuning | Contains command-line interface code for fine-tuning a model for a given RaLEs task |
| inference | Provides code for performing inference on RaLEs tasks given a pre-trained model |
| evaluation | Provides code for obtaining evaluation metrics based on model predictions for RaLEs tasks |
| results | Includes summary of results of current RaLEs benchmark |
First create and activate a conda environment:
conda create -n rales python=3.8.13
conda activate rales
Once created, we recommend first installing pytorch with your appropriate cuda version e.g.:
pip install torch torchvision torchaudio
Finally, install additional requirements:
while read requirement; do conda install --yes $requirement || pip install $requirement; done < requirements.txt
Note: RaLEs has been developed and tested using Ubuntu 22.04.3
Navigate to the datasets directory and follow the instructions in the datasets README for downloading and preprocessing the RaLEs datasets.
To fine-tune a model for a specific RaLEs task, navigate to the fine_tuning directory and use the command-line interface detailed in the fine_tuning README.
After training, you can perform inference on the RaLEs tasks using the pre-trained model. Go to the inference directory and refer to the inference README for detailed instructions.
Evaluate the performance of your model using the scripts in the evaluation directory. More details can be found in the evaluation README.
Check the results directory for a summary of benchmark results for the RaLEs tasks. The results README provides further insights.
For the current leaderboard, visit RaLEs Benchmark Leaderboard.
1. What is the purpose of the RaLEs benchmark?
The RaLEs benchmark is designed to evaluate models on various radiology language tasks. It provides a standardized dataset and evaluation metrics to compare the performance of different models in a consistent manner.
2. How can I submit my model to the leaderboard?
To submit your model to the leaderboard, follow the submission process detailed in the results README. Ensure you provide all the required details in the submission form.
3. Where can I find the datasets used in the benchmark?
The datasets can be accessed from the datasets directory. Detailed instructions on downloading and preprocessing each dataset are provided there.
4. I'd like to contribute a dataset, how can I do that?
New dataset submissions are more than welcome. Full instructions for how to format and submit a dataset can be found in the datasets directory.
5. I have a suggestion for a new feature, how can I share it?
Please use the issues tab in github. Be sure to be specific about what you'd like to see implemented. If you'd like to implement it yourself, you can submit a pull request with the feature implementation, and a brief rationale motivating it. Thank you for your contributions!
@inproceedings{
chaves2023rales,
title={Ra{LE}s: a Benchmark for Radiology Language Evaluations},
author={Juan Manuel Zambrano Chaves and Nandita Bhaskhar and Maayane Attias and Jean-Benoit Delbrouck and Daniel Rubin and Andreas Markus Loening and Curtis Langlotz and Akshay S Chaudhari},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2023},
url={https://openreview.net/forum?id=PWLGrvoqiR}
}