About Sharpness, Rank, Artifacts and CNR metrics in 2020 challenge

[mclvnot]

Hello,

in Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction, there are extra evaluation metrics like rank, sharpness and artifacts. (All of us know CNR). Which one were used for sharpness (there are some sharpness metrics using gradients), or
for rank does it depend on radiologists votes or is it Wilcoxon signed rank?, or how could I evaluate my results with artifact metrics?

Best,

[mmuckley]

Hello @mclvnot, I assume you are referring to the results in Table IV.

The ranks are not based on actual metrics, but rather radiologist Likert score assessments of the quality of the images. For the radiologist eval, we used the following:

1=extremely sharp boundaries between areas
2= clear boundaries between areas with only minor blurring that is not bothersome
3= blurring at boundaries that decreases overall image quality
4 = unacceptable blurring for standard clinical imaging

[mclvnot]

Thank you Matthew,

But I actually want to ask the artifact and sharpness metrics. According to your answer, for artifact and sharpness, ragiologists evaluate the examinations by means of likert score. But about Johnsons last RSNA article, they evaluate the reconstruction results in terms of sharpness and artifact,too. That's why, I thought there could be a standart metric for sharpness and artifact. (For sharpnes, average gradient magnitude etc. For artifact, I don't know).

Thank you,

[mclvnot]

Hello again,

I want to ask a question,too. In radiologists evaluation, how were images shown to the radiologists? With fully sampled and reconstructed slices side by side? or just only reconstructed slices which radiologists did not know anything about acceleration factor, reconstruction modality, fully sampled version of it?

Thank you!

[mmuckley]

Hello @mclvnot, we used an online DICOM viewer and uploaded the images to a database for the radiologists to access. Radiologists had knowledge of which image was the ground truth and they could display the images however they wanted using the viewer. The viewer had most of the functionality that they work with as part of their clinical routine.

[mclvnot]

RadiAnt Dicom Viewer? Yes, I heard it before. I think you mean in radiologists assesment part, each radiologists could see both the reconstruction results and ground truth by knowing which they belongs to? Firstly, I was doubtful about bias in evaluation because of seeing ground truth. However, your way made me lightened up for my preperation.

Thank you!

[mmuckley]

@mclvnot I don't think it was that one. I think there are quite a few companies that are offering online viewers now.

[mclvnot]

Ok, then. Thank you for your inform!