Only return 1 class when training COVIDNet with COVIDx4 dataset

[gwang-kim]

Description

Only return 1 class when finetuning COVIDNet and the loss is exploded.

Steps to Reproduce

I downloaded the COVIDx4 Dataset and tried training COVIDNet with COVIDx4 dataset.
However, when inference, the model returned only one class and the performance was poor.
I reported the loss every step during the training and the loss was exploded to several thousands after 1 epoch.
I try both training from scratch and fine-tuning.
How can I train your model stably?

Expected behavior

the model is trained stably

Actual behavior

the model returned only one class and the performance was poor.

Environment

Ubuntu 18.04
tensorflowgpu 1.15
And I followed the requirements.txt

[sabuj7177]

Hi @gwang-kim , Have you found any solution of this issue? I am also facing similar issue.

This is the training log for starting epochs:

Output: ./output/COVIDNet-lr0.0002
Dataset length
15952
13794 2158
Saved baseline checkpoint
Baseline eval:
[[194. 6.]
[ 9. 191.]]
Sens Negative: 0.970, Positive: 0.955
PPV Negative: 0.956, Positive: 0.970
Training started
1725/1725 [==============================] - 2445s 1s/step
Epoch: 0001 Minibatch loss= 370.629089355
[[200. 0.]
[200. 0.]]
Sens Negative: 1.000, Positive: 0.000
PPV Negative: 0.500, Positive: 0.000
Saving checkpoint at epoch 1
1725/1725 [==============================] - 4858s 3s/step
Epoch: 0002 Minibatch loss= 3805.902343750
[[199. 1.]
[200. 0.]]
Sens Negative: 0.995, Positive: 0.000
PPV Negative: 0.499, Positive: 0.000
Saving checkpoint at epoch 2
1725/1725 [==============================] - 7348s 4s/step
Epoch: 0003 Minibatch loss= 12214.270507812
[[195. 5.]
[199. 1.]]
Sens Negative: 0.975, Positive: 0.005
PPV Negative: 0.495, Positive: 0.167
Saving checkpoint at epoch 3
1725/1725 [==============================] - 9727s 6s/step
Epoch: 0004 Minibatch loss= 28461.550781250
[[200. 0.]
[200. 0.]]
Sens Negative: 1.000, Positive: 0.000
PPV Negative: 0.500, Positive: 0.000
Saving checkpoint at epoch 4

This is the command i used(it is from training instruction):

python train_tf.py
--weightspath models/COVIDNet-CXR-2
--metaname model.meta
--ckptname model
--n_classes 2
--trainfile labels/train_COVIDx8B.txt
--testfile labels/test_COVIDx8B.txt
--out_tensorname norm_dense_2/Softmax:0
--logit_tensorname norm_dense_2/MatMul:0

Environment:
Ubuntu 20.04LTS
tensorflow-gpu

I build the dataset by following the dataset generation instructions.

Thank you.

[gwang-kim]

Hi @sabuj7177,
Oh, it's almost the same situation as mine.
Unfortunately, I didn't found any solution.. I controlled the hyperparams such as LR, but it didn't work.

If you solve the problem, please let me know!

Thank you

[sabuj7177]

Hi @lindawangg @haydengunraj,
Can you please suggest any workaround of this issue? Can you please suggest what I am doing wrong?

[SmallFan7]

Hi @gwang-kim , Have you found any solution of this issue? I am also facing similar issue.

This is the training log for starting epochs:

Output: ./output/COVIDNet-lr0.0002 Dataset length 15952 13794 2158 Saved baseline checkpoint Baseline eval: [[194. 6.] [ 9. 191.]] Sens Negative: 0.970, Positive: 0.955 PPV Negative: 0.956, Positive: 0.970 Training started 1725/1725 [==============================] - 2445s 1s/step Epoch: 0001 Minibatch loss= 370.629089355 [[200. 0.] [200. 0.]] Sens Negative: 1.000, Positive: 0.000 PPV Negative: 0.500, Positive: 0.000 Saving checkpoint at epoch 1 1725/1725 [==============================] - 4858s 3s/step Epoch: 0002 Minibatch loss= 3805.902343750 [[199. 1.] [200. 0.]] Sens Negative: 0.995, Positive: 0.000 PPV Negative: 0.499, Positive: 0.000 Saving checkpoint at epoch 2 1725/1725 [==============================] - 7348s 4s/step Epoch: 0003 Minibatch loss= 12214.270507812 [[195. 5.] [199. 1.]] Sens Negative: 0.975, Positive: 0.005 PPV Negative: 0.495, Positive: 0.167 Saving checkpoint at epoch 3 1725/1725 [==============================] - 9727s 6s/step Epoch: 0004 Minibatch loss= 28461.550781250 [[200. 0.] [200. 0.]] Sens Negative: 1.000, Positive: 0.000 PPV Negative: 0.500, Positive: 0.000 Saving checkpoint at epoch 4

This is the command i used(it is from training instruction):

python train_tf.py --weightspath models/COVIDNet-CXR-2 --metaname model.meta --ckptname model --n_classes 2 --trainfile labels/train_COVIDx8B.txt --testfile labels/test_COVIDx8B.txt --out_tensorname norm_dense_2/Softmax:0 --logit_tensorname norm_dense_2/MatMul:0

Environment: Ubuntu 20.04LTS tensorflow-gpu

I build the dataset by following the dataset generation instructions.

Thank you.

I have the same situation as you. Is the problem solved now？

[gwang-kim]

@SmallFan7 Not yet, I think it's just the limitation of this work.