Skip to content

NaN error with high dropout rate

Surui Li edited this page Mar 19, 2021 · 3 revisions

Using the default dropout rate range, 0.1 to 0.9, when doing the hyperparameters search, will sometimes results in NaN values in the network weights and output. 0.9 as the maximum dropout rate is too large, which may result in a layer dropping all its units in a forward pass, causing a NaN value in training. In older version of lifelines, NaN values are incorrectly converted to True when computing c-index. Hence masking the error. However, since version 0.21.0, such errors will be raised, breaking the demo script.

Therefore to reproduce the 4Dsurvival paper, where dropout search range was 0.1 to 0.9, and NaN values were converted to True, use lifelines version of 0.14.6. This corresponds to docker image lisurui6/4dsurvival-gpu:1.0.

To further develop the code, we would suggest to change the dropout range to 0.1 to 0.5, and use the latest lifelines version (0.21.0+). This corresponds to docker image lisurui6/4dsurvival-gpu:1.1.

Clone this wiki locally