ValueError: not enough values to unpack (expected 4, got 0)

[hjafar]

Dear Marvin,

I would like to use your tcga_segmentation tool for tumor tissue prediction of primary breast cancer. I have finished the preprocessing step without any issues. I used the following command in this step:
python -m code.data_processing.main --gdc-executable '/home/hussain/Documents/breast_cancer/gdc-client_v1.6.1_Ubuntu_x64/gdc-client' --manifest '/home/hussain/Documents/breast_cancer/manifest_test/gdc_manifest_20211111_062451.txt' '/home/hussain/Documents/breast_cancer/manifest_test'

The manifest file above contains 10 tumor and 10 normal WSIs in TCGA portal. All images were downloaded and patches, labels and summary text files were created for each WSI as a result of the above command.

However, I then ran the training step as follows:
python -m code.training --source-slides-folder '/home/hussain/Documents/breast_cancer/manifest_test' --alpha 0.1 --beta 0. --max-bag-size 100 --no-download

The above command fails and throws the following error message:
ValueError: not enough values to unpack (expected 4, got 0)

Would you please explain why I am getting this error and what could have gone wrong?
Best,
Hussain

[MarvinLer]

Hi Hussain,
Seems like no data were fetched at some point. Hard to tell without the whole stack of logs and errors, could you copy/paste the complete output from your second command please (the python -m code.training ...)? In particular, I would need the logs from the program (that includes the logs from the fetched data) and the full error stack (indicating where the ValueError was raised).

Best,
ML

[hjafar]

Dear Marvin,

Thank you for your response. Kindly find the following command with the complete run output.

$ python -m code.training --source-slides-folder '/home/hussain/Documents/breast_cancer/manifest_test' --alpha 0.1 --beta 0. --max-bag-size 100 --no-download

INFO Hyper parameters
INFO { 'alpha': 0.1,
'beta': 0.0,
'cuda': False,
'dataset_max_size': None,
'do_download': False,
'early_stopping_patience': 10,
'gdc_executable': None,
'input_data_folder': './data',
'learning_rate': 0.0005,
'manifest_file': None,
'max_bag_size': 100,
'models_save_folder': './saved_models/',
'n_epochs': 20,
'n_timesteps_per_epoch': None,
'save_model': True,
'save_model_timesteps': 5,
'seed': 123,
'source_slides_folder': '/home/hussain/Documents/breast_cancer/manifest_test',
'test_size': 0.15,
'underlying_model_load_from': None,
'underlying_model_pretrained': False,
'underlying_model_type': 'resnet18',
'val_size': 0.1,
'verbose': False,
'weight_decay': 0.0001,
'with_data_augmentation': True,
'with_tensorboard': True}
INFO Data_processing control parameters:
INFO do_download: False
INFO gdc_executable: None
INFO manifest_filepath: None
INFO source_slides_folder: /home/hussain/Documents/breast_cancer/manifest_test
INFO Meta-parameters:
INFO desired_magnification 20
INFO tile_width 224
INFO expected_tile_shape (224, 224, 3)
INFO background_threshold 0.75
INFO background_pixel_value 220
INFO Performing no download from TCGA as requested; listing source slides from /home/hussain/Documents/breast_cancer/manifest_test
INFO Slides already tiled at magnification 20 -> skipping
INFO Extracting labels...
Traceback (most recent call last):
File "/home/hussain/anaconda3/envs/tcga/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/hussain/anaconda3/envs/tcga/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/hussain/Documents/breast_cancer/tools/tcga_segmentation/code/training.py", line 343, in
main(define_args())
File "/home/hussain/Documents/breast_cancer/tools/tcga_segmentation/code/training.py", line 250, in main
hyper_parameters['source_slides_folder'])
File "/home/hussain/Documents/breast_cancer/tools/tcga_segmentation/code/data_processing/main.py", line 136, in main
output_tiles_folders, svs_filenames, md5sums, cases_ids = list(map(list, zip(*tiled_content)))
ValueError: not enough values to unpack (expected 4, got 0)

The manifest_test folder contains the following list files/folders:
Folder: downloaded_slides
Folder: filtered_tiles
file: gdc_manifest_20211111_062451.txt
file: has_been_downloaded
file: has_been_moved_and_filtered
file: has_been_tiled_mag20

Best,
Hussain

[hjafar]

Dear Marvin,
This is kind reminder that I am still awaiting for your response regarding the above error.
Best regards,
Hussain

[RorryB]

Hey @hjafar, if you're still looking at this. I also got that error and the issues was just that I had a newline at the end of the manifest that was reading in an empty line as the final entry