CE/CZ-7454 A1: PSG Relation Classification

[Jingkang50]

Feel free to discuss your assignments here.

[SLU010]

For this assignment 1, is it ok to just retrain and test the model of this paper 'Panoptic Scene Graph Generation'? Thanks.

Feel free to discuss your assignments here.

[SLU010]

Thanks for the reply. So what is the criterion to judge whether a pretrain model can be used? According to dataset size that is not larger than COCO? But CLIP is pretrained on a extremely large dataset and it is allowed.

[Jingkang50]

The restrictions on full-PSG pretrained model is that one could simply do some post-processing to get the results without training. You can use COCO / ImageNet-pretrained or CLIP model, as they do not model relations explicitly.

[SLU010]

Oh, I got it, many thanks for your explanation!

[SLU010]

Hi, for the length of the report, is the reference counted?

[Jingkang50]

The maximum length is 4-page without reference. You can write 4 pages with reference counted.

[SLU010]

Hi, may I know the label of test data is from 0-49 or 6-55? since it only has 50 classes, should we change the Dataset class of the provided simple baseline? Thanks!

[Jingkang50]

It's still 56 classes but there will not be any label from the first 6 classes. So it's 6-55

[fengchengzeng]

Hi, may I know how can I obtain the file 'psg.json', in which the 133 thing/stuff category names are provided? I cannot find it in neither the project directories nor the dataset download link

[Jingkang50]

You shall replace psg.json with either psg_cls_basic.json, and psg_cls_advanced.json to explore.

[Sidney2408]

Hi Jingkang, if I know a professor who has done a similar (not exactly the same) research task, is there any problems if I read his publication and ask him about his approach? I will implement the model and training by myself of course.

[Jingkang50]

Sure, that's more than welcome and you shall cite his publication in your report.

[Sidney2408]

Hi, does anyone have issues with memory leaks when running the baseline code? @Jingkang50 what kind hardware was this dataset trained on?

[Jingkang50]

You can explore the discussion part and try to figures out the solution. The code should be able to run on colab if you made the modifications to solve the memory issue.

[ambigeV]

You can explore the discussion part and try to figures out the solution. The code should be able to run on colab if you made the modifications to solve the memory issue.

Any suggestions on modifications aside from applying "del" command, and reducing model size and batch size?

[ambigeV]

RuntimeError: DataLoader worker is killed by signal: killed.

This error happened in the first epoch no matter how large the batch size is: 16/8/4/2/1

My device is 2080Ti, and I change the baseline resnet50 to resnet34, and apply the solution of "del" before every forward operation, but the problem is still there.

[Jingkang50]

The position of your del could be incorrect. You could try to explore further.

[Sidney2408]

You may want to scrutinize the training code to see if the gradient is accumulating anywhere. Try keep track of your GPU ram usage, see if you can identify any bottlenecks, resnet50 requires a lot of RAM. Number of workers in the dataloader can also be reduced to prevent memory leaks.

By the way, may I ask what you mean by "apply the solution of "del" before every forward operation,"?

Another thing you can try is to implement the training loop from scratch.

[ambigeV]

You may want to scrutinize the training code to see if the gradient is accumulating anywhere. Try keep track of your GPU ram usage, see if you can identify any bottlenecks, resnet50 requires a lot of RAM. Number of workers in the dataloader can also be reduced to prevent memory leaks.

By the way, may I ask what you mean by "apply the solution of "del" before every forward operation,"?

Another thing you can try is to implement the training loop from scratch.

for each epoch, del batch["data"] after self.net(data) and del batch["soft labels"] after computing loss?

[Sidney2408]

@Jingkang50 may I know what the purpose of the file "grade.py" in "OpenPSG/ce7454/"? How do we use it?

[Jingkang50]

The grade.py is the program running in the grading server, only for your reference.

[Sidney2408]

Hi, does anyone have this issue when submitting? I submitted the zip file according to the submission.zip format specified by the competition, and I got this error :
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
File "/tmp/codalab/tmpYAFT80/run/program/grade.py", line 54, in
main()
File "/tmp/codalab/tmpYAFT80/run/program/grade.py", line 40, in main
with open(os.path.join(submit_dir, 'result.txt'), 'r') as reader:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/codalab/tmpYAFT80/run/input/res/result.txt'

[RoyalSkye]

You may need to rename your file as "result.txt" before zipping.

[TonnyTran]

Hi Prof @Jingkang50 , I try to run OpenPSG code but it produces error on testing and validating step.

Traceback (most recent call last):
File "tools/test.py", line 244, in
main()
File "tools/test.py", line 208, in main
args.show_score_thr)
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/mmdet/apis/test.py", line 26, in single_gpu_test
for i, data in enumerate(data_loader):
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in next
data = self._next_data()
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1085, in _next_data
return self._process_data(data)
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1111, in _process_data
data.reraise()
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/torch/_utils.py", line 428, in reraise
raise self.exc_type(msg)
IsADirectoryError: Caught IsADirectoryError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
data = fetcher.fetch(index)
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/mmdet/datasets/custom.py", line 214, in getitem
return self.prepare_test_img(idx)
File "/home3/theanhtran/02_subject/DataScience_A1/OpenPSG/openpsg/datasets/psg.py", line 289, in prepare_test_img
return self.prepare_train_img(idx)
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/mmdet/datasets/custom.py", line 239, in prepare_train_img
return self.pipeline(results)
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/mmdet/datasets/pipelines/compose.py", line 41, in call
data = t(data)
File "/home3/theanhtran/02_subject/DataScience_A1/OpenPSG/openpsg/datasets/pipelines/loading.py", line 169, in call
results = super().call(results)
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/mmdet/datasets/pipelines/loading.py", line 494, in call
results = self._load_masks_and_semantic_segs(results)
File "/home3/theanhtran/02_subject/DataScience_A1/OpenPSG/openpsg/datasets/pipelines/loading.py", line 137, in _load_masks_and_semantic_segs
img_bytes = self.file_client.get(filename)
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/mmcv/fileio/file_client.py", line 993, in get
return self.client.get(filepath)
File "/home3/theanhtran/env/anaconda/envs/A1_DataScience/lib/python3.7/site-packages/mmcv/fileio/file_client.py", line 518, in get
with open(filepath, 'rb') as f:
IsADirectoryError: [Errno 21] Is a directory: './data/coco/'

I found that attribute results['ann_info']['seg_map'] is empty in results. which causes this error. Can you show me how to fix this problem.
def _load_masks_and_semantic_segs(self, results):
...
filename = osp.join(results['seg_prefix'],
results['ann_info']['seg_map'])
...

OpenPSG/openpsg/datasets/pipelines/loading.py

Line 127 in 23f9c89

filename = osp.join(results['seg_prefix'],

Thank you!

[Jingkang50]

If you target to CE7454 assignment 1, please use https://github.com/Jingkang50/OpenPSG/tree/main/ce7454

[Sidney2408]

Hi @Jingkang50, I was wondering about the format of the report. Is there a page limit for it?

[zongqi001]

[RuntimeError: CUDA error: device-side assert triggered]

Hi all, I keep running into the following error in the evaluator (no issue for trainer):

/opt/conda/conda-bld/pytorch_1656352660876/work/aten/src/ATen/native/cuda/Loss.cu:118: operator(): block: [0,0,0], thread: [63,0,0] Assertion `input_val >= zero && input_val <= one` failed.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_3273057/2795977669.py in <module>
      5 for epoch in range(0, args.epoch):
      6     train_metrics = trainer.train_epoch()
----> 7     val_metrics = evaluator.eval_recall(val_dataloader)
      8 
      9     # show log

~/OpenPSG-main/ce7454/evaluator.py in eval_recall(self, data_loader)
     28                 target = batch['soft_label'].cuda()
     29                 loss = F.binary_cross_entropy(prob, target, reduction='sum')
---> 30                 loss_avg += float(loss.data)
     31                 # gather prediction and gt
     32                 pred = torch.topk(prob.data, self.k)[1]

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Can i check if anyone has met the same issue before and has a solution?

[Sidney2408]

Hi @Jingkang50 do you know if the assignment submission in NTULearn allows multiple resubmissions? In case we have any last minute updates?

[Jingkang50]

You can submit several times

[TonnyTran]

Dear Prof. @Jingkang50 , NTU learn has problem in last minutes of submitting assignment 1. It was 23:56. Now I cannot upload my report. Could you please give me a chance to submit

[Jingkang50]

email me

[TonnyTran]

Yes. Thank you very much. I much appreciate!!!

[TonnyTran]

Hi @Jingkang50 , should I send an email to this address jingkang001@e.ntu.edu.sg ? Thanks!

[TonnyTran]

I already sent. Thank you for receiving my report!!!