Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error is raise when training tilling ins-seg models with coco subset. #2381

Open
eunwoosh opened this issue Jul 20, 2023 · 0 comments
Open
Assignees

Comments

@eunwoosh
Copy link
Contributor

eunwoosh commented Jul 20, 2023

Describe the bug
When training tilling ins-seg model with "coco decremental dataset" which is used for regression test dataset, an error is raised due to index out of range error in dataset.
Because ins-seg models now use coco pretrained model weight, when using sub dataset of coco, task adaptive algorithm is executed.
Then, ClsIncrSampler replaces existing sampler and ClsIncrSampler decides dataset size using new_indices and old_indices from OTXDetDataset.
This can cause two abnormal cases:

  1. If tilling dataset size is bigger than original dataset size, only original dataset size of tilling dataset is used.
  2. If tilling dataset size is smaller than original dataset , Index out of range error is raised. (this is our case)

I think current tilling algorithm doesn't consider task adapt case. So, code about that needs to be updated, I think.
ps. Truning off adaptive batch size with --learning_parameters.auto_adapt_batch_size None can makes debugging easier.

Steps to Reproduce

  1. training a tilling ins-seg models with subset of coco.

Environment:

  • OS: Ubuntu 20.04
  • Framework version: Torch 1.13
  • Python version: 3.10
  • OpenVINO version: 2023.0
  • CUDA/cuDNN version: 11.7
  • GPU model and memory: RTX3090, 24GB
@eunwoosh eunwoosh changed the title No task adapt algorithm in tilling error is raise when training tilling ins-seg models with coco subset. Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants