Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Object detection locally stuck in training phase on tutorial dataset #2887

Open
BigDaddy1337 opened this issue Mar 18, 2024 · 3 comments
Open
Labels
GS Tutorial Bug from customer using Getting Started Tutorial

Comments

@BigDaddy1337
Copy link

System Information (please complete the following information):

  • Model Builder or CLI Version: 17.18.2.2415501
  • Visual Studio Version (if applicable): 2022 version 17.9.3

Describe the bug

  • On which page of the tutorial did you run into an issue (URL): https://devblogs.microsoft.com/dotnet/object-detection-ml-dotnet-model-builder/ (Step 5 of topic "Get started with object detection locally")
  • Clear description of the problem: Training locally gets stuck in the training stage, waited about 6 hours, tried CPU (Ryzen 5 7600X with 32GB ram) or GPU (RTX 3070) environment, nothing changes

Logs

2024-03-18 22:32:57.5620 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:57.5620 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:58.5660 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:58.5660 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:59.5760 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:32:59.5760 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:00.5960 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:00.5960 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:01.6030 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:01.6030 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:02.6175 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:02.6175 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:03.6262 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:03.6262 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:04.6287 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:04.6287 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:05.6395 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:05.6395 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:06.6409 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:06.6409 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:07.6492 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:07.6492 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:08.6499 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:08.6499 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:09.6540 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:09.6540 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:10.6595 DEBUG update TrainViewModel_159.RemainingSeconds (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2024-03-18 22:33:10.6595 DEBUG update TrainViewModel_159.RemainingTime (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)

To Reproduce
Steps to reproduce the behavior: "Get started with object detection locally" https://devblogs.microsoft.com/dotnet/object-detection-ml-dotnet-model-builder/

Expected behavior
With my hardware spec and 40 images with one category detection train phase should take a few minutes

Screenshots
image
image

Additional context
Classification scenario works as expected locally with same dataset

image

@BigDaddy1337 BigDaddy1337 added the GS Tutorial Bug from customer using Getting Started Tutorial label Mar 18, 2024
@LittleLittleCloud
Copy link
Contributor

Would you mind sharing the stop sign dataset with us

@BigDaddy1337
Copy link
Author

@LittleLittleCloud check public repo https://github.com/BigDaddy1337/temp/tree/main - images and vott labeling configs

@vulvquang
Copy link

while training, check your GPU usage, if nothing or small percentage is increased, the system might need to download a file. If a file required, please see #2892

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GS Tutorial Bug from customer using Getting Started Tutorial
Projects
None yet
Development

No branches or pull requests

3 participants