training stuck without response #5637
Comments
|
👋 Hello @huanghaifeng1234, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution. If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you. If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available. For business inquiries or professional support requests please visit https://ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com. RequirementsPython>=3.6.0 with all requirements.txt installed including PyTorch>=1.7. To get started: $ git clone https://github.com/ultralytics/yolov5
$ cd yolov5
$ pip install -r requirements.txtEnvironmentsYOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):
Status
If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit. |
|
@huanghaifeng1234 hi thanks for the bug report! Is this reproducible for you? Because your command will work properly on Google Colab, so we are not able to reproduce. |
Hi glenn, my work space environment is CPU with 2 cores (MAC), after I add "--worker 0" command line argument to "python train.py" script, I get training process started successfully, I wonder why worker number should be 0, because "--worker 2" still caused the same problem. Thanks! |
|
@huanghaifeng1234 like I said if we can not reproduce your issue there is no action for us to take. Currently training operates correctly with any number of workers on all operating systems including MacOS. If you have a reproducible error please let us know. |
Hi glenn, my problem is reproducible on my computer, I find other people encountered the same issue as me here: pytorch/pytorch#1355. Seems like there is some problem with pytorch dataloader's multiprocessing and opencv. Could you please provide your version of opencv and pytorch? Mine is opencv-python=4.3.0.38, torch=1.9.0, torchvision=0.10.0, python=3.7.6 |
1 similar comment
Hi glenn, my problem is reproducible on my computer, I find other people encountered the same issue as me here: pytorch/pytorch#1355. Seems like there is some problem with pytorch dataloader's multiprocessing and opencv. Could you please provide your version of opencv and pytorch? Mine is opencv-python=4.3.0.38, torch=1.9.0, torchvision=0.10.0, python=3.7.6 |
|
@huanghaifeng1234 yes understood. This is the most recent CI check from 17 hours ago: You can select your OS and see the pip package versions for this particular run. |
Thanks! I will try to accord the dependencies and see whether it works. |
|
👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs. Access additional YOLOv5 🚀 resources:
Access additional Ultralytics ⚡ resources:
Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed! Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐! |
|
I face same problem. Training stuck after few epoch, no response. |
|
@Sanjib-ac hello, Thank you for reporting this issue. Can you please provide more information about your setup? What operating system are you using, what are the specs of your machine, and what version of YOLOv5 are you running? Additionally, please provide the command you used to start the training and any error messages you received. This will help us better understand the issue and provide a solution. Thank you, and we look forward to hearing back from you soon. |
|
Hi,
Setup
YOLOv5 v7.0-162-gc3e4e94 Python-3.10.10 torch-1.13.1+cu116 CUDA:0 (NVIDIA
GeForce MX230, 2048MiB)
OS: Windows
Python 3.10.10
Command:
"python train.py --cfg yolov5m.yaml --hyp hyp.scratch-high.yaml --batch -1
--epochs 700 --data test-yolo.yaml --weights '' --name test-yolo"
Thanks,
Sanjib
…On Tue, Jun 6, 2023 at 10:05 AM Glenn Jocher ***@***.***> wrote:
@Sanjib-ac <https://github.com/Sanjib-ac> hello,
Thank you for reporting this issue. Can you please provide more
information about your setup? What operating system are you using, what are
the specs of your machine, and what version of YOLOv5 are you running?
Additionally, please provide the command you used to start the training and
any error messages you received. This will help us better understand the
issue and provide a solution.
Thank you, and we look forward to hearing back from you soon.
—
Reply to this email directly, view it on GitHub
<#5637 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A5WYNLFAKKSX5SUX5ABYS3LXJ42MDANCNFSM5H6ZHSNQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Sanjib Sarkar, Ph.D.
E: ***@***.***
www.acquiredata.com
--
CONFIDENTIALITY NOTICE: This message and any attachments or files
transmitted with it (collectively, the "Message") are intended only for the
addressee and may contain information that is privileged, proprietary
and/or prohibited from disclosure by law or contract. If you are not the
intended recipient: (a) please do not read, copy or retransmit the Message;
(b) permanently delete and/or destroy all electronic and hard copies of the
Message; (c) notify us by return email; and (d) you are hereby notified
that any dissemination, distribution or copying of the Message is strictly
prohibited.
|
|
@Sanjib-ac hello Sanjib, Thank you for providing your setup details and command. It seems like you are using a newer version of Python (3.10.10) which is not yet fully supported by YOLOv5. We recommend using Python version 3.6, 3.7, or 3.8 for better compatibility. Also, in your command, you have specified an invalid value for the batch size parameter. The value "-1" is not a valid batch size and can cause issues during training. Please specify a valid batch size for your model. Finally, please provide any error messages that you receive during training and we will be happy to assist you in resolving the issue. Thank you, |
|
Okay. Thanks for the response. I will train on the recommended python version. |
|
@Sanjib-ac, thank you for your response. Yes, you are correct that a batch size value of -1 refers to the "auto batch size" feature in YOLOv5. This feature automatically adjusts the batch size based on the input image size and available memory. However, during training if you're finding that the training is freezing or hanging, you may want to try setting a fixed batch size and adjust the learning rate accordingly. Thank you for your cooperation and we hope this resolves your issue. Please let us know if you have any further questions or concerns. |
|
Thank you for the insight regarding the batch size value -1. Thanks, |
|
@Sanjib-ac hello Sanjib, You're very welcome, glad we could help clarify the batch size value for you. Please let us know if you have any further questions or concerns regarding YOLOv5. Thank you, |
Search before asking
YOLOv5 Component
Training
Bug
using the sample training script: python train.py --img 640 --batch 16 --epochs 1 --data coco128.yaml --weights yolov5s.pt,
get stuck in the training process, like picture below
Alter stop the program by pressing Ctrl+C, I get such error
^CTraceback (most recent call last): File "train.py", line 630, in <module> File "train.py", line 527, in main # Evolve hyperparameters (optional) File "train.py", line 298, in train if ni <= nw: File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/tqdm/std.py", line 1107, in __iter__ for obj in iterable: File "/Users/geohfhuang/Documents/个人文档/工业视觉检测/repo/yolov5/utils/datasets.py", line 140, in __iter__ for i in range(len(self)): File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in __next__ data = self._next_data() File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data idx, data = self._get_data() File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1152, in _get_data success, data = self._try_get_data() File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data data = self._data_queue.get(timeout=timeout) File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/multiprocessing/queues.py", line 104, in get if not self._poll(timeout): File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/multiprocessing/connection.py", line 257, in poll return self._poll(timeout) File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/multiprocessing/connection.py", line 414, in _poll r = wait([self], timeout) File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/multiprocessing/connection.py", line 920, in wait ready = selector.select(timeout) File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/selectors.py", line 415, in select fd_event_list = self._selector.poll(timeout) KeyboardInterrupt Exception ignored in: <function tqdm.__del__ at 0x1374a5290> Traceback (most recent call last): File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/tqdm/std.py", line 1065, in __del__ File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/tqdm/std.py", line 1248, in close File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/tqdm/std.py", line 564, in _decr_instances File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/site-packages/tqdm/_monitor.py", line 51, in exit File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/threading.py", line 522, in set File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/threading.py", line 365, in notify_all File "/Users/geohfhuang/opt/anaconda3/lib/python3.7/threading.py", line 348, in notify TypeError: 'NoneType' object is not callableEnvironment
Minimal Reproducible Example
Additional
No response
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: