Cannot Complete Training After Resuming about hub HOT 4 CLOSED

👋 Hello @spielbrecher, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

from hub.

@spielbrecher hello there!

Thank you for reaching out with details about the issue you're encountering during your model training. It seems there's a discrepancy between the epochs reported by the training progress and the checkpoints being saved. Usually, a new checkpoint should be saved at regular intervals or after certain epochs have been completed.

Considering the graphics page displays results for 100 epochs, it's reassuring that the model has indeed completed the training. However, the checkpoints not updating beyond the 86th epoch is unusual. Let's try to troubleshoot the issue:

1. Ensure the training session has enough resources to run without interruptions and is allowed to save new files (check disk space and permissions).
2. Have a look at the training logs for any errors or warnings that occurred after epoch 86—these messages could provide valuable insights into why later checkpoints are not being saved.
3. Verify the configuration settings for checkpoint saving intervals to see if there has been any customization that might be affecting the expected behavior.
4. If your model's progress graphics indicate that the model has learned adequately by epoch 86, consider utilizing the last saved checkpoint to export your model. This could be especially relevant if the subsequent epochs have not resulted in significant improvements.

Should none of these steps identify or solve the problem, it might be worth considering reaching out for further assistance with detailed information about your specific setup. You may also attempt to manually trigger a checkpoint save, but without including code, I'll guide you towards checking the training scripts' section responsible for checkpoints to ensure it's operating correctly.

We appreciate your cooperation and patience as we aim to resolve this matter. Best of luck with your model training, and I'm here to assist you further if needed!

from hub.

@spielbrecher Are you able to provide us with the model id? You can find this in the URL.

It seems like something has gone wrong during the resume. Your model may have resumed but never achieved a better mAP and therefore could not upload the weights at the end of training.

from hub.

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

from hub.