Welcome to the repository! This serves as an example showcasing the seamless integration of the Valohai MLOps platform. Here, we demonstrate the implementation of YOLOv5 with the Valohai platform. Additionally, we provide insights into utilizing YOLOv8 as a library in conjunction with Valohai.
We've preserved the original YOLO code while leveraging the true magic within the valohai.yaml file, where essential configurations are defined.
We hope this serves as a helpful resource for your MLOps journey!
Login to the Valohai app and create a new project.
Using UI
Configure this repository as the project's repository, by following these steps:
- Go to your project's page.
- Navigate to the Settings tab.
- Under the Repository section, locate the URL field.
- Enter the URL of this repository.
- Click on the Save button to save the changes.
Using terminal
To run your code on Valohai using the terminal, follow these steps:
- Install Valohai on your machine by running the following command:
pip install valohai-cli valohai-utils- Log in to Valohai from the terminal using the command:
vh login- Create a project for your Valohai workflow. Start by creating a directory for your project:
mkdir valohai-yolo-example
cd valohai-yolo-exampleThen, create the Valohai project:
vh project create- Clone the repository to your local machine:
git clone https://github.com/valohai/yolo-example.git .Now you are ready to run executions and pipelines.
This repository covers essential tasks such as training, prediction, and evaluation using YOLOv5 and YOLOv5-seg models. Additionally, we provide the Ultralytics example demonstrating the usage of YOLOv8 with Valohai, including training, evaluation, prediction, and ONNX exporting.
Using UI
- Go to the Executions tab in your project.
- Create a new execution by selecting the predefined steps: train-yolov5, predict-yolov5, validate-yolov5, train-seg-yolov5, predict-seg-yolov5, validate-seg-yolov5, run-yolov8.
- Customize the execution parameters if needed.
- Start the execution to run the selected step.
Using terminal
To run individual steps, execute the following command:
vh execution run <step-name> --adhocFor example, to run the preprocess-dataset step, use the command:
vh execution run train-yolov5 --adhocYou can also set parameters for your execution (see what paramenters you can set in valohai.yaml):
vh execution run train-yolov5 --epochs=10 --adhocUsing UI
- Navigate to the Pipelines tab.
- Create a new pipeline and select out pipeline called train-val-pipeline-yolov5.
- Configure the pipeline settings.
- Start the pipeline.
Using terminal
To run pipelines, use the following command:
vh pipeline run <pipeline-name> --adhocFor example, to run our pipeline, use the command:
vh pipeline run train-val-pipeline-yolov5 --adhocDuring the execution of the code, you can monitor the graphs generated with the metadata. Additionally, once the execution is complete, you can access the output files containing the results.
To ensure the integrity of the original YOLOv5 repository, we decided not to modify it. However, we needed to connect the dataset to our S3 bucket by adjusting the file path.
We created our own version of the coco128.yaml file named datasets/coco128.yaml and datasets/coco128-seg.yaml for segmentation.
In this customized YAML files, we set the dataset path to /valohai/repository/coco128/ and /valohai/repository/coco128-seg/.
To streamline the process, we made use of Valohai inputs.
Configuring the valohai.yaml file, we defined default input datasets that include the link to our S3 bucket.
When executing the code, Valohai automatically handles the downloading and caching of the dataset, ensuring seamless integration with our workflow.
In order to make this repository accessible to all Valohai users, we have utilized a .tar dataset instead of a wildcard with the S3 bucket. This decision was made because public S3 buckets do not support wildcard functionality.
Therefore, in the valohai.yaml file, you will notice that we use the link to coco128.tar file as the default dataset input and employ the tar command (tar -xf /valohai/inputs/datasets/coco128.tar) in the command section to extract the dataset.
-
Create your own S3 bucket. Follow these steps.
-
Upload your dataset to the S3 bucket.
-
Add S3 bucket to your Data Store. To include an S3 bucket as your Data Store, navigate to the project's settings, then select Data Stores. From there, you can add a new store by choosing Amazon S3.
-
Set the link to your S3 bucket as the default dataset in
valohai.yaml, add it to the inputs section of the relevant step. Which should look like this:s3://data-yolov5/datasets/coco128/* -
Change the link to dataset in
datasets/coco128.yamlfile. Setpath:/valohai/inputs/datasets/ -
Delete the
tar -xf /valohai/inputs/datasets/coco128.tarfrom command section invalohai.yaml
The valohai_watch.py script is a valuable tool for monitoring the results of a training process while the code is still running. It serves several purposes:
-
Continuous Observation: The script utilizes the watchdog library to continuously monitor a specified directory for changes. This enables real-time observation of the evolving results of your training.
-
CSV to JSON Conversion: The script reads and extracts metadata, when CSV files are modified. This metadata is converted to JSON, enabling us to create visualizations such as charts or graphs on the Valohai page.
Overall, we use valohai_watch to convert training, testing, and validation metadata into a format that Valohai can read. While Valohai doesn't usually need monitoring scripts, we included this example to show how it works without changing the original YOLO code.
For bug reports and feature requests please visit GitHub Issues.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent