Missing key(s) in state_dict: "model.0.conv.quant.activation_post_process.scale" about sparseml HOT 5 CLOSED

Could it be that you cannot run on GPU when training using sparseml.ultralytics.train ? It seems that when I switch to a CPU kaggle environment that I am able to train the model.

from sparseml.

I got it to work eventually. I seems that running it distributed wasn't favourable. Probably adding the required pre-statement in front of it could fix that, but for now running it on a single GPU works.

from sparseml.

@thijsgelton Hi, Can you please tell me what exactly you did to load the pruned and quantized model successfully? Currently, I have yolov8n that I trained with the quantized recipe but I can not load it, unfortunately.

from sparseml.

Eventually I was able to run it using the following command on kaggle with 2 T4's (so distributed):

subprocess.run([
    "python", "-m", "torch.distributed.run", "--no_python", "--nproc_per_node", "2",
    "sparseml.ultralytics.train",
    "--model",
    "/kaggle/working/runs/detect/train/weights/best.pt",
  '--recipe',
    "zoo:cv/detection/yolov8-m/pytorch/ultralytics/voc/pruned80-none",
  '--data',
    "/kaggle/working/spaceship.yaml", "--batch=32", '--lr0', '0.0015', '--lrf=0.1', '--momentum', '0.85', '--mosaic', '0.95', '--mixup', '0.15', '--scale', '0.9',
'--recipe_args', '{"num_epochs": 50}', "--resume"])

from sparseml.

@thijsgelton, I have trained the model using a quantization recipe. I am getting RuntimeError: Error(s) in loading state_dict for DetectionModel: Missing key(s) in state_dict: "model.0.conv.quant.activation_post_process.scale"...... while converting the trained model to onnx. I could not figure out the problem yet.

from sparseml.