[235-controlnet] Fail to prepare calibration datasets about openvino_notebooks HOT 15 CLOSED

@l-bat Can you check, please?

from openvino_notebooks.

@l-bat Yes

from openvino_notebooks.

@l-bat

I tried to use another Machine type on Google Cloud: before I was using e2-highmem-8 but after I changed to n2-highmem-8 it all worked. Also, I used the last version of the tutorial Notebook.

from openvino_notebooks.

@MarkYav, Thanks for reporting the problem. Unfortunately, I'm not able reproduce provided error. Maybe there is something additional specific details?
which datasets, controlnet-aux, pytorch, transformers and diffusers version do you use?

from openvino_notebooks.

@l-bat Thank you for your prompt reply!

On Kaggle:

datasets                                 2.17.1
controlnet_aux                           0.0.7
pytorch-ignite                           0.4.13
pytorch-lightning                        2.1.3
transformers                             4.37.0
diffusers                                0.26.3

The full list could be found here: https://pastebin.com/WGGLrYT5

Also, it runs on Colab Enterprise with installed libs (see the list mentioned below). But the problem is that after quantization the generated image is bad (see photo).

It extracts the pose correctly, tho:

Installed libs Colab Enterprise: https://pastebin.com/U0GtqC4r

from openvino_notebooks.

@MarkYav, what inference device do you use? Looks like a problem with the calibration data or with device.

from openvino_notebooks.

@l-bat I use Colab Enterprise on Google Cloud. There are details:

from openvino_notebooks.

@MarkYav, do you use CPU here?

from openvino_notebooks.

@MarkYav, could you please try downloading the unet_calibration_data.pkl and loading it into notebook instead of preparing a calibration dataset? This will help me to understand if there is problem with data preparation.

import pickle
with open('unet_calibration_data.pkl', 'rb') as f:
    unet_calibration_data = pickle.load(f)

from openvino_notebooks.

@l-bat I tried to run the notebook using the code provided.

I changed:

%%skip not $to_quantize.value

CONTROLNET_INT8_OV_PATH = Path("controlnet-pose_int8.xml")
UNET_INT8_OV_PATH = Path("unet_controlnet_int8.xml")

# These are commented lines:
# if not (CONTROLNET_INT8_OV_PATH.exists() and UNET_INT8_OV_PATH.exists()):
    # unet_calibration_data = collect_calibration_data(ov_pipe, subset_size=subset_size)

# These are new inserted lines:
import pickle
with open('unet_calibration_data.pkl', 'rb') as f:
    unet_calibration_data = pickle.load(f)

And I got an error in the next cell:

I checked the code and the length of the downloaded unet_calibration_data is 20:

In this cell the prev_idx is initially set to 0, and in the loop goes from 0 to 19. But the problem is that later we set prev_idx += num_inference_steps and now prev_idx is 20 and will go from 20 to 39 -- here we have a list index out of range exception. See photos:

Thus, I assume there is a bug in the calibration dataset preparation OR usage.

from openvino_notebooks.

@MarkYav, I uploaded small subset of unet_calibration_data, which contains 20 samples.
I forgot to mention that you should also change size of subset_size to 20 (instead of 200) to align input_data with unet_calibration_data

from openvino_notebooks.

@l-bat I tried to run it but got the same output:

Right after that I generated a picture using the NOT quantized model and got this:

I was using ov_pipe(prompt, pose, 20, negative_prompt=negative_prompt). Also, I am surprised that the generated image is square.

from openvino_notebooks.

@MarkYav, if you can provide cat /proc/cpuinfo maybe I can find a suitable configuration to reproduce the error

from openvino_notebooks.

@MarkYav could we close the issue?

from openvino_notebooks.

@l-bat Yes, please.

from openvino_notebooks.