Difference in Baseline FP32 accuracy numbers for MobileNetV2 and ResNet18 as compared to DFQ about zeroq HOT 4 CLOSED

Hi Thanks for reaching out, the MobileNetV2 that we are using is from PytorchCV which has a baseline of 73.03. Please see the link below:

https://pypi.org/project/pytorchcv/

We specifically wanted to test against the higher baseline accuracy to see if there is any accuracy degradation with the strong baseline.

from zeroq.

Hi Thanks for reaching out, the MobileNetV2 that we are using is from PytorchCV which has a baseline of 73.03. Please see the link below:

https://pypi.org/project/pytorchcv/

We specifically wanted to test against the higher baseline accuracy to see if there is any accuracy degradation with the strong baseline.

Thankyou for the quick response.
If you chose to use a higher accuracy baseline different from DFQ, In such case do the comparison results against DFQ and other methods are valid.? I am curious to know whether on this baseline also DFQ resulted in same accuracy as they claimed in their work.?
Sorry to ask you again. I would like to experiment more on quantization starting with your approach, hence putting forward all my queries and collecting valuable suggestions from you.
Thank you.

from zeroq.

The reason we decided to test a strong baseline for all of the tasks is simple. If we use a lower baseline accuracy, then it is not possible to test quantization's impact. For example, it is actually possible to get higher accuracy after quantization if you use a weak baseline. This can be misleading, because the reason for accuracy improvement is not quantization, but that the FP32 baseline was a weak baseline. To avoid such confusion, we decided to test with the strongest accuracy for each of the models. Recovering the accuracy of the stronger baseline is harder and that is why we targeted that.

Now you can use three methods to compare ZeroQ with other methods in the literature (including DFQ which I believe is interesting):

1/ see how much the accuracy drops from each respective baseline. This gives you some estimate but I have to caution you that an accuracy drop of say 0.1 with a weak baseline is not the same as with a strong baseline (accuracy gains become exponentially harder as accuracy increases).

2/ Our code is available online and you can simply load the MobiletNetV2 with any pretrained accuracy and test our performance and directly compare with other results in the literature.

3/ If the other paper in the literature has their code available online, repeat step 2 with the stronger baseline.

Hope this helps

from zeroq.

The reason we decided to test a strong baseline for all of the tasks is simple. If we use a lower baseline accuracy, then it is not possible to test quantization's impact. For example, it is actually possible to get higher accuracy after quantization if you use a weak baseline. This can be misleading, because the reason for accuracy improvement is not quantization, but that the FP32 baseline was a weak baseline. To avoid such confusion, we decided to test with the strongest accuracy for each of the models. Recovering the accuracy of the stronger baseline is harder and that is why we targeted that.

Now you can use three methods to compare ZeroQ with other methods in the literature (including DFQ which I believe is interesting):

1/ see how much the accuracy drops from each respective baseline. This gives you some estimate but I have to caution you that an accuracy drop of say 0.1 with a weak baseline is not the same as with a strong baseline (accuracy gains become exponentially harder as accuracy increases).

2/ Our code is available online and you can simply load the MobiletNetV2 with any pretrained accuracy and test our performance and directly compare with other results in the literature.

3/ If the other paper in the literature has their code available online, repeat step 2 with the stronger baseline.

Hope this helps

Yes that makes it very clear. Thank you so much.

from zeroq.