About optimized weights about aki65.github.io HOT 6 CLOSED

This is complicated and can't be explained in a few sentences.

I'm sorry, I don't have the time to give support here.

from aki65.github.io.

This is complicated and can't be explained in a few sentences.

I'm sorry, I don't have the time to give support here.

Ok, thanks! What if I just simply export the katago weights as int8 in bin.gz file. Does it work for the current engine? If not, would you consider supporting it in the future, thanks!

from aki65.github.io.

This approach won't work because:
a) It would keep the original model structure, and this structure can't be accelerated on the NPU due to some limitations in NNAPI interface and driver
b) If you just replace the katago float weights by the nearest int8, the error will be huge and the network results will be unusable. You need an individually tuned mapping from float to int8 for each operation (sometimes for each channel), the quantization scale.

from aki65.github.io.

I tried to convert the quantized weights using tf.lite.converter (MLIR). What I did was I first converted the saved model to a frozen graph, and then do the quantization. Since I don't have enough sample data, I just randomly generated the representative dataset within the reasonable range to see if it would work. And the structure of the resulting tflite weights was the same as those before v1.8.2. I found that batchnorm beta was replaced by an identical convolutional layer with bias in the new version of optimized weights (after v1.8.2). Anyways, my quantized weights was able to run but the speed was kind of slow (around 10v/s) . It seems that the optimization didn't really work. So would you be willing to give some quick instructions on the quantization process (like how to generate representative datasets). And by the way. it seems that the quantized weitghts is able to run on edge tpu. Since they both have the same architecture, would the app be able run on it? Thanks. @aki65

My quantized weights:
https://drive.google.com/file/d/1nDC4OXu3nqMvJ-GrTr2xKpAvPUAAx2w8/view?usp=sharing

from aki65.github.io.

A followed up question. Do you only use python or half python half c++ to do the full inference ?

from aki65.github.io.

I just found that the concatenations in the gpools are also replaced by padding and addition. The tensordot is replaced by reshape and matmul

from aki65.github.io.