When I want to port CKB to my project, some errors occur about ckb HOT 6 CLOSED

Hi, @FRESHMAN11392 !

Thank you for your interest.

It seems that the kernel matrix K_ZtZs in CKButils.py, line 130 is ill-conditioned or not well-defined. I guess it may be induced by the used data.

Usually, the theoretically well-defined of K_ZtZs is because the kernel matrix (induced by a positive kernel $k(·,·)$) on $n$ distinct points is always positive definite. Since in our experiments, the variable $Z$ is taken as the DNN features of images, they are reasonable to be different, i.e., they are distinct.

So, I suggest checking the kernel matrix K_ZtZs w.r.t. its values/rank (or other criteria related to the singular problem) and the data that are used to input as $Z$, i.e., the fea_s/fea_t input in function CKB_Metric(...).

Note that if fea_s and fea_t are exactly (or nearly) the same, the kernel matrix will be singular and the rank will be small. For example, if they are the labels with one-hot coding in a mini-batch without shuffle, they are likely to be the same. Then the kernel matrix will be a matrix with all elements equal to 1, which is a rank-1 matrix.

from ckb.

Thanks to Dr. Luo for your comment. And thank you very much for your excellent work
Due to the characteristics of our data, there are indeed very similar data with the same label in a mini-batch, is there a singular problem in this case?

from ckb.

Hi!

Sorry for the ambiguity. Actually, the last example means the case that input fea_s and fea_t are the labels from the same class. Then any two samples from fea_s and fea_t are the same one-hot vector, and the distance between is zero. Thus, the distance matrix D_ZtZs in CKButils.py, line 116 is a matrix with all elements equal to 0, and the kernel matrix K_ZtZs in CKButils.py, line 130 is a matrix with all elements equal to 1. This will lead to a singular and rank-1 kernel matrix, which may induce the RuntimeError.

In general, if your data are similar (but not the same), the correlations (somehow similar to the linear dependence in linear algebra) between the rows/columns of K_ZtZs will be large. In this case, the matrix may be ill-conditioned.

Overall, I am not sure whether similar data induce the singular problem. So, I suggest checking the rank of K_ZtZs via some built-in functions like torch.matrix_rank and the values of K_ZtZs by print/display. Besides, may you provide some details about the characteristics of your data, e.g., the dimension of tensor/matrix 'fea_s/fea_t', type (image or others), and the criteria mentioned above?

Note. If every mini-batch data are guaranteed to have the same labels, you could just use the Kernel Bures Metric [1] to align them as there are no different values for the conditional variable $Y$.
[1] Zhen Zhang, Mianzhi Wang, and Arye Nehorai. Optimal transport in reproducing kernel Hilbert spaces: Theory and applications. IEEE TPAMI.

from ckb.

Thanks again to the author for answering my question.
I'm sure the problem might be related to my datasource rather than the code. We checked the rank of K_ZtZs, it is indeed not full rank.
The dimension of tensor/matrix 'fea_s/fea_t' is 32, each data sample is a one-dimensional row vector, in fact they are current values in the physical sense.
Thanks again for your generous answer and sorry for taking up your precious time.

from ckb.

Glad to be of help!

In fact, I think the problem may also be the built-in SVD function itself since there are many similar bug reports about the problem.

If the matrix is rank-deficient, one can perform truncated-SVD to obtain the solution. But I have checked the pytorch document where only Economy-size SVD (i.e., the default SVD in pytorch) is provided.

If the CKB is used as a metric but not a loss function, which means no gradients are required for optimization, we can use other packages (e.g., numpy) or languages (e.g., matlab) for more numerical stable results.

If you have any further questions, please feel free to comment here.

from ckb.

hh
We also considered the problem of the SVD function itself, which is more difficult for us to solve. We will continue to find a solution to the problem and appreciate your help. (If our applied work based on CKB is published, we will thank you for your help in the statement.)

from ckb.