bayeswatch / deep-kernel-transfer Goto Github PK

I dont see .json files in the dataset, however, script requires the json files to train the model. iam i missing something.

base_file = configs.data_dir[params.dataset] + 'base.json'
KeyError: 'C:\Users\shant_w5mrdz3\OneDrive\Desktop\deep-kernel-transfer-master\filelists\emnist\emnist+base.json'

It looks like one of the files needed to setup the ImageNet data as used in this repo has moved. When I run the download_miniImagenet.sh file I get:

--2022-01-11 10:15:16--  http://image-net.org/image/ILSVRC2015/ILSVRC2015_CLS-LOC.tar.gz
Resolving image-net.org (image-net.org)... 171.64.68.16
Connecting to image-net.org (image-net.org)|171.64.68.16|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently

The website says links to Kaggle as a download source, but I'm not able to find the ILSVRC2015_CLS-LOC.tar.gz file available anywhere. Do you have an alternate location to download from by chance?

Hello,could you tell me why corresponding code for “CosSim” is "gpytorch.kernels.ScaleKernel(gpytorch.kernels.LinearKernel())"， not
"gpytorch.kernels.ScaleKernel(gpytorch.kernels.ScaleKernel(gpytorch.kernels.CosineKernel()))"

We are trying to run the following command for regression:
python train_regression.py --method="DKT" --seed=1
However, we face the following error:

Traceback (most recent call last): File "train_regression.py", line 37, in <module> model.train_loop(epoch, optimizer) File "/deep-kernel-transfer/methods/DKT_regression.py", line 53, in train_loop predictions = self.model(z) File "/home/.local/lib/python3.7/site-packages/gpytorch/models/exact_gp.py", line 256, in __call__ raise RuntimeError("You must train on the training inputs!")
We use Conv3 as backbone Spectral kernel and we observe that in self.model(z) in line 53, DKT_regression.py, NaN values are generated for z.
We are unable to find the reason for input values to get converted to NaN. However code runs fine with RBF kernel.
Kindly help.

While running testloop, ideally we won't have target variable or y_support[n] then why do you write self.model.set_train_data(inputs=z_support, targets=y_support[n], strict=False) in line 84. Will it not cause a target leak(or testing the same data which is in train set) while testing?

Hi,
The paper has not explained details of how DKT is creating the difference except in metrics which shows experiement results - Need how DKT is identified as the influencer/factor, what is the role/objective of DKT. Also in papaer, we see Baseline and Baseline++ models , what are these? Could you please give more details/explanations of these Baseline/Baseline++

Hi there,

I was wondering why the saved and loaded model performs poorly compared to the original model. I used the exact same test data in both cases and got R2_score = 0.97, -0.001 for the original and loaded model respectively.

Best

"wget http://image-net.org/image/ILSVRC2015/ILSVRC2015_CLS-LOC.tar.gz" is not available.
The requested URL /image/ILSVRC2015/ILSVRC2015_CLS-LOC.tar.gz was not found on this server.
So, could you please share the miniImagenet dataset used in your experiments? (such as google drive)
Thanks!

Your paper mentions about in-range and out-range mse in the regression use case. However, we don't find these errors separately in the codebase. On running test_regression.py we receive a single MSE. Can you kindly elaborate what this value represents (in-range or out-range) ?
Is the code and data available only for QMUL or sines also.

Hello I tried to run sine regression task DKT scripts as python train_DKT.py in sine folder and met the following issue. Environment
gpytorch 1.6.0
python 3.9
torch 1.10.1

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py:38: NumericalWarning: A not p.d., added jitter of 1.0e-06 to the diagonal
warnings.warn(
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py:38: NumericalWarning: A not p.d., added jitter of 1.0e-05 to the diagonal
warnings.warn(
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py:38: NumericalWarning: A not p.d., added jitter of 1.0e-04 to the diagonal
warnings.warn(
Traceback (most recent call last):
File "/Users/xichen.dxc/Desktop/project/gitlab/deep-kernel-transfer/sines/train_DKT.py", line 287, in
main()
File "/Users/xichen.dxc/Desktop/project/gitlab/deep-kernel-transfer/sines/train_DKT.py", line 183, in main
loss = -mll(predictions, gp.train_targets)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/module.py", line 30, in call
outputs = self.forward(*inputs, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/mlls/exact_marginal_log_likelihood.py", line 62, in forward
res = output.log_prob(target)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/distributions/multivariate_normal.py", line 169, in log_prob
inv_quad, logdet = covar.inv_quad_logdet(inv_quad_rhs=diff.unsqueeze(-1), logdet=True)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/lazy/lazy_tensor.py", line 1291, in inv_quad_logdet
cholesky = CholLazyTensor(TriangularLazyTensor(self.cholesky()))
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/lazy/lazy_tensor.py", line 1004, in cholesky
chol = self._cholesky(upper=False)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/memoize.py", line 59, in g
return _add_to_cache(self, cache_name, method(self, *args, **kwargs), *args, kwargs_pkl=kwargs_pkl)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/lazy/lazy_tensor.py", line 435, in _cholesky
cholesky = psd_safe_cholesky(evaluated_mat, upper=upper).contiguous()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py", line 63, in psd_safe_cholesky
L = _psd_safe_cholesky(A, out=out, jitter=jitter, max_tries=max_tries)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/gpytorch/utils/cholesky.py", line 45, in _psd_safe_cholesky
raise NotPSDError(f"Matrix not positive definite after repeatedly adding jitter up to {jitter_new:.1e}.")
gpytorch.utils.errors.NotPSDError: Matrix not positive definite after repeatedly adding jitter up to 1.0e-04.

Hi @mpatacchiola
I'm using DKT code for classification on CUB [5 way- 1 shot]. I save two models during meta-training, best_model and last_model.
in test.py I add some code to test best_model and last_model.
I create two instantiates of DKT class:

elif params.method == 'DKT':
        # model           = DKT(model_dict[params.model], **few_shot_params)
        last_model      = DKT(model_dict[params.model], **few_shot_params)
        best_model      = DKT(model_dict[params.model], **few_shot_params)

and load saved files for each of them:

        best, last = True, True
        modelfile = None
        if params.save_iter != -1:
            modelfile   = get_assigned_file(checkpoint_dir,params.save_iter)
        if best:
            best_modelfile   = get_best_file(checkpoint_dir)
            print(f'\n best model {best_modelfile}')
        if last:
            files = os.listdir(checkpoint_dir)
            nums =  [int(f.split('.')[0]) for f in files if 'best' not in f]
            num = max(nums)
            print(f'\nModel at last epoch {num}')
            last_modelfile = os.path.join(checkpoint_dir, '{:d}.tar'.format(num))
            print(f'\nlast model {last_modelfile}\n')
       if best and best_modelfile is not None:
            best_model = best_model.cuda()
            tmp = torch.load(best_modelfile)
            best_model.load_state_dict(tmp['state'])

        if last and last_modelfile is not None:
            last_model = last_model.cuda()
            tmp = torch.load(last_modelfile)
            last_model.load_state_dict(tmp['state'])

when I run test.py for these models:

         if last:
            print('last')
            last_model.eval()
            acc_mean, acc_std = last_model.test_loop( novel_loader, return_std = True)
            print("-----------------------------")
            print('Test Acc last model = %4.2f%% +- %4.2f%%' %(acc_mean, acc_std))
            print("-----------------------------") 

        if best:
            print('Best') 
            best_model.eval()
            acc_mean, acc_std = best_model.test_loop( novel_loader, return_std = True)
            print("-----------------------------")
            print('Test Acc best model = %4.2f%% +- %4.2f%%' %(acc_mean, acc_std))
            print("-----------------------------") 

I have a problem with ACC.
If I run best_model or last model alone, I get some ACCs; If I run both of them (best, last= True, True) I get different ACC for the model that is run in the second step. (running last_model after best_model test, changes the ACC of last_model.)

gpytorch 1.6.0
torch 1.10.0

Could you please help me to figure out what the problem is?
I really appreciate any help you can provide.

Can you provide the code or let me know how we can include Multi GPU for training this model? Due to ResNet + GP structure, I am unable to directly implement that.

Hi,

Many thanks for making this code available. I was wondering if you are going to release it under an open license?

Kind regards,

Ivan

Hi,

After following the steps mentioned in the README file for regression, I am getting the following error using spectral kernel.

python train_regression.py --method="DKT" --seed=1
[0] - Loss: nan MSE: nan noise: nan
Traceback (most recent call last):
File "train_regression.py", line 37, in
model.train_loop(epoch, optimizer)
File "/deep-kernel-transfer/methods/DKT_regression.py", line 53, in train_loop
predictions = self.model(z)
File "/home/user/.local/lib/python3.7/site-packages/gpytorch/models/exact_gp.py", line 256, in call
raise RuntimeError("You must train on the training inputs!")
RuntimeError: You must train on the training inputs!

Can you please how to correct it and get the desired results?

Also, using rbf kernel I am getting Average MSE: 0.243 +- 0.18 with the default settings on QMUL data but the reported MSE is much lower. Is it in-range or out-range?

Thanks.

I think it will be better to update the parameters on the query set, rather than on the support set.
For the reason that the evaluation process on the query set is the simulation for the meta-test setting.

I want to prepare the dataset for miniImageNet but the URL for it doesn't exist. Kindly provide a valid link.

Hi @mpatacchiola, @elliotjcrowley

Could you provide the code for MAML regression on QMUL (head pose detection) please?
Thanks in advancce.