Unable to produce any kind of stylized image about nlut HOT 5 CLOSED

Hi folks,

I'm trying to produce any kind of stylized image with this repo and, unfortunately, have been unable to do so even with the default image data in the data/ folder. I'm issuing the below command:

python3 inference_finetuning_image.py --pretrained ./experiments/336999_style_lut.pth --resume ./experiments/336999_style_lut.pth 

The above command should operate on the default data specified in parameter_finetuning.py. It seems the generated output, however, is pretty much the same as the input content image.

Am I doing something wrong with the usage of the stylization script itself, or is something wrong with the pre-trained checkpoitn that was previously available (336999_style_lut.pth)? As observed below, the losses during fine-tuning seem pretty high as well, but I'm more surprised that it seems like absolutely no stylization is applied to the output video.

now device is cuda:0
n=2048 s=32 w=32 ++
Total params: 22.01M
--------loading checkpoint----------
=> loading checkpoint './experiments/336999_style_lut.pth'
/home/akshaypa/.local/lib/python3.8/site-packages/torch/nn/functional.py:1795: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
  warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
iter 0   time/iter: 0.05  lr: 0.000100 loss_mn: 0.0009 loss_c: 0.0000   loss_s: 2.2449 losses: 2.2458 
iter 10   time/iter: 0.56  lr: 0.000100 loss_mn: 0.0693 loss_c: 0.0000   loss_s: 2.2448 losses: 2.3141 
iter 20   time/iter: 0.56  lr: 0.000100 loss_mn: 0.0405 loss_c: 0.0000   loss_s: 2.2447 losses: 2.2852 
iter 30   time/iter: 0.56  lr: 0.000100 loss_mn: 0.0283 loss_c: 0.0000   loss_s: 2.2446 losses: 2.2729 
iter 39   time/iter: 0.43  lr: 0.000100 loss_mn: 0.0223 loss_c: 0.0000   loss_s: 2.2444 losses: 2.2667 
n=2048 s=32 w=32 ++
Total params: 22.01M
--------loading checkpoint----------
=> loading checkpoint './experiments/336999_style_lut.pth'
save to: data/city2.jpg

Hi, you should not use the same as "--resume" file with "--pretrained" file. If so, the new checkpoint will replace the old one once you run the code. I guess the checkpoint you using has been changed now, please download the pretrained
checkpoint from here https://pan.baidu.com/s/1VddHbq2cBy5RcKOp8S5eSg and try it again. thanks a lot.

from nlut.

Thanks for the reply! I'll try the new pre-trained checkpoint. Just to confirm, with what I'm trying to do (just getting a stylized output) what would the correct command look like? This command below (ignore the .pth name potentially being different)?

python3 inference_finetuning_image.py --pretrained ./experiments/336999_style_lut.pth

from nlut.

yes, you are correct.

from nlut.

Ok, I've gone ahead and tried this again with the latest checkpoint you uploaded. The below command:

python3 inference_finetuning_image.py --pretrained ./experiments/336999_style_lut.pth

Produces this:

now device is cuda:0
data/cityframeat0m0s.png
data/city.jpg
n=2048 s=32 w=32 ++
Total params: 22.01M
--------loading checkpoint----------
=> loading checkpoint './experiments/336999_style_lut.pth'
/home/akshaypa/.local/lib/python3.8/site-packages/torch/nn/functional.py:1795: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
  warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
iter 0   time/iter: 0.05  lr: 0.000100 loss_mn: 0.0015 loss_c: 0.0000   loss_s: 2.2436 losses: 2.2450 
iter 10   time/iter: 0.56  lr: 0.000100 loss_mn: 0.0513 loss_c: 0.0000   loss_s: 2.2434 losses: 2.2947 
iter 20   time/iter: 0.56  lr: 0.000100 loss_mn: 0.0305 loss_c: 0.0000   loss_s: 2.2431 losses: 2.2736 
iter 30   time/iter: 0.56  lr: 0.000100 loss_mn: 0.0218 loss_c: 0.0000   loss_s: 2.2428 losses: 2.2646 
iter 39   time/iter: 0.43  lr: 0.000100 loss_mn: 0.0176 loss_c: 0.0000   loss_s: 2.2423 losses: 2.2599 
n=2048 s=32 w=32 ++
Total params: 22.01M
--------loading checkpoint----------
=> loading checkpoint 'experiments/resume_style_lut.pth'
save to: data/city2.jpg

Which is very similar to the previous result, with the style loss being very high still and the output result being too similar to the input content (e.g., no stylization appears to happen). Is there something that might be missing from this code repo? Are you able to reproduce the city image stylization result using this repo?

from nlut.

Never mind, I figured out my mistake - it seems I somehow overwrote my pre-trained model download before when using the --resume option. I fixed that, and then received the below error which I also fixed:

Traceback (most recent call last):                                                                                                                                                             
  File "inference_finetuning_image.py", line 233, in <module>                                                                                                                                  
    lut = finetuning_train(opt, original, example)                                                                                                                                             
  File "inference_finetuning_image.py", line 86, in finetuning_train                                                                                                                           
    model.load_state_dict(checkpoint['state_dict'])                                                                                                                                            
  File "/home/akshaypa/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1482, in load_state_dict                                                                           
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(                                                                                                                  
RuntimeError: Error(s) in loading state_dict for NLUTNet:
        Unexpected key(s) in state_dict: "blurer.op.1.weight", "SB1.conv1.conv2d.weight", "SB1.conv1.conv2d.bias", "SB1.conv1.bn.weight", "SB1.conv1.bn.bias", "SB1.conv1.bn.running_mean", "SB1.conv1.bn.running_var", "SB1.conv1.bn.num_batches_tracked", "SB1.conv2.conv2d.weight", "SB1.conv2.conv2d.bias", "SB1.conv2.bn.weight", "SB1.conv2.bn.bias", "SB1.conv2.bn.running_mean", "SB1.conv2.bn.running_var", "SB1.conv2.bn.num_batches_tracked". 

I fixed the above error by changing this line in inference_finetuning_image.py to:

model.load_state_dict(checkpoint['state_dict'], strict=False)

Which allows for incompatible model shapes. @semchan, maybe you can consider making this quick change in inference_finetuning_image.py and wherever else necessary, if you think that's a good idea? I'm closing this issue now as my problem has been resolved.

from nlut.