alterzero / rbpn-pytorch Goto Github PK

As I see in your paper, you were experimenting only with 4x upscaling factor. Maybe you had some proceedings with greater factors? What issues may arise, if any?

wrong with Line74 in dataset.py: seq = [x for x in range(4-tt,5+tt) if x!=4]
if nFrames=11 and tt=int(nFrames/2)=5, seq will start from -1, however, there isn't im"-1".png .

I'm using a RTX2060 GPU, apparently the memory of it is not enough( 6 GiB) .

What's the lowest capacity to train RBPN? I noticed that you used 2 GPUS with video memory of 16GiB totally. But I can only afford a 8GiB one, is 8GiB available? or is there other ways to solve this problem so that I can train on a RTX2060?

thanks a lot,

a noob

Hi, Thank you for your code!
I want to train the network on my own dataset , How should I handle my own dataset?
Thanks!

Hi,

May I know the value of RPBN/6 loss with Vimeo-90K at the end of your training?
Thanks

Hi:
Thanks for your sharing, I get the images and weights, set scale 2, use model weights/RBPN_2x.pth, but got a very big result, psnr is nan, how can I fix this bug

This is not a bug, but I'm curious that why you do not use batch normalization? This is not discussed in the paper, can you shed some light on this?

Great Job , thank you! I'am very interesting on your work.
And I have a question to ask, when we do the testing, the input img is based on the sampling of the original image. Now I do not have the original image, only the low-quality image.
Assuming that I only have 7 consecutive low-resolution images, no corresponding high-definition original image, how should I synthesize a high-quality image? Look forward to your reply！

Can u tell me how to ues pyflow? And I cannot install pflow correctly.

It really comfuse me! I have passed the pyflow demo and also use python3.5.
But I have the same error above. How can I solve this problem?

Hi,

Thanks for your great work. If it is possible, may you release your training dataset list?

Hi, thanks for your nice code!
i have a problem of downloading the Vimeo-90k Dataset. i have tried several times to doanload it. however, it did not download the full dataset as it is a very big dataset , thus, could you share your trianing and test set?
thanks a lot.

Hi,
How to solve it? Thanks!

it will randomly say "All the pixels are invalid in estimation Laplacian noise" while training. is it okay?

mostly the training looks like this.
.....................................................................
Pyramid level 2
Pyramid level 1
Pyramid level 0
Constructing pyramid...done!
Pyramid level 2
Pyramid level 1
Pyramid level 0
Constructing pyramid...done!
Pyramid level 2
Pyramid level 1
Pyramid level 0
Constructing pyramid...done!
Pyramid level 2
Pyramid level 1
Pyramid level 0
===> Epoch1: Loss: 13.6273 || Timer: 0.1491 sec.

Hi, I find it will take a very long time to download the dataset and train the model.
Is there any pre-trained model can be provided for an easy and quick start?

I found this problem: FileNotFoundError: [Errno 2] No such file or directory: './vimeo_septuplet/sequences/sep_trainlist.txt'
how to sovle the problem.

Hi,

I successfully run follows:

python setup.py build_ext -i
cp pyflow*.so ..

When I try to run

python main.py

I encounter a problem about skimage

Traceback (most recent call last):
File "main.py", line 13, in
from data import get_training_set, get_eval_set
File "/home/ryluo/Desktop/RBPN-PyTorch/data.py", line 7, in
from dataset import DatasetFromFolderTest, DatasetFromFolder
File "/home/ryluo/Desktop/RBPN-PyTorch/dataset.py", line 10, in
from skimage import img_as_float
ModuleNotFoundError: No module named 'skimage'

Any ideas what causing this probelm?

Thanks

python demo.py
Traceback (most recent call last):
File "demo.py", line 11, in
import pyflow
ImportError: /home/hai/anaconda3/envs/tianchi/lib/python3.7/site-packages/pyflow.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __warn_memset_zero_len

I found this problem: FileNotFoundError: [Errno 2] No such file or directory: './vimeo_septuplet/sequences/sep_trainlist.txt'
how to sovle the problem.

when training the network using 9 frames or 11 frames, is it necessary to adjust the learning rate?

How long does the code need to train in total?

The error message is gcc: error: src/Coarse2FineFlowWrapper.cpp: No such file or directory when I install pyflow.

It is recommended to install from the original repository (pyflow) rather than the subdirectory of this repository.

I run the eval.py and meet dim error.how could I solve the problem?

the memory of my GPU is 8G-12G, when we train ,the patch_size is 64*64, the gpu memory is eough. However when testing, the test data Vid4 and SPMCS is larger than vimeo-90k, when I test model, set the testBatchsize=1, but still CUDA out of memory. If I what to recurrent the score in paper, how to test on dataset Vid4 or SPMCS? Thanks~

When using optical flow, instead of the straight concatenation [It-k, Ft-k, It], a natural alternative choice is to first warp the It-k to get It-k_warped, then concatenate it with It to get [It-k_warped, It], and finally feed [It-k_warped, It] to the MISR block. But in the experiments of your paper, only cases with or without optical flow were compared. Any consideration about that?

Hi!
I am anxious to experiment with your RBPN code. I have downloaded onto a Windows 10 machine with a GPU, and installed Python 3.5, PyTorch 1.0.1, and Pyflow dependencies (plus any lower order functions that were missing). In running the eval.py script, I get the following error (I needed to interrupt the stalled process at the end)...

Namespace(chop_forward=False, data_dir='./Vid4', file_list='foliage.txt', future_frame=True, gpu_mode=True, gpus=1, model='weights/RBPN_4x.pth', model_type='RBPN', nFrames=7, other_dataset=True, output='Results/', residual=False, seed=123, testBatchSize=1, threads=1, upscale_factor=4)
===> Loading datasets
===> Building model RBPN
Pre-trained SR model is loaded.
Namespace(chop_forward=False, data_dir='./Vid4', file_list='foliage.txt', future_frame=True, gpu_mode=True, gpus=1, model='weights/RBPN_4x.pth', model_type='RBPN', nFrames=7, other_dataset=True, output='Results/', residual=False, seed=123, testBatchSize=1, threads=1, upscale_factor=4)
===> Loading datasets
===> Building model RBPN
Pre-trained SR model is loaded.
Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files\Python35\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Program Files\Python35\lib\multiprocessing\spawn.py", line 115, in _main
prepare(preparation_data)
File "C:\Program Files\Python35\lib\multiprocessing\spawn.py", line 226, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Program Files\Python35\lib\multiprocessing\spawn.py", line 278, in _fixup_main_from_path
run_name="mp_main")
File "C:\Program Files\Python35\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Program Files\Python35\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Program Files\Python35\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Steve\Downloads\RBPN-PyTorch-master\RBPN-PyTorch-master\eval.py", line 182, in
eval()
File "C:\Users\Steve\Downloads\RBPN-PyTorch-master\RBPN-PyTorch-master\eval.py", line 79, in eval
for batch in testing_data_loader:
File "C:\Program Files\Python35\lib\site-packages\torch\utils\data\dataloader.py", line 819, in iter
return _DataLoaderIter(self)
File "C:\Program Files\Python35\lib\site-packages\torch\utils\data\dataloader.py", line 560, in init
w.start()
File "C:\Program Files\Python35\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Program Files\Python35\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Program Files\Python35\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\Program Files\Python35\lib\multiprocessing\popen_spawn_win32.py", line 34, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Program Files\Python35\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
_check_not_importing_main()
File "C:\Program Files\Python35\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Traceback (most recent call last):
File "eval.py", line 182, in
eval()
File "eval.py", line 79, in eval
for batch in testing_data_loader:
File "C:\Program Files\Python35\lib\site-packages\torch\utils\data\dataloader.py", line 631, in next
idx, batch = self._get_batch()
File "C:\Program Files\Python35\lib\site-packages\torch\utils\data\dataloader.py", line 610, in _get_batch
return self.data_queue.get()
File "C:\Program Files\Python35\lib\multiprocessing\queues.py", line 94, in get
res = self._recv_bytes()
File "C:\Program Files\Python35\lib\multiprocessing\connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "C:\Program Files\Python35\lib\multiprocessing\connection.py", line 306, in _recv_bytes
[ov.event], False, INFINITE)
KeyboardInterrupt

I am not very familiar with Python, but from trying to understand this error, it appears that it may be unique to Windows because of the way it spawns off processes compared to Linux. See note under torch.utils.data.DataLoader here...

https://pytorch.org/docs/stable/data.html?highlight=dataloader%20py#torch.utils.data.DataLoader

The suggested correction is to include checks using
if __name__ == '__main__':
in the eval.py code at the appropriate places. When that is done correctly, the code should work correctly on both Windows and Linux. I experimented with adding this check to several locations in the code, but was unsuccessful to get a complete run. Can you suggest how to modify the code to work on Windows or another change that would allow it to run?

Thanks
-Steve

The total size of the zip files is 1.01G. I download it for full size but something is wrong with unzipping it. How can I solve this problem?

Good code helps others reproduce your work

According to the Pillow Docs, 'PIL.ImageOps.mirror(image)' flips image horizontally (left to right) and 'PIL.ImageOps.flip(image)' flips the image vertically (top to bottom). So maybe a misusage exists in

I notice that the code load a pretrained weight file. Which dataset is the model pretrained on? What's the performance on Vimeo90K without pretraining?

Thanks for your great work!
I have a question that how to predict the first or the last frames that no former or latter frames?
In your work, it seems you use neighbor frames around the key frames to help predict the target frame.

For example, one short video, 1-10 frames are in scene 1, 11-20 frames are in scene 2.
So there are two situations(using future=True, nFrames=7):
1>1-3 lack of previous neibors， and 17-20 lack of future neibors. Replace input LR as the lack neibors, and the flow is zero.
2>for 7-13, the neighbors cross two scene, and the flows are disorderly.
Will the network be well generalized in these two situations when testing? If not, how to solve these two problems?

I looked at the training set you used. There is no situation 2, because every pairs are 7 frames from the same scene.
If I cut the video into different scenes, situation 2 ↓，but situation 1 ↑.

Thank you for making your code public. Can your code be reproduced on a CPU basis, and how to do it. Because I had this problem during testing:

Exception: No GPU found, please run without --cuda.

Looking forward to your reply.

Hi, thanks for your work.
I downloaded the pretrained model in your link and tested the RBPN_4x.pth model on Vid4 test set. However, the results confused me. I got the following performance: calendar=22.2495 (paper
23.99), city=26.1635 (paper 27.73), foliage=24.7383 (paper 26.22), walk=29.2091 (paper 30.70). I wonder why it underperforms the expeted results in the paper. It would be nice of you if you could tell me how to reproduce the reported results using the pretrained models.
Thanks a lot.

BTW, I think there is a small bug in eval.py line73. It should be count=0.

Hi, I trained the model on dataset vimeo-90k, nFrame=7, patch-size=64, future-frame=True, after about 20 epochs(four days; 8 gpus; bs=8), the loss is around 0.009. However when I test the model, I can't get the awesome result in the paper. Vid4 walk 28.12 (paper=30.70), vimeo slow test data 31.91(paper=34.18)。 Should I wait for more epochs? I will be appreciated if you could give more training or testing detail.

All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!

(pytorch100) hym@vipa210:~/pycharm_remote_files/RBPN-pytorch$ CUDA_VISIBLE_DEVICES=3 python eval.py
Namespace(chop_forward=True, data_dir='./Vid4', file_list='foliage.txt', future_frame=True, gpu_mode=True, gpus=1, model='weights/RBPN_4x.pth', model_type='RBPN', nFrames=7, other_dataset=True, output='Results/', residual=False, seed=123, testBatchSize=1, threads=1, upscale_factor=4)
===> Loading datasets
===> Building model RBPN
/temp_disk/hym/pycharm_remote_files/RBPN-pytorch/dbpns.py:50: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
torch.nn.init.kaiming_normal(m.weight)
/temp_disk/hym/pycharm_remote_files/RBPN-pytorch/dbpns.py:54: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
torch.nn.init.kaiming_normal(m.weight)
/temp_disk/hym/pycharm_remote_files/RBPN-pytorch/rbpn.py:74: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
torch.nn.init.kaiming_normal(m.weight)
/temp_disk/hym/pycharm_remote_files/RBPN-pytorch/rbpn.py:78: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
torch.nn.init.kaiming_normal(m.weight)
neigbor frame- is not exist
neigbor frame- is not exist
neigbor frame- is not exist
Constructing pyramid...done!
Pyramid level 6All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!

Pyramid level 5All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!

Hello, author. You have done a great job and have a very good result.Thank you for your work.But when I'm running the seven-frame model, the time cost is 1200ms per frame.Graphics card is 1080TI.Can you tell me how to optimize the code to speed up?Thank you~

Hi,
I'm trying to run the eval.py script with all of the default command-line arguments and have been experiencing a couple of (inter-related?) issues.

The pyflow folder included in the repository doesn't build for me, out-of-the-box. I ran into issues due to the fact that all of the .cpp files are missing (which exist in the official pyflow repo). After re-cloning this repo, I was able to build pyflow.

However, when running the provided eval.py script, I then get the following error:

File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/code/pytorch_projects/upres/rbpn/rbpn.py", line 76, in forward
    feat_frame.append(self.feat1(torch.cat((x, neigbor[j], flow[j]),1)))
RuntimeError: Expected a Tensor of type torch.cuda.FloatTensor but found a type torch.cuda.DoubleTensor for sequence element 2  in sequence argument at position #1 'tensors'

I was able to get around this by modifying line #76 of rbpn.py to include an explicit cast to a floating-point tensor as follows:

feat_frame.append(self.feat1(torch.cat((x, neigbor[j], flow[j].float()),1)))

After this change, the code runs, but there are continuous error messages of the form:

All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!

The code finishes execution and outputs into the Results/ folder. However, all of these images are the same as the original resolution (720 x 480), so it would seem that the up-resolution code didn't actually do anything.

I know this is a lot, but any ideas of what I might be doing wrong? Thanks in advance.

Hi!
I have started working with your excellent code and even prepared my own input files to see how good they look upscaled using your approach. However regardless of the value used for --upscale_factor (I tried 2 and 4), the resolution of the output is the same as the input. For example, my input images are 160x160 and so are my output images. Is there another flag that I need to set to get the upscaled images? If not, can you tell me how to modify the code to obtain upscaled images?
Thanks!

PS. I have found that for most of the images I am trying out with your code, I need to use the --chop_forward flag set to True or I get a GPU memory error since I have a relatively small amount of GPU memory

I tried to run the eval code on the vid4 dataset, but the psnr calculated according to the method in the project is inconsistent with the paper. Is there any other code to compute psnr?

Hi, thanks for your nice code!
In line 53 of dataset.py, I think the code should be
"
if nFrames%2 == 0:
seq = [x for x in range(-tt,tt) if x!=0] # or seq = [x for x in range(-tt+1,tt+1) if x!=0]
else:
seq = [x for x in range(-tt,tt+1) if x!=0]
"
because when nFrames is an even number, " seq = [x for x in range(-tt,tt+1) if x!=0]" will make the number of neigbor equal to nFrames instead of nFrames-1.
You can try running the command "python main.py --other_dataset True --nFrames 6".

Hi, thanks for sharing.

There's a doubt when training about the function get_flow(). I got the three model inputs (img, neighbor, flow), of which the img and the neighbor seems valued between (0,1). But the flow concatenated by u,v seems almost (3.xxx, 9.xxx ), and also the rescale_flow function is commented by default.

It would be appreciated if you could like share the related thoughts. Thanks

Following the readme, the following error is occurred during the test.
RuntimeError: Error(s) in loading state_dict for Net: Missing key(s) in state_dict:

anyone knows?
thx

in rbon.py, there is a correction in line 90: it should be

feat_frame.append(self.feat1(torch.cat((x.float(), neigbor[j].float(), flow[j].float()), 1)))

Hello, @alterzero , thanks for your excellent work! I have some questions about the training phase.

1. The LR image in the Vimeo-90k dataset is 11224, and you set the patch size to 64 while training. How to set the patch size if I train the RBPN network using 480270 LR images?
2. In your training code, if the target frame is the last frame of the video, you set the future neighbor frames as the target frame and the flow between these frames and target frame will be zero. I want to know if some potential problems will occur while concatenating the frames and flow.

Thanks and hope your reply.

Hello, thank you very much for your code, I use your test code, found that the test results are more blurred than the original, and print the following information, can you give me some suggestions？

Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!
All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!

===> Processing: 47 || Timer: 0.2439 sec.