akshaydudhane16 / burstormer Goto Github PK

View Code? Open in Web Editor NEW

98.0 8.0 4.0 18.42 MB

SOTA for Burst Super-resolution, Low-light Burst Image Enhancement, Burst Image De-noising

Python 99.93% Shell 0.07%

alignment backprojection burst de-noising fusion low-light-image-enhancement python restoration super-resolution

burstormer's Introduction

Burstormer: Burst Image Restoration and Enhancement Transformer (CVPR 2023)

Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan, and Ming-Hsuan Yang

News

Feb 27, 2023: Paper accepted at CVPR 2023 🎉

Abstract: On a shutter press, modern handheld cameras capture multiple images in rapid succession and merge them to generate a single image. However, individual frames in a burst are misaligned due to inevitable motions and contain multiple degradations. The challenge is to properly align the successive image shots and merge their complimentary information to achieve high-quality outputs. Towards this direction, we propose Burstormer: a novel transformer-based architecture for burst image restoration and enhancement. In comparison to existing works, our approach exploits multi-scale local and non-local features to achieve improved alignment and feature fusion. Our key idea is to enable inter-frame communication in the burst neighborhoods for information aggregation and progressive fusion while modeling the burst-wide context. However, the input burst frames need to be properly aligned before fusing their information. Therefore, we propose an enhanced deformable alignment module for aligning burst features with regards to the reference frame. Unlike existing methods, the proposed alignment module not only aligns burst features but also exchanges feature information and maintains focused communication with the reference frame through the proposed referencebased feature enrichment mechanism, which facilitates handling complex motions. After multi-level alignment and enrichment, we re-emphasize on inter-frame communication within burst using a cyclic burst sampling module. Finally, the inter-frame information is aggregated using the proposed burst feature fusion module followed by progressive upsampling. Our Burstormer outperforms state-ofthe-art methods on burst super-resolution, burst denoising and burst low-light enhancement.

Network Architecture

Installation

See install.yml for the installation of dependencies required to run Burstormer.

conda env create -f install.yml

Citation

If you use Burstormer, please consider citing:

@article{dudhane2023burstormer,
          title={Burstormer: Burst Image Restoration and Enhancement Transformer},
          author={Dudhane, Akshay and Zamir, Syed Waqas and Khan, Salman and Khan, Fahad Shahbaz and Yang, Ming-Hsuan},
          journal={arXiv preprint arXiv:2304.01194},
          year={2023}
        }

Also, check our CVPR-22 paper: Burst Image Restoration and Enhancement

Code is available here

@inproceedings{dudhane2022burst,
                title={Burst image restoration and enhancement},
                author={Dudhane, Akshay and Zamir, Syed Waqas and Khan, Salman and Khan, Fahad Shahbaz and Yang, Ming-Hsuan},
                booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
                pages={5759--5768},
                year={2022}
              }

Contact

Should you have any question, please contact [email protected]

Acknowledgment: This code is based on the NTIRE21_BURSTSR toolbox.

burstormer's People

Contributors

Stargazers

Watchers

Forkers

ayankumarbhunia bluedream1121 liuguoyou pariasm

burstormer's Issues

Reg.. testing burst denoising on real dataset

Hello @akshaydudhane16!

I want to test burst denoising on real dataset. I've seen that network takes as an input both noisy image together with its noise_estimate calculated from read and shot noises.

Why are we sending this noise_estimate as an image in concatenation with the noisy image as an input?
If I want to test real image, how can I calculate this noise_estimate?

Can you please clarify the same above..

Thank you
Gopi

Can't reproduce the result for BurstSR dataset(Track2)

Hi, thanks for your wonderful work. I am also working on BurstSR and your codes have helped me a lot.

While I have some problems when I try to fine-tune an SR model on the BurstSR dataset. I firstly train the SR model on the Synthetic dataset and everything is OK. Then I fine-tune this model on the BurstSR dataset. The training loss keeps decreasing while the validation PSNR only grows at the beginning and drops gradually after it achieves the best result.
Below image is PSNR result per epoch. X axis is epoch and Y axis is PSNR.

Changed points were as below.

spatial_color_alignment.py : Change lstsq from "c = torch.lstsq(ir.t(), iq.t())" to "c = torch.linalg.lstsq(iq.t(), ir.t())" due to Pytorch version.
correlation.py : Change "@cupy.util.memoize(for_each_device=True)" to "@cupy.memoize(for_each_device=True)" due to version.
PWC pre-train : I used download.bash in pwcnet folder.
Train epoch setting is 25 epoch because your original code was 25 epoch.
Train patch size is 24 and validation patch size is 80 as like your released code.

Have you met the same problem? I would appreciate it a lot if you could help me figure this out.

Thanks.

Ask for results of other model's visualization results

Hi, thanks for your excellent work!
Could you please upload the visualization results of compared models?

Checkpoints for Synthetic Burst SR does not match with the model

Dear authors!

Thank you for sharing your work with the community. Recently, I tried to inference Burstormer on Synthetic Burst SR validation set and found that provided checkpoint does not match with the model:

Missing key(s) in state_dict: "back_projection1.feat_fusion.0.weight", "back_projection1.feat_fusion.0.bias", "back_projection1.feat_expand.0.weight", "back_projection1.feat_expand.0.bias", "back_projection2.feat_fusion.0.weight", "back_projection2.feat_fusion.0.bias", "back_projection2.feat_expand.0.weight", "back_projection2.feat_expand.0.bias".
Unexpected key(s) in state_dict: "back_projection1.diff_fusion.weight", "back_projection1.feat_fusion.weight", "back_projection1.feat_expand.weight", "back_projection2.diff_fusion.weight", "back_projection2.feat_fusion.weight", "back_projection2.feat_expand.weight".
size mismatch for align.alignment0.back_projection.encoder1.0.norm1.body.weight: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([48]).
size mismatch for align.alignment0.back_projection.encoder1.0.norm1.body.bias: copying a param with shape torch.Size([96]) from checkpoint, the shape in current model is torch.Size([48]).
size mismatch for align.alignment0.back_projection.encoder1.0.attn.qk.weight: copying a param with shape torch.Size([192, 96, 1, 1]) from checkpoint, the shape in current model is torch.Size([96, 48, 1, 1]).

I increased the num. of channels twice in ref_back_projection.encoder1, but still there is a mismatch in no_ref_back_projection. Checkpoint contains diff_fusion block weights, but the Burstormer model does not have it.

Can you please help to resolve this issue?

Request for Enhancement code release

Hi,
I am very thankful for what you have done!I would like to do some experiments with enhancement function.I hope to see enhancement code to do me this favor.Thank you !

Download link of the pre-trained weights is wrong. It is BIPNet pre-trained model link.

Hello. Thank you for releasing your impressive CVPR 2023 work, Burstormer!

I have one question about the download links of pre-trained model.

When I go into the trained model page of Burst Super-Resolution, the pre-trained weights is BIPNet, not Burstormer.

I think it is a mistake of the link.

Can you update the link of pre-trained weights to the one of Burstormer?

Thank you.

Question for Cyclic Burst Sampling in NRFE.

I saw the Cyclic Burst Sampling module in your article, and I think this module is very cleverly designed, but I can't find the definition of the code for this module in your code, can I ask you where this code is? There is one more question, the weights in the trace 1 part you gave me don't seem to match the model, is the weight file uploaded wrongly?

thank you

Question for training time

Thanks for release codes.

I would like to train your Burst SR model using released codes but I want to ask something before training.

In the paper, it seems the model trained with 4 RTX6000 GPUs.

Could you share how long the train took?

Thanks.
Luis Kang.

Regarding Batchsize

Dear authors,
thanks for opensource the code!
I noticed that for the network, the fine line in the forward model is always burst = burst[0]
So does the training and inference only support batchsize=1? Or if there is any tricks we can do larger batchsize?
Thanks,
Ke

Question for pre-trained weights link.

Hello, thank you for open source code!

I have trouble using pre-trained weights of Burst Super-resolution page.
There's no weights about back_projection.

I check the similar question posted earlier.
Can you check one more time if it is right for Burstormer's pretrained weights?

Thanks.

Request for denoising and low-light enhancement training code

Thanks for share great network.

I'm waiting your training code for both denoising and low-light enhancement!

Will you have plan to release those training codes?

About val datasets in SR task

Hi,
syn_burst_val and burstsr_dataset datasets can not be download now .Server not found.

Question for training of Burst De-noising.(loss function, data generation and training tricks)

Hello,

We noticed that you mentioned in the article that all training utilized L1 Loss. However, in the Burst Denoising section, you referred to the experimental setups of KPN and BPN, where KPN employed the Basic Loss (L2 Loss + gradient loss). We are curious about the Loss function employed in your Burst Denoising.

Furthermore, in Burst Denoising, when converting Open Image data, apart from what was mentioned in KPN, are there any additional steps or specific considerations? During the training phase of Burst Denoising, were there any other training techniques applied? Currently, we are unable to replicate the training accuracy.

We eagerly await your response!

What is the purpose of the BFF module in the network.

Hello, thank you very much for open source code. What is the function of the BFF module designed in this paper? I read the description of the BFF module in the paper. Unfortunately, I didn't get it. Can the author explain the role of BFF module in more detail? Thank you very much and look forward to your reply.