Giter Club home page Giter Club logo

torchsr's Introduction

DOI PyPI GitHub

Super-Resolution Networks for Pytorch

Super-resolution is a process that increases the resolution of an image, adding additional details. Methods using neural networks give the most accurate results, much better than other interpolation methods. With the right training, it is even possible to make photo-realistic images.

For example, here is a low-resolution image, magnified x4 by a neural network, and a high resolution image of the same object:

Pixelated image of a butterfly Smooth magnified image High resolution image

In this repository, you will find:

  • the popular super-resolution networks, pretrained
  • common super-resolution datasets
  • Pytorch datasets and transforms adapted to super-resolution
  • a unified training script for all models

Models

The following pretrained models are available. Click on the links for the paper:

Newer and larger models perform better: the most accurate models are EDSR (huge), RCAN and NinaSR-B2. For practical applications, I recommend a smaller model, such as NinaSR-B1.

Expand benchmark results
Set5 results
Network Parameters (M) 2x (PSNR/SSIM) 3x (PSNR/SSIM) 4x (PSNR/SSIM)
carn 1.59 37.88 / 0.9600 34.32 / 0.9265 32.14 / 0.8942
carn_m 0.41 37.68 / 0.9594 34.06 / 0.9247 31.88 / 0.8907
edsr_baseline 1.37 37.98 / 0.9604 34.37 / 0.9270 32.09 / 0.8936
edsr 40.7 38.19 / 0.9609 34.68 / 0.9293 32.48 / 0.8985
ninasr_b0 0.10 37.72 / 0.9594 33.96 / 0.9234 31.77 / 0.8877
ninasr_b1 1.02 38.14 / 0.9609 34.48 / 0.9277 32.28 / 0.8955
ninasr_b2 10.0 38.21 / 0.9612 34.61 / 0.9288 32.45 / 0.8973
rcan 15.4 38.27 / 0.9614 34.76 / 0.9299 32.64 / 0.9000
rdn 22.1 38.12 / 0.9609 33.98 / 0.9234 32.35 / 0.8968
Set14 results
Network Parameters (M) 2x (PSNR/SSIM) 3x (PSNR/SSIM) 4x (PSNR/SSIM)
carn 1.59 33.57 / 0.9173 30.30 / 0.8412 28.61 / 0.7806
carn_m 0.41 33.30 / 0.9151 30.10 / 0.8374 28.42 / 0.7764
edsr_baseline 1.37 33.57 / 0.9174 30.28 / 0.8414 28.58 / 0.7804
edsr 40.7 33.95 / 0.9201 30.53 / 0.8464 28.81 / 0.7872
ninasr_b0 0.10 33.24 / 0.9144 30.02 / 0.8355 28.28 / 0.7727
ninasr_b1 1.02 33.71 / 0.9189 30.41 / 0.8437 28.71 / 0.7840
ninasr_b2 10.0 34.00 / 0.9206 30.53 / 0.8461 28.80 / 0.7863
rcan 15.4 34.13 / 0.9216 30.63 / 0.8475 28.85 / 0.7878
rdn 22.1 33.71 / 0.9182 30.07 / 0.8373 28.72 / 0.7846
DIV2K results (validation set)
Network Parameters (M) 2x (PSNR/SSIM) 3x (PSNR/SSIM) 4x (PSNR/SSIM) 8x (PSNR/SSIM)
carn 1.59 36.08 / 0.9451 32.37 / 0.8871 30.43 / 0.8366 N/A
carn_m 0.41 35.76 / 0.9429 32.09 / 0.8827 30.18 / 0.8313 N/A
edsr_baseline 1.37 36.13 / 0.9455 32.41 / 0.8878 30.43 / 0.8370 N/A
edsr 40.7 36.56 / 0.9485 32.75 / 0.8933 30.73 / 0.8445 N/A
ninasr_b0 0.10 35.77 / 0.9428 32.06 / 0.8818 30.09 / 0.8293 26.60 / 0.7084
ninasr_b1 1.02 36.35 / 0.9471 32.51 / 0.8892 30.56 / 0.8405 26.96 / 0.7207
ninasr_b2 10.0 36.52 / 0.9482 32.73 / 0.8926 30.73 / 0.8437 27.07 / 0.7246
rcan 15.4 36.61 / 0.9489 32.78 / 0.8935 30.73 / 0.8447 27.17 / 0.7292
rdn 22.1 36.32 / 0.9468 32.04 / 0.8822 30.61 / 0.8414 N/A
B100 results
Network Parameters (M) 2x (PSNR/SSIM) 3x (PSNR/SSIM) 4x (PSNR/SSIM)
carn 1.59 32.12 / 0.8986 29.07 / 0.8042 27.58 / 0.7355
carn_m 0.41 31.97 / 0.8971 28.94 / 0.8010 27.45 / 0.7312
edsr_baseline 1.37 32.15 / 0.8993 29.08 / 0.8051 27.56 / 0.7354
edsr 40.7 32.35 / 0.9019 29.26 / 0.8096 27.72 / 0.7419
ninasr_b0 0.10 31.97 / 0.8974 28.90 / 0.8000 27.36 / 0.7290
ninasr_b1 1.02 32.24 / 0.9004 29.13 / 0.8061 27.62 / 0.7377
ninasr_b2 10.0 32.32 / 0.9014 29.23 / 0.8087 27.71 / 0.7407
rcan 15.4 32.39 / 0.9024 29.30 / 0.8106 27.74 / 0.7429
rdn 22.1 32.25 / 0.9006 28.90 / 0.8004 27.66 / 0.7388
Urban100 results
Network Parameters (M) 2x (PSNR/SSIM) 3x (PSNR/SSIM) 4x (PSNR/SSIM)
carn 1.59 31.95 / 0.9263 28.07 / 0.849 26.07 / 0.78349
carn_m 0.41 31.30 / 0.9200 27.57 / 0.839 25.64 / 0.76961
edsr_baseline 1.37 31.98 / 0.9271 28.15 / 0.852 26.03 / 0.78424
edsr 40.7 32.97 / 0.9358 28.81 / 0.865 26.65 / 0.80328
ninasr_b0 0.10 31.33 / 0.9204 27.48 / 0.8374 25.45 / 0.7645
ninasr_b1 1.02 32.48 / 0.9319 28.29 / 0.8555 26.25 / 0.7914
ninasr_b2 10.0 32.91 / 0.9354 28.70 / 0.8640 26.54 / 0.8008
rcan 15.4 33.19 / 0.9372 29.01 / 0.868 26.75 / 0.80624
rdn 22.1 32.41 / 0.9310 27.49 / 0.838 26.36 / 0.79460

All models are defined in torchsr.models. Other useful tools to augment your models, such as self-ensemble methods and tiling, are present in torchsr.models.utils.

Datasets

The following datasets are available. Click on the links for the project page:

All datasets are defined in torchsr.datasets. They return a list of images, with the high-resolution image followed by downscaled or degraded versions. Data augmentation methods are provided in torchsr.transforms.

Datasets are downloaded automatically when using the download=True flag, or by running the corresponding script i.e. ./scripts/download_div2k.sh.

Usage

from torchsr.datasets import Div2K
from torchsr.models import ninasr_b0
from torchvision.transforms.functional import to_pil_image, to_tensor

# Div2K dataset
dataset = Div2K(root="./data", scale=2, download=False)

# Get the first image in the dataset (High-Res and Low-Res)
hr, lr = dataset[0]

# Download a pretrained NinaSR model
model = ninasr_b0(scale=2, pretrained=True)

# Run the Super-Resolution model
lr_t = to_tensor(lr).unsqueeze(0)
sr_t = model(lr_t)
sr = to_pil_image(sr_t.squeeze(0).clamp(0, 1))
sr.show()
Expand more examples
from torchsr.datasets import Div2K
from torchsr.models import edsr, rcan
from torchsr.models.utils import ChoppedModel, SelfEnsembleModel
from torchsr.transforms import ColorJitter, Compose, RandomCrop

# Div2K dataset, cropped to 256px, width color jitter
dataset = Div2K(
    root="./data", scale=2, download=False,
    transform=Compose([
        RandomCrop(256, scales=[1, 2]),
        ColorJitter(brightness=0.2)
    ]))

# Pretrained RCAN model, with tiling for large images
model = ChoppedModel(
    rcan(scale=2, pretrained=True), scale=2,
    chop_size=400, chop_overlap=10)

# Pretrained EDSR model, with self-ensemble method for higher quality
model = SelfEnsembleModel(edsr(scale=2, pretrained=True))

Training

A script is available to train the models from scratch, evaluate them, and much more. It is not part of the pip package, and requires additional dependencies. More examples are available in scripts/.

pip install piq tqdm tensorboard  # Additional dependencies
python -m torchsr.train -h
python -m torchsr.train --arch edsr_baseline --scale 2 --download-pretrained --images test/butterfly.png --destination results/
python -m torchsr.train --arch edsr_baseline --scale 2 --download-pretrained --validation-only
python -m torchsr.train --arch edsr_baseline --scale 2 --epochs 300 --loss l1 --dataset-train div2k_bicubic

You can evaluate models from the command line as well. For example, for EDSR with the paper's PSNR evaluation:

python -m torchsr.train --validation-only --arch edsr_baseline --scale 2 --dataset-val set5 --chop-size 400 --download-pretrained --shave-border 2 --eval-luminance

Acknowledgements

Thanks to the people behind torchvision and EDSR, whose work inspired this repository. Some of the models available here come from EDSR-PyTorch and CARN-PyTorch.

To cite this work, please use:

@misc{torchsr,
  author = {Gabriel Gouvine},
  title = {Super-Resolution Networks for Pytorch},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Coloquinte/torchSR}},
  doi = {10.5281/zenodo.4868308}
}

@misc{ninasr,
  author = {Gabriel Gouvine},
  title = {NinaSR: Efficient Small and Large ConvNets for Super-Resolution},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Coloquinte/torchSR/blob/main/doc/NinaSR.md}},
  doi = {10.5281/zenodo.4868308}
}

torchsr's People

Contributors

coloquinte avatar txfs19260817 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

torchsr's Issues

Error in loading URL

This error maybe due to breaking changes in torchvision module.

The following line throws an error:

from torchvision.models.utils import load_state_dict_from_url

The following change works:

from torch.hub import load_state_dict_from_url

A question of how you make it not exceed the the cuda memory

I have also a very intersting question to ask you that when I test the model on div2k validation set, my own codes always tells me that my cuda memory is run out, but in your code the same model works perfectly on the div2k validation set so how do you make this work?

random-crop

hi

sorry for bothering, and recently i have used your scripts which really great work.

may i ask one question that "i have found "RandomCrop" is appended to training dataset, and i wonder how it works?

thanks a lot for your attention and support.

br,
sc

Collaboration

Hi there!

First of all, amazing work! I am developing a CV library called glasses and I was thinking to also add SR models and I came across your repo.

What if we merge the two? Would you like to talk about it?

Let me know!

Thank you :)

Cheers,

Francesco

SRCNN and FSRCNN

Alot of these networks are still too heavy for cpu inference would you be open to a PR to add these models ?

Citing NinaSR?

First, thanks a lot for creating this one-stop repository!

I want to use the NinaSR in one of my works and would like to cite it. Any chance you could share that information on the readme file?

Artefacts in results

If I run the sample code on image indexed 5:

from torchsr.datasets import Div2K
from torchsr.models import ninasr_b0
from torchvision.transforms.functional import to_pil_image, to_tensor

# Div2K dataset
dataset = Div2K(root="./data", scale=2, download=False)

# Get the first image in the dataset (High-Res and Low-Res)
hr, lr = dataset[5]

# Download a pretrained NinaSR model
model = ninasr_b0(scale=2, pretrained=True)

# Run the Super-Resolution model
lr_t = to_tensor(lr).unsqueeze(0)
sr_t = model(lr_t)
sr = to_pil_image(sr_t.squeeze(0))
sr.show()

I get the results dotted with artefact pixels. See blue, red dots:
image

My env:

certifi==2023.7.22
charset-normalizer==3.2.0
filelock==3.12.2
fire==0.5.0
idna==3.4
Jinja2==3.1.2
MarkupSafe==2.1.3
mpmath==1.3.0
networkx==3.1
numpy @ file:///D:/bld/numpy_1688887186903/work
Pillow==10.0.0
requests==2.31.0
six==1.16.0
sympy==1.12
termcolor==2.3.0
torch==1.8.1
torchsr==1.0.4
torchvision==0.9.1
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1688315532570/work
urllib3==2.0.4

It feels like so some sort of overflow but I could not find it.

About the ssim and psnr of the package piq

Hello, I really love your program because it helps me a lot of understanding the image super-resolution via pytorch. However, I want to tell you something which I have found during my usage of running your code. I have compared the value of a same image group (Sr images and Hr images) from your code and other official methods, and I found a little difference when it executes the calculation of SSIM and PSNR between the PIQ (yours) and the official method of matlab (from the code of RCAN). Then I found the way of calculation of SSIM and PSNR from RCAN and SwinIR has the same value which is not the same as the PIQ method. So would you like that I provide you the calculation method from SwinIR, and we could make an intergration in to your code? If you are interested in that, don't be hesitate of contacting me.

A question for the parameters during the trainig

Hi, I am really grateful that you give us a very clear implementation of these SR models, by the way I would like to ask you about the parameters(optimiser, data-augement, epochs ,etc) that you have choosen in the reproduction during the training, especially for the EDSR huge, because I have problems when I apply the parameters in the paper on your training models, as I could not reach the pnsr and ssim as high as the paper. Your response can be really helpful for me and I would like to thank you in advance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.