The official codes of our ML Reproducibility Challenge 2022 report: [Re] Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization

Original Paper: Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization by Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia, Lei Zhang.

Original Repository: EFDM by YBZh

README file of the original paper are given in OriginalPaper/README.md. The exact steps to reproduce the results can be found in this README file.

The trained weights and training logs are available at this anonymous link

Summary: In this reproducibility study, we present our results and experience during replicating the paper, titled Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization. In real‐world scenarios, the feature distributions are mostly much more complicated than Gaussian, so only mean and standard deviation may not be fully representative to match them. This paper introduces a novel strategy to exactly match the histograms of image features via the Sort‐Matching algorithm in a computationally feasible way. We were able to reproduce most of the results presented in the original paper both qualitatively and quantitatively.

Scope of Reproducibility: In the scope of this study, we aim to reproduce all the qualitative and quantitative results on two tasks, namely Arbitrary Style Transfer (AST) and Domain Generalization (DG). Moreover, we investigate the capability of forming better style representations by EFDM in another recent study.

Below we show a brief implementation of it in PyTorch:

import torch
def exact_feature_distribution_matching(content, style):
    assert (content.size() == style.size()) ## content and style features should share the same shape
    B, C, W, H = content.size(0), content.size(1), content.size(2), content.size(3)
    _, index_content = torch.sort(content.view(B,C,-1))  ## sort content feature
    value_style, _ = torch.sort(style.view(B,C,-1))      ## sort style feature
    inverse_index = index_content.argsort(-1)
    transferred_content = content.view(B,C,-1) + value_style.gather(-1, inverse_index) - content.view(B,C,-1).detach()
    return transferred_content.view(B, C, W, H)

Reproduction Results:

• Visual comparison of the reported and our produced results on standard (the first two rows) and photo‐realistic (the last two rows) style transfer.

• Illustration of style interpolation between a single content image and four style images.

• Average running time of prior methods and proposed strategy used in AST on a 512px image. Note that the compared methods run on a single Tesla V100, while our measurement has been done on a single RTX 2080Ti.

• Illustration of the content‐style trade‐off with different λ values.

• Comparison of reproduced results of different feature distribution matching strategies applied in the original paper.

• Ablation on trade‐off between content and style loss terms.

• DG results of category classification on PACS. (R) refers to our reproduced results

• DG results on person re‐ID task. (R) refers to our reproduced results.

• White‐balance correction results of the recent methods and its variant with EFDM on mixed‐illuminant evaluation set.

In our report, We have reproduced the experiments done on two selected tasks, and compared their results with the reported results. Although our experimental results are not identical to the reported ones, we can validate the claims made by the original study according to these results. The source code for reproducing all experiments can be found in EFDM/ArbitraryStyleTransfer, EFDM/DomainGeneralization/imcls, and EFDM/DomainGeneralization/reid, respectively. Our reproduced results can be found in EFDM/ArbitraryStyleTransfer/reproduced_results and EFDM/DomainGeneralization/reproduced_results.

What was easy: The given code in the original repository was easy to follow, and it was well‐written in general. The authors designed the documentation and the source code in a way that anyone who has fundamental knowledge of Python could run the experiments, or even generate their own stylized image from any content.

What was difficult: We would like to add the reproduced outputs by Histogram Matching (HM) along with the others, however the training of HM was based on CPU and the estimated time to complete a single training was around 15 days in our setup. Consequently, we could not include the reproduced outputs by HM to this report. Moreover, it could not be possible to add t‐SNE visualizations to this report, as in the original paper, due to the lack of clarity in the documentation of its script.

To cite the original paper and the reproduction paper in your publications, please use the following bibtex entries:

@inproceedings{zhang2021exact,
  title={Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization},
  author={Zhang, Yabin and Li, Minghan and Li, Ruihuang and Jia, Kui and Zhang, Lei},
  booktitle={CVPR},
  year={2022}
}