About the result about chainer-fast-neuralstyle HOT 64 OPEN

Some collected models are now available at https://github.com/gafr/chainer-fast-neuralstyle-models

please mind, all of them are trained with the old version, will update soon.

from chainer-fast-neuralstyle.

Our assumptions were correct. Size does matter. The result is much more appealing. Here's my picture processed in style of Kandinsky painting. Both models were trained with batchsize 1 for 2 epochs.

First uses the default size of 256.
Second uses 512 px style and square centered crops of training data, rescaled with bicubic interpolation so that the smallest side is 512. Training took 44 hours on GTX 970.

The downsides are that cropping loses some of the detail, while upscaling blurs the image slightly. I guess you can get rid of cropping and squeeze whatever image it is into square dimensions, don't know if it will turn out in feature distortion though.

from chainer-fast-neuralstyle.

@6o6o would you mind sharing your model?

I am also currently training a new one (256), due to the time intensive model creation, we should consider opening a central repo with some models. What do you think?

from chainer-fast-neuralstyle.

Thanks.

Yes, I know my result is a little block noisy.
The model which this repo contains was not trained with the same hyper parameter as paper's.
I didn't know the exact value of lambda_feat and lambda_style when I trained the model, because I couldn't found them in the paper.
But I found them here, and gonna update repo!

It seemed lambda_style which I used was too large.
And I think that uniform lambda_style weight might not be good, because I found that gram_matrix losses are quite different from each layer.

from chainer-fast-neuralstyle.

Yes, there are so many hyper-parameters began from the seminal work "neural algorithm for artistic stylization". Currently most re-implementation follows uniform lambda_style weights. I have done lots for experiments based justin's implementation. Seems your current result only learned the low-level style, e.g. color, no higher-level such as strokes.
And in the reddit blog , justin offer a supplementary material for more details. may be helpful for u.

from chainer-fast-neuralstyle.

@yusuketomoto any news? What weights have you tried?

I tried lambda_feat=1.0 with lambda_style from {1.5, 3.0, 5.0, 7.0, 10.0} with all other weights default. All results are noisy and they have these vertical stripes. How how you managed to train the default starrynight.model that is included in this repo?

from chainer-fast-neuralstyle.

it might be that your convolution kernel size is 4? whereas the original VGG uses 3, which should give more precise gfx.

from chainer-fast-neuralstyle.

Thx for sharing codes. I trained with default parameters in this repo on mscoco 2014 dataset, tried several style images including starrynight, results are all noisy and weird. Could anyone reproduce the default starrynight.model in this repo?

from chainer-fast-neuralstyle.

@mionyu I'm getting the same here. I've trained on several pictures and getting unusable results. I'm getting a strange frame affect on all the models , no matter what the source is.
I really wanted to use this in an art project next weekend, but unfortunately the results are not acceptable at all. See the examples. Any help is welcomed. I've used the standard command line without altering any parameters.

from chainer-fast-neuralstyle.

@valentinvieriu Try batchsize 1, you might get better result.
I can't have confidence, but I'm suspecting some bugs are in batch_matmul or style loss computation with batch.

from chainer-fast-neuralstyle.

@yusuketomoto Hello.

Thank you for a nice implementation.

I have one question:
I keep getting strange spots with random values when I apply the algorithm.
Exactly like on the last picture with cat that you have sent on the right side right near the border.
Do you know if there is a way to avoid it?

Thank you.

from chainer-fast-neuralstyle.

Thank you @yusuketomoto . The batchsize 1 did help. The image looks more nice and appealing now.

The border is still there. If you have any idea how to remove that, would be appreciated. Do you think it has anything to do with the training data?

from chainer-fast-neuralstyle.

@yusuketomoto, Thanks for a great implementation here - really enjoy playing with it. Since the features of the style image are being replicated in a pretty granular way, when the input image is larger than style img size, I wondered if you have tried to train with with larger resolution than the 256, 256 that the code resizes it to do.

I guess training will take proportionally longer time then. Just curious if you have experience with doing that? Is there a way to alter the model to create larger features in the output image once its trained?

from chainer-fast-neuralstyle.

@nikolajspetersen, I'm curious about this too. Furthermore, since the resolution is always 256x256, regardless of the aspect ratio, it tends to distort shapes in training data. Correct me if i'm wrong, or this is irrelevant.

Currently I'm training a model on 512 px style image and center crops of training data of the same size. Will post back comparison images once it's ready.

from chainer-fast-neuralstyle.

@6o6o nice work!
I'm using gtx 1080 - 8GB ram and try to train with size 512 but get out of memory at "feature_hat = vgg(y)". Can you tell me how to reduce memory size ?

from chainer-fast-neuralstyle.

@codevui thanks. Check if you have everything right. Basically, just replace all instances of 256 with the desired value and optionally set resampling filter to 2. It shouldn't be using more than 3600 MB. Actually, 8GB should allow you to go all the way up to 720px

from chainer-fast-neuralstyle.

@6o6o nice work. it really helps me a lot.
by the way, could you please tell me the exact value of lambda_feat and lambda_style you used?

from chainer-fast-neuralstyle.

@6o6o the problem is cudnn. After i make cudnn correct for chainer, everything is ok!
Thank you!

from chainer-fast-neuralstyle.

@bucktoothsir, glad to help. I haven't touched those values, as I'm not sure about them. Experimenting with trial and error takes too much time. I noticed though, in the paper possible range for lambda_tv is between 1e-4 and 1e-6, while here it's 10e-4. Don't know if it's a typo, or am I missing something. You can try adjusting lambda_tv and lambda_style and post back if anything interesting comes up

from chainer-fast-neuralstyle.

@gafr, sure. I'm all for it. Would be great if we could collect some good models for everyone to use. Should I just create a new repo? Or where do I put them?

from chainer-fast-neuralstyle.

@6o6o just invited you to a repository, give me 10 minutes for a README and a structure

from chainer-fast-neuralstyle.

As I found subtracting mean images before image transformation network is not so effective, I'd like to change codes.
However this change will break backward compatibility of models.
If you create a repository, please note that.

from chainer-fast-neuralstyle.

@yusuketomoto would you give me the permission to collect/share your current model in a repository, or should I just link it to your repository. I will note down the parameters and/or version

from chainer-fast-neuralstyle.

@gafr I updated the codes, models, readme text. Sorry for the inconvenience!

from chainer-fast-neuralstyle.

@6o6o The second result was great! Thanks for your work! I trained some models but the result was terrible. Can you help me for some details?

1. Training by the entire database will spend too much time, Do you have experience in the impact of the number of training images? How many differences between 10000 images and the entire database?
2. Is "cropping" meaning keep the aspect ratio invariant while resize the smaller side of image to 512, and then square centered crop it? I replaced "256" in "train.py" to "512", Do I need to change the size in "net.py"?
3. In addition to image preprocessing, other parameters are all default settings?

from chainer-fast-neuralstyle.

1. Reducing the dataset to 10k significantly deteriorates the quality. Definitely not recommended.
2. Scale the images so that the smallest side is of desired value, preserving the aspect ratio, then crop. This is optional. I read that VGG models were trained like this here, so I thought the same technique could be applied to styles. Changes should be done to train.py only.
3. All parameters at default. You can try increasing --lambda_tv to smth like 10e-3. Lowering it as proposed in paper tends to produce more artifacts.

If you wish I can fork the project and integrate the changes. I don't think PR is a good idea since the enhanced settings take considerably longer to train and may not be suitable for everyone.

from chainer-fast-neuralstyle.

@6o6o Write to me at my email - [email protected]
I would like to cooperate with you.

from chainer-fast-neuralstyle.

Anyone imolemented a video version? It should be simple considering the speed of this alghoritm

from chainer-fast-neuralstyle.

@6o6o Thank you very much! Please fork the project then I can learn from it. Looking forward to more communication.

from chainer-fast-neuralstyle.

@rayset I've done some experiments. The result is quite steady, preserving consistency between frames for moving objects / transitions, except for some jitter in the backgrounds, areas with subtle color change, compression artifacts, etc. See here and here

from chainer-fast-neuralstyle.

@6o6o Do you mind sharing your code for the frame by frame conversion? Even by pm if it's not clean, I'm not really good with scripts and I would love a basis to start from .

Thanks in advance

from chainer-fast-neuralstyle.

@yusuketomoto have u ever fix the batchsize( >=2) problem? mini-batch traininig will be faster and stable

from chainer-fast-neuralstyle.

@austingg No, I'm afraid not. As reconstruction without style losses work fine, there would be some bugs in style loss computation with minibatch.

from chainer-fast-neuralstyle.

I still have some glitchs, here is a close-up, does anyone know why and how to avoid them ?

from chainer-fast-neuralstyle.

I'm also fighting with those dots, I'd love to get riferimento of them

from chainer-fast-neuralstyle.

@ttoinou what is your training params and epochs?

from chainer-fast-neuralstyle.

@austingg : I used default params, 1,5 epochs (1 epoch at 80k images, and 40k images from the second epoch)

from chainer-fast-neuralstyle.

same but with 2 epochs. old version tho.

from chainer-fast-neuralstyle.

@ttoinou
I have trained a model with defautl params and 2 epochs with imagent val (50k), seem quite smooth

from chainer-fast-neuralstyle.

@yusuketomoto @6o6o @rayset
how to remove the border artifact? It come out in the first epoch, and it seems doesn't get better in the second epoch (cry)

from chainer-fast-neuralstyle.

@austingg Would you have a link to your image set ?
If you want to get rid of the border artifact and can't fix the algorithm, you could always expand the content dimensions (black or white borders, or even repetitions of image's original pixels) and then only keep the interesting area.

from chainer-fast-neuralstyle.

@ttoinou I use ILSVRC2012 val dataset for training.
sorry for the last comments typo, maybe mislead you. 😢 , actually, I haven't remove the border artifact.

I know what you said, first extend the image, and generating image and then crop.
It's weird, cause sometime there are no border artifact.

Maybe the problem is 4x4 conv filter or tv, but i am not sure. I will work on this problem.

from chainer-fast-neuralstyle.

@austingg do you train on cropped images or fullsize? I find fullsize training to decrease border and pixelation a bit, but produces artifacts. See crop vs. fullsize

from chainer-fast-neuralstyle.

@6o6o I use fullsize (just stretch the image to 256x256 or 512x512)

from chainer-fast-neuralstyle.

Eh eh did you found out about ILSVRC2012 from deepart.io?
I was just messing around with it.
There must be something wrong in most implementation(dots and border,a almost every single one on GitHub has them).

Do you still get the border on the most recent version? What about dots?

from chainer-fast-neuralstyle.

The project was updated. Several changes were integrated, including the ones I was talking about in my above comment, mainly adjustable training image size and retaining aspect ratio of dataset images by cropping, but done slightly differently.

Whether to crop or not to crop remains an open question. I made a short slideshow for comparison. I think cropping is generally more appropriate, because in most cases it gets rid of useless areas on photos, e.g. backgrounds, irrelevant details or insignificant features. Only in some rare cases I would prefer not to crop, like img10 and img20, where we loose a person's face, which might be important. I find cropped result to be superior, except for more pronounced pixel dithering pattern and a frame border around the picture, which are not as evident in a full version, but some artifacts occur there. But again, it's only one style, others may have different effects.

I updated my fork as well, some of the differences are

• toggle cropping with --fullsize option (default is to crop)
• different cropping implementation. I just use built-in function ImageOps.fit, which produces images in the slideshow above
• bicubic and Lanczos resampling when scaling dataset and input style images provides sharper shrinking

See readme in my fork for more info. @logic1988 you might be interested in it.
@rayset I also included a quick and dirty video processing bash script genvid.sh. It extracts frames, transforms them one at a time and then composes back together. Tested with YouTube .mp4 videos. The speed could be optimized by excluding model loading on every iteration, but it's ok for now.

from chainer-fast-neuralstyle.

@6o6o awesome fork, I noticed that in the update they added all the feaures you discussed.
may I ask you an exampe of the usage of genvid? I replaced the first rows like this:

INPUT=primo.mp4
OUTPUT=video.mp4
MODEL=models/composition.model

and it works till the estraction. in row 16 I get a bad substitution error (I guess when you try (?) to truncate the first 7 characters from f. what am I doing wrong?

from chainer-fast-neuralstyle.

Ok, I got to the bottom of this. @rayset change line 8 to
FILENAME="${INPUT##*/}"
It was using first argument $1 there, and since you didn't pass no params, it choked

from chainer-fast-neuralstyle.

you were right! works perfect now.

I'm reading about frame consistency: https://github.com/manuelruder/artistic-videos do you think it would be hard to implement this procedure? the video looks pretty nice even now, but I wonder if it can be improved

from chainer-fast-neuralstyle.

Ah, that algorithm is superb, the results are very impressive. I haven't tried it, but from what I heard it's

• much harder to implement since you need to couple video rendering with image generation
• much slower, since it's based on original neural-style, even though it shouldn't redraw overlapping frame areas, but still...

I doubt same technique can be applied here, since it's using preconfigured network, you can hardly alter its output at the generation step. Only by changing the model you may be able to achieve variation.

This one is much faster and easier to use, therefor better suited for general public. As for optimization / reducing jitter in some areas, it maybe be helpful to process frames with some sort of median filter before transformation to smooth out noise and subtle details but keep the edges rough. I'll do some more experiments will let you know if anything interesting comes up.

from chainer-fast-neuralstyle.

@6o6o : great ! Thanks for the updates. I also forked your project :) !
Look this post : #25 .

from chainer-fast-neuralstyle.

Hi everyone,
this implementation uses 4 size kernel and I am pretty sure a lot of people(including me) tried to use kernel size 3. But whoever tried it, knows that deconvolution layer does not handle sizes correctly.
So, this pull request deals with it :) Not sure if it's gonna be accepted by the main repo :)
ddd8643360282a074d9fcbdcedbac470713132d3

Good luck :)

from chainer-fast-neuralstyle.

Does the kernel size of 3 really remove the border artifact and the dots?

from chainer-fast-neuralstyle.

emmm not sure yet :) it's in the process.
Also, do you still get borders when using batch_size 1?

from chainer-fast-neuralstyle.

Yep ,the default value is 1

from chainer-fast-neuralstyle.

Really, no one else can create a quality model for "yusuketomoto/chainer-fast-neuralstyle"
photo processing? A total of 6 models, but they look good - kandinsky, edtaonisl.
If use a similar pattern?

from chainer-fast-neuralstyle.

What would be a quality model for photo processing ?!

What is the benefit of using kernel size 3 ?

from chainer-fast-neuralstyle.

@ttoinou several reasons I see there.

1. being consistent with the original paper, as the paper states that the kernel size should be 3 on 2nd and 3rd convolution layers. Also on 1st and 2nd deconvolution layers.
2. Less parameters to train
3. Also, I think usually kernel size and other parameters are cross-validated. So it's not a coincidence that it was chosen to be 3.

You can experiment as much as you want :) I just gave the ability to do so.

from chainer-fast-neuralstyle.

@yusuketomoto @cryptexis @rayset
I think that the border artifact is produced by the total variance module, which is implemented by convolution function with zero padding, so the borders have stronger gradients.

from chainer-fast-neuralstyle.

@cryptexis thanks. Do I also have to change the pad parameter in the neighbouring layers? Haven't found anything in the papers about it.

from chainer-fast-neuralstyle.

@yusuketomoto great implementation, really enjoying working with it. I thought I would share a render of some test footage using @6o6o 's Kandinsky model. @rayset definitely keen on looking for a way to implement https://github.com/manuelruder/artistic-videos in some way. Here is the 1080p output: https://youtu.be/sKOa3tgHQho (youtube does make a bit of a meal of it) and a comparison with the source: https://youtu.be/ovMKXzkPMsw

Example still:

I am training some models now, and will be sure to share them when done.

from chainer-fast-neuralstyle.

@mstrazds I notice that the artifacts and dots are much harder to find in the video implementation. Why do you think that is?

from chainer-fast-neuralstyle.

@DylanAlloy When the video was encoded on youtube it reduced the quality significantly. As a part of this the 'dots' or pixels seem to be either removed to the high compression or harder to perceive. The video was constructed by rendering out stills from generate.py and then stitching them together using ffmpeg using x264 for compression. There were no tricks for merging frames or anything of the like.

from chainer-fast-neuralstyle.

Check out http://neuralstyler.com/
Turn Your Videos/Photos/GIF into Art

https://github.com/rupeshs/neuralstyler

from chainer-fast-neuralstyle.