removing the 'dots' about chainer-fast-neuralstyle HOT 69 OPEN

There is a paper with what they claim is a solution to address the issue here.

from chainer-fast-neuralstyle.

4 days. 1070 should be twice as fast out of the box. Plus, theoretically you could double that speed by leveraging fp16 datatype, so 4 times faster total, though I'm not sure it's widely supported yet, you may need to tweak the code

from chainer-fast-neuralstyle.

So I am halfway through the third epoch. The image is looking pretty good, but I am still getting a pattern of dots.

Here you can see a closeup:

And like ttoinou mentioned I am getting these blobs every now and then. They don't seem to be going away with more training.

I suppose I just need to wait for a code update?

from chainer-fast-neuralstyle.

I went through 2 and a half epochs of the full coco dataset. 512 image size, 512 style size. Nothing too crazy. Here is the style image:

Here is the model if you want to play with it. Just rename it to .model

candy-512_2_49000.zip

I stopped it early, but I was training for like 30 hours on my 1070. Its a slow process. I might resume the training later and see if the dots get any better.

from chainer-fast-neuralstyle.

The images are always upscaled so that the smallest side is image_size. The problem is that most of the images in the dataset are 640x480, and 480 -> 512 causes slight blurring, but it's ok i guess, not a big deal. The problem would arise if you want to go higher. On 8GB GPU you could technically go up to 720px and that's nearly 2x upscaling - not a good thing. The solution would be to redownload every image from flickr in its original size, at the same time crop and scale to desired dimensions. This would also exclude resizing from training, saving you 0.05 sec from an image and that's more than an hour per epoch, but still probably an overkill for most cases.

from chainer-fast-neuralstyle.

I've been trying to solve this problem for some time now, as well style feature scaling without retraining the model. I thought about what Yusuke Tomoto said about initialization weights of network layers and got curious how the transformation looks like on clean, untrained (freshly initialized) model. These are the results on default settings and stride set to 1 on all layers.

As you can see the dots are initially present in a new model (image 1), so I thought it was an initialization problem. But after trying every initialization method available and not seeing any difference, I decided it was a strided deconvolution problem, where upscaling is not being learned properly even after many epochs, since if we get rid of stride (image 2) the picture looks smoother, but leads to smaller receptive field and huge memory consumption.

A few weeks later a got some insider information that Prisma might be actually using waifu2x to smooth things out and actually scale features. It indeed turned out to be very helpful in noise removal, but not powerful enough on its own to deal with all sorts of artifacts produced by the transformation. So we need to help it a bit by denoising a pic slightly as a preprocessing step before passing it to waifu. The problem is that most noise removal techniques are way too destructive for small subtle details. As I was going through all the GIMP filters, I bumped into one with a weird name "Van Gogh (LIC)" which turned out to be very efficient in blending highly saturated pixels while preserving even the smallest details. After a little research I found that LIC stands for Line Integral Convolution and is used in data flow visualizations. There are python implementations available, but haven't had the time to integrate one. It has a downside of being slow. There are faster alternatives with blending a blurred version of the image in HSV / HSL color space with a little less efficiency.

So, to sum things up, the steps needed for this to work:

1. Transform your image regularly
2. Denoise in GIMP with Filters -> Artistic -> Van Gogh (LIC) with Filter length of 2
3. Reduce noise in waifu2x with -noise_level 3

The results (left - original, right - blended):

from chainer-fast-neuralstyle.

@6o6o No problem, glad it helped. I'm still just dipping my toes in this AI stuff, but traditional image processing/computer vision is something I am very familiar with. I decided to do some tests with it today so we could have a comparison. Here are the results (click on them to see larger versions):

Original	Van Gogh Filter	FFT Spectrum Mask

Overall I'm very pleased with these results. You can see that the vertical stripes in the first Van Gogh filter are gone, and the second image is much sharper. Unfortunately there are still a few checkerboard areas (eg. the lower lip), and there is are some minor horizontal line artifacts in the first picture's result (bottom right corner).

And a filtered version of the original post for completeness:

FFT Spectrum Mask Procedure:
This was done with ImageMagick(with FFTW support) and waifu2x. The commands are:

convert <input> -fft \( +clone  -write phase.png +delete \) +delete -write magnitude.png  -colorspace gray -auto-level -evaluate log 100000  spectrum.png
convert magnitude.png <mask> -compose multiply -composite phase.png -ift <output>

Where <input> is the input file path, <output> is the output file path, and <mask> is the path to the spectrum mask image, which in this case was:

(this will need to be scaled to match your image dimensions to be useful)

Then I ran the output through waifu2x with a noise level of 3. I haven't benchmarked any of this, but waifu2x is probably going to be slowest part, FFTs are pretty fast (it's in the name!), and they can be further sped up with GPUs if needed.

from chainer-fast-neuralstyle.

I updated the code and put it into a separate repo for ease of use: 6o6o/fft-descreen

from chainer-fast-neuralstyle.

I'm creating checkpoint models and I select afterward the best model I can find, but yeah there's always theses dots...
I'm also producing 4K images for video, so the dots will be hidden in the mp4 encoding artefacts.

Did you try https://github.com/DmitryUlyanov/texture_nets ? I think there are no dots in this implementation

from chainer-fast-neuralstyle.

I have tried it! It is great in some ways, but t has its own issues:
DmitryUlyanov/texture_nets#28

The problem with 4k is that the "brush size" would be super small wouldn't it? Is there a way to scale up the pattern without having to train with a super huge style image? Also I assume you have to do 4k in CPU right? That must be pretty slow.

from chainer-fast-neuralstyle.

Yes you're right (brush size impossible to change, huge training) but I'm ready to let my computer crunch data for 40 hours straight ^^ . I'm creating videos for a music album.

I'm dividing the 4K image into 9 or 16 images, processing them with the GPU and then joining them with the CPU. Joining is slow because I'm not a good pythoner, but I'm looking into multiprocessing right now

from chainer-fast-neuralstyle.

@Heartsie a quick hack would be to upscale the image about 1.2 times, do the transformation and then downscale back to back to original size. Downscaling will blend the dots together, but may produce slightly smaller features.

I'm also curious about upscaling feature size without retraining. I assume this might be possible, though I'm not an ML expert to say for sure.

from chainer-fast-neuralstyle.

fwiw Im seeing those same sorts of dots in a lot of my work with https://github.com/DmitryUlyanov/texture_nets

from chainer-fast-neuralstyle.

I have found that texture_nets is slightly better with the artifacts, and
the training is a LOT faster. But unfortunately the artistic quality of the
image just isn't there. It feels more like a grid-like filter than a actual
painting or something.

Are the dots static from image to image? If not we can train a solid grey
style image, then use it to make a stylized image that is composited of
only dots. Then those dots can be subtracted from the artistic images to
remove the static patterns.

If the dots move from each generation then maybe a image can be generated 5
times and then they could all be averaged together.

On Tue, Aug 9, 2016 at 7:12 PM, Chris Novello [email protected]
wrote:

fwiw Im seeing those same sorts of dots in a lot of my work with
https://github.com/DmitryUlyanov/texture_nets

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#33 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJqpIo93QZoQYJZy99WTH5D0Imh69fEeks5qeQlzgaJpZM4JgEw9
.

-=Robin Graham
Postrgram.com

from chainer-fast-neuralstyle.

They depend on many factors, the style image, the training data. Sometimes they may be barely visible, and quite pronounced in other cases. I also noticed they tend to fade away with additional training, the 4-th epoch produces a more smooth image that 2-nd.

from chainer-fast-neuralstyle.

4th! That must take forever. How many images do you train with? The full coco?

from chainer-fast-neuralstyle.

Yep ) I don't do it regularly, just once, for testing only. Here's a few samples. This style had a particularly pronounced pattern, but most of the times it's not worth the effort.

from chainer-fast-neuralstyle.

Wow, how long did that take? I have a 1070 and I bet that would take me 2 days with a 512 size style image

from chainer-fast-neuralstyle.

I thought FP16 was just 2x more memory and that the operations were still floating ops.
I'll have to test this 4 epoch thing. From what I've seen so far, the models is never stable and doesn't really get better when training too far.

from chainer-fast-neuralstyle.

Only do it if you're not satisfied with your current results. Usually it hardly gets any better and sometimes lower epochs may look even superior.

UPD: NVIDIA turned out to be greedy bastards, artificially limiting FP16 compute performance on 10xx series of consumer cards. You'd have to go for a higher-end card based on GP100 chip, namely Tesla or Titan XP, to benefit from speed boost. This is common practice among big companies, intentionally crippling lower-end products in order not to hurt the higher-end sales. I hope AMD will kick their ass, though this is unlikely since they have a poor software support and OpenCL isn't as efficient as CUDA at the moment.

from chainer-fast-neuralstyle.

Are the dots static from image to image?

I'm producing a model each 2000 images used to train, this gives an animation with generate.py. Results : the dot are static, but theirs intensities change (not always decreasing). Sometimes I can see big ugly glob appearing and disappearing the next 2000 iterations.

I wonder how the images used to train affect this evolution, for example if there's some bad images that produces the globs...

Maybe I should upload a GIF

NVIDIA turned out to be greedy bastards

I used to think like that. I don't have a lot of information about nvidia but it seems like a good company, providing good cheap hardware. I can understand that FP16 is useless for the people targeted by 10xx GPU. If you really need FP16 you'll pay the price so that nvidia engineers develop the kind of features you'll need in a few years. Or you can wait for the feature to be included in the next generation gamer GPU :) .

from chainer-fast-neuralstyle.

@Heartsie The result was so good. I have trained this style before, but the output was terrible, I used the default params and the image size was 512. What params used in your training?

The dots in your result was not obvious, one of the way to improve it is increase the style image size, but the style image must be changed at the same time. and the blobs can be moved by selected the model which's blobs in the corner.

from chainer-fast-neuralstyle.

@Heartsie what parameter set did you use ?

from chainer-fast-neuralstyle.

@logic1988 what do you mean by "and the blobs can be moved by selected the model which's blobs in the corner."?

from chainer-fast-neuralstyle.

@Heartsie hi, you write "512 image size, 512 style size" means set parameter image_size to 512 and style_size to 512, is that right? i can find the image_size in train.py but can not find any param like style size. does "512 style size" means use the style image file with size 512?

from chainer-fast-neuralstyle.

Image size is style size. Just set image_size to 512 and the training images will be automatically adjusted.

from chainer-fast-neuralstyle.

@6o6o got it, thanks

from chainer-fast-neuralstyle.

@Heartsie did I get it correct that you left other parameters untouched ? like lambda_style, lambda_feat and so on...

from chainer-fast-neuralstyle.

untouched! Although I do want to experiment with them.

On Thu, Aug 11, 2016 at 2:25 PM, Vahe Hakobyan [email protected]
wrote:

@Heartsie https://github.com/Heartsie did I get it correct that you
left other parameters untouched ? like lambda_style, lambda_feat and so
on...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#33 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJqpIjGRRVMinRoIxMUZoORyMH-8T_ZZks5qe2kJgaJpZM4JgEw9
.

-=Robin Graham
Postrgram.com

from chainer-fast-neuralstyle.

I just looked at the coco dataset images, and most have a dimension of 512 by smaller than 512. Do you think that could be causing issues with the noise? Or are smaller images than what is being trained OK?

from chainer-fast-neuralstyle.

Could be... Is there any code in there to scale up everything to at least 512/512? That might be a good idea

from chainer-fast-neuralstyle.

I've already implemented scaling up if image size is less than specified value.
https://github.com/yusuketomoto/chainer-fast-neuralstyle/blob/master/train.py#L12-L19

I think that dots are likely caused by eigher of:

• initialization weights of network layers
• total variation
• optimizer(?)

from chainer-fast-neuralstyle.

Do you think that could be causing issues with the noise?

I've deleted images that were too small and I have noise.

initialization weights of network layers

Seems plausible from the images I get of the evolution of the trained model.

from chainer-fast-neuralstyle.

@yusuketomoto do you think any of these techniques would result in any benefit?

• using deeper networks for feature extraction, like VGG19 or GoogLeNet
• using higher / more layers when training (conv3_3, conv4_3) to pick up more complex features
• using LBFGS if that's possible

Thanks

from chainer-fast-neuralstyle.

@6o6o I think neither deeper networks nor different layers result in any benefit against the dots problem. LBFGS might improve (I wrote "optimizer(?)" above means using LBFGS).
Chainer LBFGS implementation can be found here. However I would like to try it, I can't bacause I'm currently on a long business trip.

from chainer-fast-neuralstyle.

@Heartsie Save the model per 10k images and choose the model which blobs was in the corner, extend the input image, generating image by model, and then crop it. Always the blobs was in the corner, if not, this method does not work.

from chainer-fast-neuralstyle.

did you guys tried the parameter "--lambda_tv"? does it have the same meaning with "tv_weight" in jcjohnson's neural-style? since i tried tv_weight=0.000085 in neural-style method, and it works fine there.

from chainer-fast-neuralstyle.

@yusuketomoto thanks for the insight. I am interested in improving quality in general, not just noise removal. In particular, extracting bigger, more complex features. I'll post results of my experiments with different datasets, if they prove to be any useful.

from chainer-fast-neuralstyle.

@6o6o and you considered the option to do the image with original colors??

from chainer-fast-neuralstyle.

@nandexsp I gave it a shot. See here @c273247 . I do not consider this feature incredibly useful, but it may have its uses.

from chainer-fast-neuralstyle.

I think that this is useful if you want to preserve the photo, but models spends a lot of time :S

from chainer-fast-neuralstyle.

I train the model takes - 10 hours. 2 epochs, 80000 images in data ms coco.
512 px - size style
gtx 970 Video Card - 4GB
can such a small time due to - SSD?
Training 5 images takes 1 second.

from chainer-fast-neuralstyle.

@rogozin70 No. Training with 512 image_size takes about 0.8 sec per image on GTX 970. SSD would increase file access time, but it certainly won't make it 4 times faster. Check your settings, it is likely you are training on 256 image size.

from chainer-fast-neuralstyle.

Thank you, you're absolutely right.

from chainer-fast-neuralstyle.

@6o6o gtx 1070 - 512 image_size - 0.4 sec? 22 hours training time?

from chainer-fast-neuralstyle.

Probably. I don't own that card, so cannot confirm.

from chainer-fast-neuralstyle.

@6o6o Can I use a code and models for online project? All filters are available online - and everyone can render much as he wants? It will be convenient. Everyone can put a filter on the universal use. Perhaps with the help of the site, I will be able to attract people to create models for filters.

from chainer-fast-neuralstyle.

Yeah, sure thing. Keep in mind tho, there's abundance of these apps already. To get any better you need to make something different, be unique in some way.

from chainer-fast-neuralstyle.

I started the server.
http://photofuneditor.com/prisma-dreamscope-filters
Add your filters-models to the forum and I'll include them in the online processing.

from chainer-fast-neuralstyle.

I have a 1080, I will begin training models of various detail as soon as I compartmentalize my Ubuntu installation from everything else (and it plays nice with my motherboard, things are so difficult now).

Any recommendations? What do people wish they could try but can't for hardware reasons? I'll try it.

from chainer-fast-neuralstyle.

style image:

output:

model:
https://github.com/markz-nyc/chainer_neural_style_models

Took me almost a day to train this model, hope it can be useful, will post more model in the future:

@6o6o I used your forked repo to train the model, thanks for the improvement

from chainer-fast-neuralstyle.

Hi @markz-nyc , are you training with size 512 and epoch 2? looks like wonderfull :)

from chainer-fast-neuralstyle.

@nandexsp Thanks, I use this repo to train the model:
https://github.com/6o6o/chainer-fast-neuralstyle

I remembered I use the default setting, and with half the COCO dataset.

from chainer-fast-neuralstyle.

These results are pretty cool, but still very noisy. Does anyone have a better handle on removing the noise? I guess we would have to do a update to the code huh? It doesn't seem possible by just changing the training parameters.

from chainer-fast-neuralstyle.

Hi @Heartsie you need to train with -epoch 5 or 6 and gets better results.

from chainer-fast-neuralstyle.

Ugh! Haha training with a size of 512 and a epoch of 6 would take like 3 days. Even then I have never seen a result from this code without the dots. There has to be another way, the dots are so even that you would think there would be a easy fix in the code to help smooth everything out.

Sent from my iPhone

On Sep 16, 2016, at 7:41 PM, Nandexsp [email protected] wrote:

Hi @Heartsie you need to train with -epoch 5 or 6 and gets better results.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

from chainer-fast-neuralstyle.

I was using size 720 with epoch 4 and I remembered that the results were great! People play with Lambda options and gets beautiful results, but I don't know how you could use these options.

from chainer-fast-neuralstyle.

How long will that take? That has to take forever.

On Fri, Sep 16, 2016 at 8:12 PM, Nandexsp [email protected] wrote:

I was using size 720 with epoch 4 and I remembered that the results were
great! People play with Lambda options and gets beautiful results, but I
don't know how you could use these options.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#33 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJqpIp6Fri6c-OciaZFD_ilQCjk1K56Mks5qqzBugaJpZM4JgEw9
.

-=Robin Graham
Postrgram.com

from chainer-fast-neuralstyle.

I remembered that spends a lot of time, 4-5 days. But you could change lambda options. I think that people use this options and the results are better.

from chainer-fast-neuralstyle.

I tried 9 different models of the same style image once. Each time changing
the lambda options just a little bit. The noise doesn't really change at
all unfortunately. I think it is a coding issue.

On Fri, Sep 16, 2016 at 8:38 PM, Nandexsp [email protected] wrote:

I remembered that spends a lot of time, 4-5 days. But you could change
lambda options. I think that people use this options and the results are
better.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#33 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJqpIi7h-tA3lpJUCi2fIldu8JayzTRLks5qqzaEgaJpZM4JgEw9
.

-=Robin Graham
Postrgram.com

from chainer-fast-neuralstyle.

@6o6o Nice results! I have been unable to test this theory yet because of some plugin installation issues, but I suspect that a descreen filter would work really well for filtering out the dots as well.

from chainer-fast-neuralstyle.

Yes, I'm sure there are better and more efficient ways to achieve this. It all depends on level of initial sharpness you'd like to keep.

Also, please note, with the latest patch 528f8fb this will probably no longer be required, but for old models this trick may still come in handy.

from chainer-fast-neuralstyle.

I'm using a slight bilateral blur to remove the dots. I've created two 4K videos #70

from chainer-fast-neuralstyle.

@ttoinou bilateral blur is way too destructive for subtle details and slow as well.
@scribblemaniac thank you for pointing me in the right direction. The term descreen is what I should have been looking for in the first place. This FFT-based plugin does an impressive job.

Turns out Fourier Transform is super-awesome at this sort of task of detecting periodically repeating patterns. It removes any pattern more efficiently than all of my previous methods put together, while at the same time retaining more sharpness of the image. The images are just crazily sharp! Shouldn't be hard to implement in python too. Not sure about the speed, but whatever it is, it's well worth it.

from chainer-fast-neuralstyle.

Wow, that's a thorough explanation. Thanks for highlighting the paper.

from chainer-fast-neuralstyle.

http://distill.pub/2016/deconv-checkerboard/ this blog does some research about the Checkerboard Artifacts

from chainer-fast-neuralstyle.

I'm not sure there's a resize method in chainer. Perhaps the regular unpooling would work just fine?

@scribblemaniac, that is some extensive testing, nice job. As people have pointed out, the problem is being studied, so probably it would be more reasonable to exclude the source of the problem, than to fight the consequences, but just for the sake of completeness I decided to recreate the above FFT method in python. Took me some time to figure our the difference between regular Fourier transform and the GIMP plugin. Turns out it is using a custom normalization method along with dynamic boosting, where pixels further away from the center are being assigned more energy, making them brighter. This allows us to use regular binary threshold to localize high frequency areas and mask them automatically, without touching the middle, which is generally desirable in this particular case. As for the mask you provided, I assume you created it manually?

This is the piece of code I came up with. You may try it out if it's still relevant. It comes down to determining minimum acceptable threshold per style without degrading the image. I find levels 80-96 to work well. At right settings it gets rid of all sorts of patterns, including checkerboard. I may add it to the repo after some additional refinement and optimization.

Edit: per-channel masking appears to give better results in some problem areas

import cv2
import numpy as np

def normalize(img):
    h, w = img.shape
    x = np.floor(img % w)
    y = np.floor(img / w)
    cx = np.abs(x - w/2);
    cy = np.abs(y - h/2);
    energy = (cx ** 0.5 + cy ** 0.5);
    return np.maximum(energy*energy, 0.01)

def ellipse(w, h):
    offset = (w+h)/2./(w*h)
    y, x = np.ogrid[-h: h+1., -w: w+1.]
    return np.uint8((x/w)**2 + (y/h)**2 - offset <= 1)

img = np.float32(cv2.imread('example.png').transpose(2, 0, 1))
rows, cols = img.shape[-2:]
coefs = normalize(np.arange(rows * cols).reshape(rows, cols))
mask = np.empty_like(img)
fft = np.empty((3, rows, cols, 2))

for i in range(3):
    fft[i] = cv2.dft(img[i],flags = 18)
    fft[i] = np.fft.fftshift(fft[i])
    spectrum = 20*np.log(cv2.magnitude(fft[i,:,:,0],fft[i,:,:,1]) * coefs)

    # the following steps are equivalent in GIMP to:
    # threshold, select white, subtract middle, grow selection by 6, feather by 6, fill with white
    ret, thresh = cv2.threshold(np.float32(np.maximum(0, spectrum)), 92, 255, cv2.THRESH_BINARY)
    ew, eh = cols/8, rows/8
    pw, ph = (cols-ew*2)/2, (rows-eh*2)/2
    middle = np.pad(ellipse(ew, eh), ((ph,rows-ph-eh*2-1), (pw,cols-pw-ew*2-1)), 'constant')
    thresh *= 1-middle
    thresh = cv2.dilate(thresh, ellipse(6,6))
    thresh = cv2.GaussianBlur(thresh, (0,0), 2, 0, 0, cv2.BORDER_REPLICATE)
    mask[i] = thresh
    thresh = 1 - thresh / 255

    img_back = fft[i] * np.repeat(thresh[...,None], 2, axis = 2)
    img_back = np.fft.ifftshift(img_back)
    img_back = cv2.idft(img_back)
    img[i] = cv2.magnitude(img_back[:,:,0],img_back[:,:,1])

cv2.imwrite('example_mask.png', mask.transpose(1, 2, 0))
cv2.imwrite('example_filtered.png', img.transpose(1, 2, 0))

And a few magnified samples after final filter with waifu2x.

from chainer-fast-neuralstyle.

@6o6o Nice script, it works well! Yes I did indeed make the last mask manually, but I had considered implementing something similar to this. Obviously addressing the source of the issues would be the ideal solution, and I hope that the proposal in that paper helps fix it. This is for until it gets fixed, or if the proposed solution is not sufficient. I personally think the developers should focus on fixing the other type of noise that can't be fixed as easily with post-processing first.

from chainer-fast-neuralstyle.

Awesome! I want to give this a shot!

from chainer-fast-neuralstyle.