Comments (7)
thanks for info, let me know if you find it better than SSIM (i'm not planning to investigate such details soon).
good point about blurring as well; it should probably be done before applying that structure loss. i believe that CLIP comparison must be done with original image to keep all important features intact.
from aphantasia.
When I set it to recreate an image using just SSIM, no encoding whatsoever, it tends to look really smoothed out but the edges are there. However, when I tried it with DISTS it let all these neat shapes remain.
Also, it turns out earlier it wasn't even using clip_fft.py in the notebook, I had to put it in the Generate code cell and figure out why it kept complaining about all the tensors not being on the same device, only to realize that the loss apparently wasn't.
To actually get it working, had to add from DISTS_pytorch import DISTS
and change ssim_loss = ssim.SSIM(window_size = 11)
to ssim_loss = DISTS().cuda()
and change ssim_loss(img_out, img_in)
to ssim_loss(img_out, img_in, require_grad=True, batch_average=True)
, though it started off giving higher values than expected, like 0.55 or so, so I decided to square the output of it and also changed how it's used to calculate loss for testing, with
ssim_loss_output = ssim_loss(img_out, img_in, require_grad=True, batch_average=True)**2.0
loss = loss * (1.0 - sync) + ssim_loss_output * sync
which, at least for this, works well.
It doesn't overly match the image, but the overall general shapes are familiar.
with a sync of 0.5 that doesn't change as steps increase.
And I've been experimenting with a touch of weight decay in Adam in an attempt to break up big areas of nothing. Doesn't seem to help with that, doesn't seem to hurt, but it does make it wiggle and jiggle a bit in the final video which is really dreamlike, especially when the loss is turned up.
Aphantasia_DISTS_Test.zip
Here's my Colab notebook with it all set up. Had to manually !wget the weights for DISTS and move it, somehow the install didn't handle that on its own.
from aphantasia.
i tested DISTS and didn't find it convincing. besides quite fuzzy output, it tends to eat GPU RAM way more than SSIM.
what i used instead, was your idea about blurring: simple 8x downsampling before SSIM removed sharp edges and delivered exactly that level of details which i've been seeking (see below). so thanks for putting me up to this, and best of luck with further investigations.
text input = "winter", 200 steps, 240 samples, image input:
from aphantasia.
@eps696 Not sure if you noticed, but this SSIM change (8x downsample) introduced a grid of tiny white or black pixels across the whole image. I was going to share my own example, but if you look closely at the 2 images you just posted, you should be able to see it.
from aphantasia.
sure noticed, they go with time (200-300 iters was enough for me)
from aphantasia.
Odd, the grid doesn't seem to ever fade away in my experience. I've done several 1000+ iteration generations this afternoon, and it seems like it's a permanent effect.
from aphantasia.
well, few dots stay permanent indeed (like on the pics above), but their grid disappeared pretty quickly on all my tests. so i neglected this in favour of kinda proper reconstruction (any other trick didn't make it).
[i tried to fix it with additional blurring, but then details and edges of original image started appearing again, so i kept it like this]
from aphantasia.
Related Issues (20)
- Pytorch Import torch.irfft Update to torch.fft.irfft HOT 1
- Specify GPU HOT 2
- DeepSpeed integration for training on local cheaper GPUs. HOT 3
- Invalid Syntax when trying to run the first time HOT 10
- Error when running "Generate" HOT 2
- Illustrip3D - problems with video output quality HOT 3
- Can't generate video HOT 14
- init_image support HOT 4
- Colab notebook "How to just use Aphantasia" HOT 3
- clip_fft.py won't start HOT 4
- Error HOT 2
- IndexError in IllusTrip3D HOT 1
- TypeError: 'float' object is not subscriptable HOT 3
- integrate with Lightning ecosystem CI HOT 1
- Something changed since 04/19/2022 HOT 1
- Incorporating the -notext option into IllusTrip3D.ipynb HOT 2
- NameError: name 'aug_transform' is not defined HOT 1
- AssertionError: Torch not compiled with CUDA enabled HOT 1
- torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB (GPU 0; 2.00 GiB total capacity; 1.58 GiB already allocated; 0 bytes free; 1.74 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF HOT 1
- getting this error while running illustrip3d : `!ssize.empty() in function 'resize'` HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aphantasia.