Comments (8)
I found the leak, it is in DainDataset.py line 49.
my_combo["original"] = numpy.array(PIL.Image.open(self.combos[index]["f1"]).convert(self.frameFormat))
The dict where this image is added is still referenced by member variable self.combos, so you effectively store the entire raw dataset in RAM when iterating over the DataLoader. Just deleting this line should fix the leak. I didn't see combo["original"] being used anywhere. Another possible fix would be changing line 48 from
my_combo = self.combos[index]
to
my_combo = self.combos[index].copy()
from dain-app.
The pytorch model use Vram, of course some parts of the core require some ram.
This is a leak in some part of the dataset generation, and this message is from the pytorch part of the code.
https://github.com/pytorch/pytorch/search?q=buy+new+ram
I'm still looking for a way to fix this leak.
from dain-app.
I found the leak, it is in DainDataset.py line 49.
my_combo["original"] = numpy.array(PIL.Image.open(self.combos[index]["f1"]).convert(self.frameFormat))
The dict where this image is added is still referenced by member variable self.combos, so you effectively store the entire raw dataset in RAM when iterating over the DataLoader. Just deleting this line should fix the leak. I didn't see combo["original"] being used anywhere. Another possible fix would be changing line 48 from
my_combo = self.combos[index]
to
my_combo = self.combos[index].copy()
Can you please upload the modified program as a package? Because there are many people, including me, who are not familiar with Python. Thank you very much.
from dain-app.
Confirmed. Getting this leak too.
from dain-app.
I was able to keep it running, despite this huge leak. Here`s how.
- Disable Windows memory compression. This service tries to compress frames in memory, but instead of helping it is just ballooning inside your RAM, thus leaving no more free space for allocation.
- Manually set your virtual memory paging file size to something huge. I had to use 100GB paging file to interpolate 3700 4k frames.
- Optionally you can use app like Process Lasso to periodically manually flush DAIN RAM artifacts to virtual memory.
from dain-app.
Pytorch return error string "buy new ram? " lmao
from dain-app.
@lmichalke File "Dain-App/my_DAIN_class.py", line 1102:
cleanFiles.append( {"data" : combo['original'][tt].numpy(), 'timestamp' : start})
Deleting line 49 in DainDataset.py will cause an error.
from dain-app.
Try the other thing I suggested.
Btw I recommend using this tool instead, it is much more polished.
from dain-app.
Related Issues (20)
- Output video frame rate problem HOT 1
- [suggest] Is possible copy audio ? HOT 2
- Ultra HFR support (real time 240fps, 360fps, 480fps on true 240Hz, 360Hz, 480Hz displays) HOT 5
- Empty interpolated frames folder
- 'int' object is not subscriptable HOT 1
- Blank error on attempt HOT 1
- Blank error on attempted interpolation
- DAIN output FPS
- When the memory leak problem will be fixed? HOT 1
- Too old Grafic card
- Export as mp4 h265?
- Sky extreme artifacts
- instant crash on startup
- How do I create a working version from the source code? HOT 3
- "CUDA out of memory" on RTX3090 HOT 3
- Can I use DAIN-APP on Google Colab? HOT 1
- Error while creating conda environment HOT 4
- No video output?????
- Is it expected to see duplicate objects
- Unrecognised symbol
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dain-app.