0xharu / nerd-storage Goto Github PK

Writing this as an open-ended discussion and will be submitting the same information to the respective packages involved in this issue. Please note that this issue has been raised several times before, but has yet to be fixed / resolved. (Flask #2086, Werkzeug #1041, Werkzeug #510, Codalab-Worksheet #301, see also: StackOverflow Question)

Currently, the default_stream_factory that is embedded in Werkzeug (a Flask Dependency) splits files and performs one of two operations:

• If under 500kb, store the uploaded file in-memory.
• If over 500kb, store the file in a temp file in /tmp.

This creates one main visible problem:

• Machines with a smaller /tmp may send a traceback to the user with a "No Space on Device" OSError, not allowing uploads to complete.

And one hidden problem:

• Large File Uploads (>500kb as defined in default_stream_factory) still eat into the RAM of the host machine, despite being sent to /tmp via TemporaryFile or SpooledTemporaryFile.

This redundancy can lead to a severe issue.
I was testing NS on a 4GB machine and performing a 4GB upload, which crashed the machine, and led me do two things:

• Up my /tmp size to 10GB in /etc/fstab
• Investigate the Memory Leak

I've not found a fix for this or any surefire way of counteracting the problem. Despite ensuring that /tmp has more than enough storage, RAM is still consumed. If the upload eats the RAM to 100% a system crash will occur, throwing the server offline. Somewhere in the reading I've been doing it's been suggested that Werkzeug stores the files into memory regardless of size (rendering formparser useless?) and has sloppy management with the data.

Any possible fixes or monkey patches would be appreciated; collaboration and discussion is welcome!