Refactor http support for file uploads about twisted HOT 11 CLOSED

#!html
<pre>
Glyph and DP tell me Quotient has code that does this, but it's hacky. Possibly
we could do this at pycon.
</pre>

#!html
<pre>
This depends on distrib refactor, which is why I had to back
this out from 0.99.7 or whenever I did it originally.
Benefits would be not storing 20MB uploaded files in memory,
and a few other things which prevent DoS attacks.

</pre>

#!html
<pre>
i keep hearing that twisted caches uploaded files in 
memory, but every time I go look at the http code, it 
seems that this is wrong. Here is the code:

        if length &lt; 100000:
            self.content = StringIO()
        else:
            self.content = tempfile.TemporaryFile()

where "length" is the Content-length that the client has 
sent. So what's the deal?

</pre>

#!html
<pre>
That's because you keep looking at the wrong code ;)
The relevant line is:

  cgi.parse_multipart(self.content, pdict)

</pre>

#!html
<pre>
glyph points out that FieldStorage blocks, which is no good.
So we're going to have to write our own solution.

</pre>

#!html
<pre>
Feel like doing some web programming?

</pre>

#!html
<pre>
To rephrase:

1. Store large uploaded files on disk.
2. Don't lose metadata like filename and mimetype which are
still available.
3. Default API for accessing files would be as file-like
objects.
3. Backwards compatability - there is code that expects
files to be strings. should work, while still not actually
storing them as strings in memory until the users actually
do this (mmap, maybe? they look kinda like strings.)

</pre>

#!html
<pre>
instead of using mmap, I'd prefer using magical __getattr__ code
which would create these attributes by reading in the files.
</pre>

#!html
<pre>
So I think this is basically waiting for someone to rewrite cgi.parse_multipart in a way that can be fed 
incremental data.
</pre>

#!html
<pre>
I did this a while back: twisted.web2.fileupload
</pre>