StreamWrapper - handling large file uploads about gaufrette HOT 10 OPEN

@djoos As it seems you managed to make it work in your "extension" https://github.com/escapestudios/EscapeGaufretteExtension, is it possible for you to make a PR based on this?

from gaufrette.

I've taken the liberty to integrate the temp file buffer functionality from EscapeGaufretteExtension into my own fork (but don't use this - see next comment): https://github.com/Uplink03/Gaufrette/tree/enhance/escape-gaufrette

I didn't take the OpenCloud changes.

I didn't put too much thought into it, so now the InMemoryBuffer isn't used and there's no option to select it. But maybe it helps somebody. If you review the changes and find them good for a PR, I'll make a PR.

from gaufrette.

Top tip when dealing with large file uploads to OpenCloud: make sure to define RAXSDK_TIMEOUT with a higher-than-default (default = 30) value.

eg.

define('RAXSDK_TIMEOUT', 1800);

Hope this helps!

Kind regards,
David

from gaufrette.

I am having the same issue with handling large file uploads. All adapters that do support resource write() call Util/Size which uses mb_strlen to determine content size. This should be avoided.

from gaufrette.

@djoos Is this still relevant?

from gaufrette.

Hi @wysow,

yes, the issue is still relevant: AFAIK the file content is still kept in memory in the bundle when not using our tempfile workaround (see above)...
This is an issue when handling (very) large file uploads...

Hope this helps!

Kind regards,
David

from gaufrette.

Definitely, I'll keep you updated!

from gaufrette.

Is there working build available now?

from gaufrette.

Hi @SrgSteak,

I haven't got round to starting the work of getting our "extension" into Gaufrette...

However, I haven't forgotten this thread!

Kind regards,
David

from gaufrette.

UPDATE: DON'T USE THIS CODE. It's introduces an incompatible API change. Just check out what happens in TempFileBuffer's flush method: the content is the temp file's filename. Without reviewing the OpenCloud changes, I'm gonna say that they address this behaviour, but if you go and use, say, S3, you get the temp file's filename instead of the content.

While researching this I noticed that I can pass fopen handles to filesystem->write, so I can replace all file_get_contents with fopen and get rid of the memory issues. Both AwsS3 and Local adapters accept this usage, and I would suggest it gets documented and expected across the board from all other adapters.

This also means I don't need to use streams at all.

A few days ago I said:

I've taken the liberty to integrate the temp file buffer functionality from EscapeGaufretteExtension into my own fork: https://github.com/Uplink03/Gaufrette/tree/enhance/escape-gaufrette

I didn't take the OpenCloud changes.

I didn't put too much thought into it, so now the InMemoryBuffer isn't used and there's no option to select it. But maybe it helps somebody. If you review the changes and find them good for a PR, I'll make a PR.

from gaufrette.