Weird Size Issue - 600KB video file results in 18MB upload traffic. 8 MB file results in 228MB outgoing traffic about google-api-php-client HOT 7 CLOSED

google-api-php-client-1.0.0-alpha does not have this problem. Upload Traffic Size is almost equal to file size.

So some issue cropped up after the gzip stuff was added

from google-api-php-client.

Very odd, I can see the same thing when uploading to drive. Disabling the gzip seems to bring it back to normal, which is rather confusing. I'll poke around a little more, but we may just need to automatically disable gzip for file upload, which is straightforward.

Workaround is to disable it manually using: $client->setClassConfig("Google_Http_Request", "disable_gzip", true);

from google-api-php-client.

So what I can see is compress.zlib doing an absolute tonne of writes on the upload, which is not very helpful. Unfortunately, I can't find a wonderful work around that lets us deal with the stream filter in a decent way, so I think disabling on uploads will be the best bet.

from google-api-php-client.

Patched in #65

from google-api-php-client.

I'm seeing something very like this with ownCloud's Google Drive support (which I'm still working to get ported to 1.x). It's using extremely simple upload code, ripped straight from the examples:

public function writeBack($tmpFile) {
    if (isset(self::$tempFiles[$tmpFile])) {
        $path = self::$tempFiles[$tmpFile];
        $parentFolder = $this->getDriveFile(dirname($path));
        if ($parentFolder) {
            // TODO Research resumable upload
            $mimetype = \OC_Helper::getMimeType($tmpFile);
            $data = file_get_contents($tmpFile);
            $params = array(
                'data' => $data,
                'mimeType' => $mimetype,
                'uploadType' => 'media'
            );
            $result = false;
            if ($this->file_exists($path)) {
                $file = $this->getDriveFile($path);
                $result = $this->service->files->update($file->getId(), $file, $params);
            } else {
                $file = new \Google_Service_Drive_DriveFile();
                $file->setTitle(basename($path));
                $file->setMimeType($mimetype);
                $parent = new \Google_Service_Drive_ParentReference();
                $parent->setId($parentFolder->getId());
                $file->setParents(array($parent));
                $result = $this->service->files->insert($file, $params);
            }
            if ($result) {
                $this->setDriveFile($path, $result);
            }
        }
        unlink($tmpFile);
    }
}

and I'm currently testing with version 1.0.6 of the library. But I seem to be seeing just the same issue as the OP here - uploading a simple ~4MB file results in over 100MB of traffic. Any ideas?

from google-api-php-client.

Ah. I think the fix for this may have only fixed it for use of the Google_Http_MediaFileUpload class, not the simple files->insert method. Let me see if my theory holds up.

from google-api-php-client.

Hum. Actually, I think I was using a packaged copy of 1.0.3 I had lying around and somehow with 1.0.6 the problem doesn't happen any more, though I can't clearly see why it would affect 1.0.3 but not 1.0.6. But hey, so long as it works...hey ho.

Aha! So I think I'm not hitting it in 1.0.6 because it's using Curl instead of Stream and bypassing the whole thing. If I force use of Stream instead of Curl, I can cause it to happen again.

In MediaFileUpload.php , the bit where it conditionally disables gzip for uploads is only in nextChunk(), which is only used for resumable uploads. I've tried copying it into process() (with the obvious change to $this->request instead of $httpRequest), and the code gets hit (I put a debug log in to check), but it doesn't seem to do the job - the amount of data transferred is still huge. But if I disable gzip with the big hammer - $client->setClassConfig("Google_Http_Request", "disable_gzip", true); - that does solve the problem. Not quite sure why just trying to disable it in process() doesn't work.

from google-api-php-client.