googlecloudplatform / appengine-gcs-client Goto Github PK

View Code? Open in Web Editor NEW

126.0 53.0 113.0 4.4 MB

App Engine-Cloud Storage custom client library

License: Apache License 2.0

Java 64.65% HTML 0.68% Python 34.66%

app-engine cloud-storage gcs gae google-cloud google-cloud-platform google-cloud-storage files blob-storage blob-store

appengine-gcs-client's People

Contributors

Stargazers

Watchers

Forkers

yyao19 eamaillard rbruyere madhu-hub rohitbarnwal asafronau yusun90 guffywave brstrat feenamaria surya-m kunskap aozarov rkchy03 jessehammons afornie c0debrain k-cmms roofcat speedplane brandony prakashayappan projecterp vkaya swapnillipare ludwick vishalsan iordacheclaudiu digideskio dw3e9 laurieaw gnithin ajkannan tmatsuo nickretallack richieescarez erlichmen appscale tmarballi tesodara vinay453 evanj gcp-deepak-sharma shyamdadimuni achandebois ppromsuban spencerlee92 duythangdecima ashamenon26 elharo samsammurphy abhayp908 jerryzhao123 manojreddy22 jhbae200 sumitshrm devlance xuesongma catherineh totangoram manuelmena rramona2 ajithkumart yanchhuong aibaars savinka wix-playground poojapathak0001 anna-poliakova shikhamaniar kenjiyamamoto michan85 sotakahashi1173 smiksha alexdg18 antoniolatela rangastartup rubenszimbres poojafuelx rajshekhar-heremath virajindasrao thainamariani kseals281 efratshachor atulanandnitt raychorn saeedseyyedi jmanojkumar27 ssitb https-www-quauntumdecaygames-de aaron-je manohar52 weiyue-z anku-ai hashiamkadhim sarahttan107 antoinepupin plux-kivra david-loveholidays kvfairchild

appengine-gcs-client's Issues

Incompatibility with appstats and dev app server

Greetings,

I'd like to report a problem that seems to originate from appstats' incompatibility with this library.

web.xml:

...
    <filter>
        <filter-name>appstats</filter-name>
        <filter-class>com.google.appengine.tools.appstats.AppstatsFilter</filter-class>
    </filter>
    <filter-mapping>
        <filter-name>appstats</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>
    <filter>
        <filter-name>appstats</filter-name>
        <filter-class>com.google.appengine.tools.appstats.AppstatsFilter</filter-class>
        <init-param>
            <param-name>calculateRpcCosts</param-name>
            <param-value>true</param-value>
        </init-param>
    </filter>
...

Servlet.java:

GcsFilename gcsFilename = new GcsFilename("bucketName", "objectName");
GcsFileOptions opts = new GcsFileOptions.Builder().mimeType("image/png").build();
GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance()).createOrReplace(gcsFilename, opts);

war/WEB-INF/lib directory contains the following jars:

appengine-api-1.0-sdk-1.9.44.jar
appengine-api-labs.jar
appengine-endpoints-deps.jar
appengine-endpoints.jar
appengine-gcs-client-0.6.jar
appengine-jsr107cache-1.9.44.jar
commons-codec-1.3.jar
commons-logging-1.1.1.jar
datanucleus-appengine-1.0.10.final.jar
datanucleus-core-1.1.5.jar
datanucleus-jpa-1.1.5.jar
geronimo-jpa_3.0_spec-1.1.1.jar
geronimo-jta_1.1_spec-1.1.1.jar
google-api-client-1.22.0.jar
google-api-client-appengine-1.22.0.jar
google-api-client-servlet-1.22.0.jar
google-api-services-storage-v1-rev68-1.21.0.jar
google-http-client-1.22.0.jar
google-http-client-appengine-1.22.0.jar
google-http-client-jackson2-1.22.0.jar
google-http-client-jdo-1.22.0.jar
google-oauth-client-1.22.0.jar
google-oauth-client-appengine-1.22.0.jar
google-oauth-client-servlet-1.22.0.jar
guava-19.0-rc1.jar
guava-jdk5-17.0.jar
httpclient-4.0.1.jar
httpcore-4.0.1.jar
jackson-core-2.1.3.jar
jdo2-api-2.3-eb.jar
joda-time-2.3.jar
jsr107cache-1.1.jar
jsr305-1.3.9.jar
servlet-api-2.5.jar
transaction-api-1.1.jar

Essentially everything that Google App Engine needs to run a bare minimum application, and appengine-gcs-client version 0.6 jars.
Run the dev app server (with Eclipse), hit the servlet, and you get:

java.io.IOException: java.lang.NoSuchMethodException: com.google.appengine.tools.appstats.Recorder.getService(java.lang.String)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.getInstance(LocalRawGcsService.java:186)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.access$000(LocalRawGcsService.java:109)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.ensureInitialized(LocalRawGcsService.java:194)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.beginObjectCreation(LocalRawGcsService.java:249)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.beginObjectCreation(LocalRawGcsService.java:92)
    at com.google.appengine.tools.cloudstorage.GcsServiceImpl$1.call(GcsServiceImpl.java:74)
    at com.google.appengine.tools.cloudstorage.GcsServiceImpl$1.call(GcsServiceImpl.java:70)
    at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:108)
    at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:166)
    at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:156)
    at com.google.appengine.tools.cloudstorage.GcsServiceImpl.createOrReplace(GcsServiceImpl.java:70)
    at appstatstest.AppstatstestServlet.doGet(AppstatstestServlet.java:27)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at com.google.appengine.tools.appstats.AppstatsFilter.doFilter(AppstatsFilter.java:143)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:128)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:122)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
    at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
    at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:95)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:508)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.NoSuchMethodException: com.google.appengine.tools.appstats.Recorder.getService(java.lang.String)
    at java.lang.Class.getDeclaredMethod(Class.java:2017)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.<init>(LocalRawGcsService.java:123)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.getInstance(LocalRawGcsService.java:184)
    ... 50 more

I believe the problem comes from appstats because commenting out the lines I gave in web.xml solves the issue, but appstats is something that I prefer not to discard.

More information, if needed:

Eclipse Luna with Google Plugin 4.4
JDK 1.7.0_80

Add support for compose

Just creating an issue for this as this was fixed in
db38d680899f49dfe23b9b00ca0dd5a4bf1d7b0

Inconsistent handling of unicode for open / listbucket / delete

Hi,

The cloudstorage.listbucket(..) gives you GCSFileStat objects, which will decode UTF-8 encoded object names for you so that GCSFileStat.filename is a unicode instance. This is nice.

But passing a unicode instance to the open or delete functions gives you a KeyError if the string includes non-ASCII characters.

Traceback (most recent call last):
  File "/base/data/home/apps/e~davidwtbuxton-test/cloudstorage-utf8-bug.394332959138233059/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/base/data/home/apps/e~davidwtbuxton-test/cloudstorage-utf8-bug.394332959138233059/bottle.py", line 1732, in wrapper
    rv = callback(*a, **ka)
  File "/base/data/home/apps/e~davidwtbuxton-test/cloudstorage-utf8-bug.394332959138233059/wsgi.py", line 32, in create_utf8
    return create_file(u'Señor') #.encode('utf-8'))
  File "/base/data/home/apps/e~davidwtbuxton-test/cloudstorage-utf8-bug.394332959138233059/wsgi.py", line 38, in create_file
    with cloudstorage.open(dest, 'w') as fh:
  File "/base/data/home/apps/e~davidwtbuxton-test/cloudstorage-utf8-bug.394332959138233059/cloudstorage/cloudstorage_api.py", line 91, in open
    filename = api_utils._quote_filename(filename)
  File "/base/data/home/apps/e~davidwtbuxton-test/cloudstorage-utf8-bug.394332959138233059/cloudstorage/api_utils.py", line 94, in _quote_filename
    return urllib.quote(filename)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib.py", line 1263, in quote
    return ''.join(map(quoter, s))
KeyError: u'\xf1'

It would be nice if the cloudstorage library automatically encoded unicode object names to UTF-8, as well as decoding them.

For example, in this test project which creates objects with UTF-8 encoded names, the filename has to be encoded again when deleting all objects in a bucket.

Thank you,

David B.

Cannot use GcsService, because of old guava dependency.

I ran to this problem, see the error:
java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.createUnstarted()Lcom/google/common/base/Stopwatch;
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:156)
at com.google.appengine.tools.cloudstorage.GcsServiceImpl.createOrReplace(GcsServiceImpl.java:70)

I was searching a lot what is the problem and I think that it's guava dependency problem.
See this issue: googleapis/google-api-java-client#903

IMHO appengine-gcs-client uses the google-api-java-client, which used guava v13.
The guava v13 doesn't contain the Stopwatch.createUnstarted() method, it's available from v15.

As written in the issue 903 (link is above), author updated google-api-java-client to use the guava v15.

It's a big problem, because I cannot use GcsService anymore.
Thanks.
Filip.

Unable to fetch transient dependency com.google.http-client:google-http-client:[1.19.0,2.0)

My app-engine project depends on this library (using gradle):
compile com.google.appengine.tools:appengine-gcs-client:0.6

But, when i try to compile project with the command 'gradlew clean compileJava' i get an error:

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve all dependencies for configuration ':compile'.
> Could not find any version that matches com.google.http-client:google-http-client:[1.19.0,2.0).
  Versions that do not match:
      1.5.3-beta
      1.5.0-beta
  Searched in the following locations:
      https://repo1.maven.org/maven2/com/google/http-client/google-http-client/maven-metadata.xml
  Required by:
      :my-project:unspecified > com.google.appengine.tools:appengine-gcs-client:0.6

That's happens because file in path https://repo1.maven.org/maven2/com/google/http-client/google-http-client/maven-metadata.xml really outdated and contains information about only two versions 1.5.0-beta and 1.5.3-beta.

Pin specific versions in pom.xml

That is no version ranges. As I type this, customer builds and deploys are broken by a google-http-client push. This wouldn't have happened with specific pinned versions. See b/112072623

Set up continuous integration

Use real cloud storage on dev server

When I run the development server gcs-client stores the blobs on the local filesystem. How can I force it to use the actual cloud storage even on the development server?

Link to "the site" in the project README

This is exacerbating the issue of fragmented documentation that the Google Cloud Platform is already suffering from.

User-Agent doesn't include the version.

For Java at least, this should get pulled out of the .pom file via generated resources, as in How do I filter resource files. Not sure about Python.

Please follow best-practices for versioning/releases

The history of this repository indicates that developers are working from and pushing directly to the master branch. The only indication of the Python package version is contained within setup.py and was last updated in December 2014 to 1.9.15.0. There have been many commits since there without also bumping the version, which can lead to wildly inconsistent behavior when following the instructions to install the package directly from GitHub.

I highly recommend the use of Git Flow, which will enable proper versioning/releasing through the use of branches, tags, and releases. I also recommend semantic versioning as a best-practice for versioning software packages.

This library is documented as being the supported Python client when working with Google Cloud Storage, so following these best-practices will make a significant difference for developers like myself who depend on it. Thank you!

GcsServiceOptions.java missing copyright

appengine-gcs-client/java/src/main/java/com/google/appengine/tools/cloudstorage/GcsServiceOptions.java is missing a copyright statement

Does Not Work On Python Dev Server

Trying to use the python client library with the development server. Each time I try to write to the bucket I get this error:

It works fine when deployed on app engine

ServerError: Expect status [201] from Google Storage. But got status 500.
Path: '/*****mybucket******/7jj601gxnc20160925124634jpgdl0.jpg'.
Request headers: {'content-type': 'image/jpeg', 'accept-encoding': 'gzip, *', 'x-goog-api-version': '2', 'x-goog-resumable': 'start'}.
Response headers: {'server': 'Development/2.0', 'date': 'Sat, 26 Nov 2016 17:48:14 GMT', 'transfer-encoding': 'chunked'}.
Body: ''.
Extra info: None.

my python code, pretty much taken directly from https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/read-write-to-cloud-storage

abbreviated version

 bucket_name = os.environ.get('BUCKET_NAME',
                               app_identity.get_default_gcs_bucket_name())
   file_name = slugify(request.args.get('pic'), separator = '')[24:] + '.jpg'
    bucket_file = '/{}/{}'.format(bucket_name, file_name) #generally turns into something like /app_default_bucket/7jj6012fnc20250925124634jpgdl0.jpg' on the development server

write_retry_params = gcs.RetryParams(backoff_factor = 1.1)
gcs_file = gcs.open(bucket_file, 'w', 'image/jpeg', retry_params = write_retry_params)
out_string.seek(0) # StringIO object
gcs_file.write(out_string.getvalue())
gcs_file.close()

As I mentioned works perfectly when deployed.

Something else I discovered I can get it to give me a 404 not found error if I just try to open random files.

Shade 3rd party libs, especially httpcomponents

There are several versions of httpcomponents which are not compatible with each other. Please shade the version used in this library so that it does not conflict with other libs.

Incompatibility with Java Dev App Server

The problem seems to be in calling DevAppServer2Delegate.getService(String)

pom.xml:

    <dependency>
      <groupId>com.google.appengine.tools</groupId>
      <artifactId>appengine-gcs-client</artifactId>
      <version>0.6</version>
    </dependency>
...

      <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>appengine-maven-plugin</artifactId>
        <version>0.1.2</version>
      </plugin>

Servlet.java:

GcsFilename gcsObj = new GcsFilename("bucket", "object.txt");
    GcsService createGcsService = GcsServiceFactory.createGcsService();
    GcsOutputChannel createOrReplace = createGcsService.createOrReplace(gcsObj,
            GcsFileOptions.getDefaultInstance());

Run Dev App Server:

mvn appengine:run

Hit servlet.

Result:

java.io.IOException: java.lang.NoSuchMethodException: com.google.appengine.tools.development.devappserver2.DevAppServer2Delegate.getService(java.lang.String)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.getInstance(LocalRawGcsService.java:186)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.access$000(LocalRawGcsService.java:109)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.ensureInitialized(LocalRawGcsService.java:194)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.beginObjectCreation(LocalRawGcsService.java:249)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.beginObjectCreation(LocalRawGcsService.java:92)
    at com.google.appengine.tools.cloudstorage.GcsServiceImpl$1.call(GcsServiceImpl.java:74)
    at com.google.appengine.tools.cloudstorage.GcsServiceImpl$1.call(GcsServiceImpl.java:70)
    at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:108)
    at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:166)
    at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:156)
    at com.google.appengine.tools.cloudstorage.GcsServiceImpl.createOrReplace(GcsServiceImpl.java:70)
    at com.google.meltsufin.saltshaker.MainServlet.doGet(MainServlet.java:44)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.devappserver2.RequestIdFilter.doFilter(RequestIdFilter.java:36)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:128)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:50)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:98)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:511)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.NoSuchMethodException: com.google.appengine.tools.development.devappserver2.DevAppServer2Delegate.getService(java.lang.String)
    at java.lang.Class.getDeclaredMethod(Class.java:2132)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.<init>(LocalRawGcsService.java:123)
    at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.getInstance(LocalRawGcsService.java:184)
    ... 44 more

GcsService.update overwrites the mime-type

Attempting to update a file's options without explicitly setting the MIME type overrides the MIME type with the default "application/octet-stream".

The expected behavior is for the MIME type to remain unchanged unless explicitly set.

Example-

// Existing file with "application/javascript" MIME type
final GcsFilename gcsFilename = new GcsFilename("BUCKET", "OBJECT");

final GcsService gcs = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());

// Note MIME type is not explicitly set
final GcsFileOptions gcsFileOptions = new GcsFileOptions.Builder()
   .acl("public-read")
   .build();

gcs.update(gcsFilename, gcsFileOptions);

// File is publicly accessible, but HTTP GET returns "Content-Type: application/octet-stream"

A workaround is to first retrieve the file's MIME type via GcsService.getObject, and then set it explicitly to the same value in GcsService.update. However this results in another round trip, and in theory introduces concurrency issues.

Cannot import cloudstorage with demo in Pycharm?

I guess this is similar to issue

#42

But user suggested method didnt work for me as I am not sure which 'lib' folder he is referring to when the folder structure is

src/cloudstorage/cloudstorage_api.py?

URLFetchUtils error parsing "Tue, 22 Oct 2013 19:28:45 GMT"

When trying to build java version of library with maven, URLFetchUtilsTest fails at line 52. It also seems strange, that there's no import for com.google.appengine.tools.cloudstorage.oauth.URLFetchUtils but somehow it's static method parseDate(String) is called and assertNotNull fails for method's result.
Probably caused by my locale/time zone. Here's surfire report:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite failures="1" time="1.2" errors="0" skipped="0" tests="6" name="com.google.appengine.tools.cloudstorage.oauth.URLFetchUtilsTest">
  <properties>
    <property name="java.runtime.name" value="Java(TM) SE Runtime Environment"/>
    <property name="sun.boot.library.path" value="/usr/lib/jvm/java-7-oracle/jre/lib/amd64"/>
    <property name="java.vm.version" value="24.72-b04"/>
    <property name="java.vm.vendor" value="Oracle Corporation"/>
    <property name="java.vendor.url" value="http://java.oracle.com/"/>
    <property name="path.separator" value=":"/>
    <property name="guice.disable.misplaced.annotation.check" value="true"/>
    <property name="java.vm.name" value="Java HotSpot(TM) 64-Bit Server VM"/>
    <property name="file.encoding.pkg" value="sun.io"/>
    <property name="user.country" value="CZ"/>
    <property name="sun.java.launcher" value="SUN_STANDARD"/>
    <property name="sun.os.patch.level" value="unknown"/>
    <property name="java.vm.specification.name" value="Java Virtual Machine Specification"/>
    <property name="user.dir" value="/home/martin/git/appengine-gcs-client/java"/>
    <property name="java.runtime.version" value="1.7.0_72-b14"/>
    <property name="java.awt.graphicsenv" value="sun.awt.X11GraphicsEnvironment"/>
    <property name="java.endorsed.dirs" value="/usr/lib/jvm/java-7-oracle/jre/lib/endorsed"/>
    <property name="os.arch" value="amd64"/>
    <property name="java.io.tmpdir" value="/tmp"/>
    <property name="line.separator" value="
"/>
    <property name="java.vm.specification.vendor" value="Oracle Corporation"/>
    <property name="os.name" value="Linux"/>
    <property name="classworlds.conf" value="/usr/local/apache-maven/apache-maven-3.2.1/bin/m2.conf"/>
    <property name="sun.jnu.encoding" value="UTF-8"/>
    <property name="java.library.path" value="/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib"/>
    <property name="java.specification.name" value="Java Platform API Specification"/>
    <property name="java.class.version" value="51.0"/>
    <property name="sun.management.compiler" value="HotSpot 64-Bit Tiered Compilers"/>
    <property name="os.version" value="3.5.0-54-generic"/>
    <property name="user.home" value="/home/martin"/>
    <property name="user.timezone" value="Europe/Prague"/>
    <property name="java.awt.printerjob" value="sun.print.PSPrinterJob"/>
    <property name="file.encoding" value="UTF-8"/>
    <property name="java.specification.version" value="1.7"/>
    <property name="user.name" value="martin"/>
    <property name="java.class.path" value="/usr/local/apache-maven/apache-maven-3.2.1/boot/plexus-classworlds-2.5.1.jar"/>
    <property name="org.slf4j.simpleLogger.defaultLogLevel" value="info"/>
    <property name="java.vm.specification.version" value="1.7"/>
    <property name="sun.arch.data.model" value="64"/>
    <property name="java.home" value="/usr/lib/jvm/java-7-oracle/jre"/>
    <property name="sun.java.command" value="org.codehaus.plexus.classworlds.launcher.Launcher -e install"/>
    <property name="java.specification.vendor" value="Oracle Corporation"/>
    <property name="user.language" value="cs"/>
    <property name="awt.toolkit" value="sun.awt.X11.XToolkit"/>
    <property name="java.vm.info" value="mixed mode"/>
    <property name="java.version" value="1.7.0_72"/>
    <property name="java.ext.dirs" value="/usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext"/>
    <property name="securerandom.source" value="file:/dev/./urandom"/>
    <property name="sun.boot.class.path" value="/usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes"/>
    <property name="java.vendor" value="Oracle Corporation"/>
    <property name="maven.home" value="/usr/local/apache-maven/apache-maven-3.2.1"/>
    <property name="file.separator" value="/"/>
    <property name="java.vendor.url.bug" value="http://bugreport.sun.com/bugreport/"/>
    <property name="sun.cpu.endian" value="little"/>
    <property name="sun.io.unicode.encoding" value="UnicodeLittle"/>
    <property name="sun.desktop" value="gnome"/>
    <property name="sun.cpu.isalist" value=""/>
  </properties>
  <testcase time="0.061" classname="com.google.appengine.tools.cloudstorage.oauth.URLFetchUtilsTest" name="testParseDateRoundtrip"/>
  <testcase time="0.004" classname="com.google.appengine.tools.cloudstorage.oauth.URLFetchUtilsTest" name="testParseDateSampleDate">
    <failure type="java.lang.AssertionError">java.lang.AssertionError
    at org.junit.Assert.fail(Assert.java:92)
    at org.junit.Assert.assertTrue(Assert.java:43)
    at org.junit.Assert.assertNotNull(Assert.java:526)
    at org.junit.Assert.assertNotNull(Assert.java:537)
    at com.google.appengine.tools.cloudstorage.oauth.URLFetchUtilsTest.testParseDateSampleDate(URLFetchUtilsTest.java:52)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
</failure>
  </testcase>
  <testcase time="1.12" classname="com.google.appengine.tools.cloudstorage.oauth.URLFetchUtilsTest" name="testDescribeRequestAndResponseForApiClient"/>
  <testcase time="0.01" classname="com.google.appengine.tools.cloudstorage.oauth.URLFetchUtilsTest" name="testDescribeRequestAndResponseF"/>
  <testcase time="0.004" classname="com.google.appengine.tools.cloudstorage.oauth.URLFetchUtilsTest" name="testGetSingleHeader"/>
  <testcase time="0.001" classname="com.google.appengine.tools.cloudstorage.oauth.URLFetchUtilsTest" name="testCopyRequest"/>
</testsuite>

Create stub for unit testing

It would be nice to have stub methods available for unit testing the cloud storage API just like there are for the mail API, memcache API, etc.

Urgent. LocalRawGCSService is broken in 0.5

We had to rollback from 0.5 to 0.4.4 as LocalRawGCSService was broken in new version. Basically all local writes were failing with security.AccessControlException exception. This is I guess very serious, as effectively one can't use this whole library locally.

    Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
        at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.storeBlob(LocalRawGcsService.java:161)
        at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.finishObjectCreation(LocalRawGcsService.java:336)
        at com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1.call(GcsOutputChannelImpl.java:201)
        at com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1.call(GcsOutputChannelImpl.java:198)
        at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:108)
        ... 74 more
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.google.appengine.tools.development.agent.runtime.Runtime.invoke(Runtime.java:130)
        at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService$BlobStorageAdapter.storeBlob(LocalRawGcsService.java:153)
        ... 78 more
    Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "D:\_code\myProject\target\myProj\WEB-INF\appengine-generated\encoded_gs_key:L2dzL2x4X2NlbnRyYxlL2FnZW5jeXNjcmlwdC9jaGFyZ2UtdHlwZXMuanNvbg" "write")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
        at java.security.AccessController.checkPermission(AccessController.java:559)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
        at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:429)
        at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
        at com.google.appengine.api.blobstore.dev.FileBlobStorage.storeBlob(FileBlobStorage.java:46)
        ... 84 more
Caused by: java.io.IOException: java.lang.reflect

problem is caused most likely by this

Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "D:\_code\myProject\target\myProj\WEB-INF\appengine-generated\encoded_gs_key:L2dzL2x4X2NlbnRyYxlL2FnZW5jeXNjcmlwdC9jaGFyZ2UtdHlwZXMuanNvbg" "write")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)

Properly expose copy-in-the-cloud functionality

Currently the GCS API's copy-in-the-cloud method is exposed as an "internal only" function cloudstorage.storage_api._copy2. This is a useful function for general users and it would be nicer to be able to access this functionality with this library officially, rather than having to use the Google API Client to access the lower level JSON API just for this.

Demo can't find its libs

I followed the tutorial but it failed to import cloudstorage. To make it work, I added this to the top of python/demo/main.py:

from google.appengine.ext import vendor
vendor.add('lib')

Update build instructions for Java

Do we need both maven and ant?
Which is preferred

Replace compose hack (Python) for dev_appserver once implemented in SDK

Differences between appengine-gcs-client and google-cloud-storage

Hi,
when should I use this library and when should I use google-cloud-storage? They seem to do the same.

Google cloud Storage 308 Error while writing file

Hi All,
I tried to pull the report from Double Click Search API and put it into Google Cloud Storage by using google app engine(Python).

The following code is used to pull the report and place in google cloud storage.

If I hard code the report_run_id in ds_check_status_report it is writing the file properly but when I try to pull the report_run_id from bigquery and pass then it is throwing 308 error.

class ds_check_status_report(webapp.RequestHandler):

    _apptitle      = None
    _projectid     = None
    _projectnumber = None
   
    def get(self):
        cfg=appsettings()
##      report_run_id="kdajdsjflsjfl"  
        result=dsbqfuns._dsbqquery()
        result1=json.dumps(result['stacktrace'])
        res=json.loads(result1)
        self.response.write(res['rows'][0]['f'][0]['v'])
        dsutils.poll_report(res['rows'][0]['f'][0]['v'])

        


def poll_report(report_id):
  """Poll the API with the reportId until the report is ready, up to ten times.

  Args:
    service: An authorized Doublelcicksearch service.
    report_id: The ID DS has assigned to a report.
  """
  print "Enter into poll_report"
  cfg=appsettings()
  creds = create_credentials(cfg._client_id , cfg._client_secret, cfg._refresh_token)
  service = get_service(creds)
  try:
    request = service.reports().get(reportId=report_id)
    json_data = request.execute()
    if json_data['isReportReady']:
      print('The report is ready.')

      # For large reports, DS automatically fragments the report into multiple
      # files. The 'files' property in the JSON object that DS returns contains
      # the list of URLs for file fragment. To download a report, DS needs to
      # know the report ID and the index of a file fragment.
      for i in range(len(json_data['files'])):
        print('Downloading fragment ' + str(i) + ' for report ' + report_id)
        download_files(service, report_id, str(i)) # See Download the report.
        
    else:
      print('Report is not ready. I will try again.')

  except:
    exc_type, exc_value, exc_traceback = sys.exc_info()
    print exc_type, exc_value, exc_traceback 


def download_files(service, report_id, report_fragment):
  """Generate and print sample report.

  Args:
    service: An authorized Doublelcicksearch service.
    report_id: The ID DS has assigned to a report.
    report_fragment: The 0-based index of the file fragment from the files array.
  """
  print "Enter into download_files", report_id
  filename="/awstogcs/DoubleClickSearch_Campaign"+report_id+"_"+report_fragment+".csv"
  write_retry_params = _gcs.RetryParams(backoff_factor=1.1)
  gcs_file=_gcs.open(filename, 'w', content_type='text/plain',retry_params=write_retry_params)
  request = service.reports().getFile(reportId=report_id, reportFragment=report_fragment)
  gcs_file.write(request.execute())
  gcs_file.close()

Error Message:

INFO     2017-06-26 13:23:37,417 module.py:809] default: "PUT /_ah/gcs/awstogcs/DoubleClickSearch_CampaignAAAndQT-U9IDyFYX_0.csv?upload_id=encoded_gs_file%3AYXdzdG9nY3MvRG91YmxlQ2xpY2tTZWFyY2hfQ2FtcGFpZ25BQUFuZFFULVU5SUR5RllYXzAuY3N2 HTTP/1.1" 308 -
INFO     2017-06-26 13:23:37,710 module.py:809] default: "PUT /_ah/gcs/awstogcs/DoubleClickSearch_CampaignAAAndQT-U9IDyFYX_0.csv?upload_id=encoded_gs_file%3AYXdzdG9nY3MvRG91YmxlQ2xpY2tTZWFyY2hfQ2FtcGFpZ25BQUFuZFFULVU5SUR5RllYXzAuY3N2 HTTP/1.1" 308 -
INFO     2017-06-26 13:23:37,976 module.py:809] default: "PUT /_ah/gcs/awstogcs/DoubleClickSearch_CampaignAAAndQT-U9IDyFYX_0.csv?upload_id=encoded_gs_file%3AYXdzdG9nY3MvRG91YmxlQ2xpY2tTZWFyY2hfQ2FtcGFpZ25BQUFuZFFULVU5SUR5RllYXzAuY3N2 HTTP/1.1" 308 -
INFO     2017-06-26 13:23:38,157 module.py:809] default: "PUT /_ah/gcs/awstogcs/DoubleClickSearch_CampaignAAAndQT-U9IDyFYX_0.csv?upload_id=encoded_gs_file%3AYXdzdG9nY3MvRG91YmxlQ2xpY2tTZWFyY2hfQ2FtcGFpZ25BQUFuZFFULVU5SUR5RllYXzAuY3N2 HTTP/1.1" 308 -
INFO     2017-06-26 13:23:38,440 module.py:809] default: "PUT /_ah/gcs/awstogcs/DoubleClickSearch_Campa

listbucket raises UnicodeEncodeError

Here is part of a trace:

  File "/base/data/home/apps/......py", line ...., in .....
    for file_stat in gcs.listbucket(path):
  File "XXXX/cloudstorage/cloudstorage_api.py", line 550, in __iter__
    self._path + '?' + urllib.urlencode(self._options))
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib.py", line 1307, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbc' in position 45: ordinal not in range(128)

At this point, self._options contains
{'marker': u'ZZZZ/check/\xbc', 'prefix': u'ZZZZ/'}
Apparently, str(u'ZZZZ/check/\xbc') fails.

A fix I implemented seems to overcome this issue on our code base, but I don't want to invest too much time into understanding the implementation of appengine-gcs-client:

Index: cloudstorage/cloudstorage_api.py
===================================================================
--- cloudstorage/cloudstorage_api.py    (revision 9268)
+++ cloudstorage/cloudstorage_api.py    (revision 9269)
@@ -645,7 +649,11 @@
     if next_marker is None:
       self._options.pop('marker', None)
       return False
-    self._options['marker'] = next_marker
+    if isinstance(next_marker, unicode):
+      self._options['marker'] = next_marker.encode('utf8')
+    else: # Can this happen?
+      self._options['marker'] = str(next_marker) 
+
     return True

   def _find_elements(self, result, elements):

Unable to upload to download large files

I realize that there is a 32 meg limit on response from app engine.
What is the solution for an gae project to upload and download large files ?
Can such an example be added to the python sample ?

Regex error while writing file to cloud storage

Here is the snippet of code at which I am receiving a ValueError:

def write_to_gcs(self, blob_info):
    #out_file_name = '/gs/symptom-images/' + blob_info.filename
    out_file_name = '/gs/falsebucket/testimage'
    outfile = gcs.open(out_file_name, 'w', content_type=blob_info.content_type)
    self.file_transfer(blob_info.open(), outfile)
    return blobstore.create_gs_key(out_file_name)

Here is my stack trace:

  File "/Users/Shayak/Documents/SourceCode/Shades/shade-server/app/common_handlers.py", line 79, in write_to_gcs

    outfile = gcs.open(out_file_name, 'w', content_type=blob_info.content_type)

  File "/Users/Shayak/Documents/SourceCode/Shades/shade-server/app/cloudstorage/cloudstorage_api.py", line 80, in open

    common.validate_file_path(filename)

  File "/Users/Shayak/Documents/SourceCode/Shades/shade-server/app/cloudstorage/common.py", line 243, in validate_file_path

    'but got %s' % path)

ValueError: Path should have format /bucket/filename but got /gs/symptom-images/testImage1.png

Digging deeper into the regex:

_GCS_BUCKET_REGEX_BASE = r'[a-z0-9\.\-_]{3,63}'
_GCS_FULLPATH_REGEX = re.compile(r'/' + _GCS_BUCKET_REGEX_BASE + r'/.*')

Seems like the fullpath regex match will always fail with the starting /gs because it needs a minimum of 3 characters. However, removing the leading /gs invokes an error as well:

ValueError: filename must start with "/gs/": /symptomimages/testImage1.png

Solution seems to be to remove the leading /gs and then match on the remaining file path

Please move to java-docs-samples & python-docs-samples

This repo should move to the *-doc-samples repos -- and then be archived.

@GoogleCloudPlatform/devrel-samples FYI

Can I use gcs-client in GAE managed vm?

Not people answer this issue
http://stackoverflow.com/questions/30511807/store-data-to-cloud-storage-in-gae-managed-vms

Add Blobstore-like upload/download operations

Greetings,

Following encouragement to move away from Blobstore, I am trying to use GCS client library for operations I previously used Blobstore for. I would like to have an operation for uploading directly to Cloud Storage and then receiving a callback request to an endpoint in my application, and an operation for creating a short-live public URL for serving a non-public object from Cloud storage.

I know such operations are available with Cloud Storage API and in dedicated libraries, but the Cloud Storage API (and therefore libraries that work with it) are not available in the GAE development server, making it very difficult to develop with.

I would rather not upload/download objects through my application, as it seems an unnecessary trip and drain of the application resources, and really a set back compared to upload/download through Blobstore.

Thanks.

Implement functionality to list blob versions

There is no way to list versions of a blob.

Java: cannot depend on repackaged classes form appengine API jar

Starting 1.9.54 GAE ApI is changing the repackaged classes so that they cannot be used anymore from customers.

I've a PR pending for this change.

InternalError(5:) on rest_api.py

Hi,

I am having problems using this library, I did a very simple python program to list files in my bucket and I am getting an unexpected error, the same happens when trying to read files from a bucket.

Code

# -*- coding: utf-8 -*-
from google.appengine.ext import deferred
from google.appengine.ext import ndb
import logging
import os
import cloudstorage as gcs
import webapp2

from google.appengine.api import app_identity


class Handler(webapp2.RequestHandler):
    def get(self):
        bucket_name = os.environ.get('BUCKET_NAME',
                                     app_identity.get_default_gcs_bucket_name())

        self.response.headers['Content-Type'] = 'text/plain'
        self.response.write('Demo GCS Application running from Version: '
                            + os.environ['CURRENT_VERSION_ID'] + '\n')
        self.response.write('Using bucket name: ' + bucket_name + '\n\n')
        ll = gcs.listbucket ("/" + bucket_name)        
        for i in ll:
            self.response.write(i + " \n");
        #gcs_file = gcs.open("mylist.csv", mode="r")
        #line = gcs_file.readline()
        #self.response.write(line)

app = webapp2.WSGIApplication([('/view', Handler)])

Output on logs:

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/e~xxxdb-test/v1.394782655728550693/viewer.py", line 92, in get
    for i in ll:
  File "/base/data/home/apps/e~xxxdb-test/v1.394782655728550693/cloudstorage/cloudstorage_api.py", line 544, in __iter__
    status, resp_headers, content = self._get_bucket_fut.get_result()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
    self.check_success()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/data/home/apps/e~xxxdb-test/v1.394782655728550693/cloudstorage/storage_api.py", line 128, in do_request_async
    deadline=deadline, callback=callback)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/data/home/apps/e~xxxdb-test/v1.394782655728550693/cloudstorage/rest_api.py", line 198, in do_request_async
    follow_redirects=False)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/data/home/apps/e~xxxdb-test/v1.394782655728550693/cloudstorage/api_utils.py", line 188, in run
    raise e
InternalError: 5:

Does anyone has an idea of what I could do to solve that? Or has anyone else experienced similar issues?

Thanks in advance

copy2() resets content-type to binary/octet-stream if metadata is set

A file copied with copy2 (is there a copy1?) resets the content-type to binary/octet-stream on the destination file if the file is copied. The documentation does not specify that the metadata should contain the content-type (or how it could contain a content-type), and content-type is also a separate parameter using open(), so the expectation is that the content-type should be copied with such a call:

copy2(src, dst, {'x-goog-acl': 'public-read'})

Please provide at least documentation of how to copy the content type, or set it, or that it is removed.

Warnings still thrown from java dev server

Console output from the local dev server still has this warning:

WARNING: The Google Cloud Storage Java API is deprecated and will soon be removed. Please use the Google Cloud Storage Client library instead. Migration documentation is available here: https://cloud.google.com/appengine/docs/java/googlecloudstorageclient/migrate

I think this is caused by com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService still referencing com.google.appengine.api.files.*

This warning will mask any real dependencies I have in my code on the deprecated api...

User-Agent string doesn't conform to RFC

RFC 7231 §5.5.3 describes User-Agent strings, and "App Engine GCS Client" doesn't conform to the token specification.

I keep getting invalid stream header error

This is the code ,just like in the official documentation

GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(gcsFileName, 0, 1024 * 1024);
try (ObjectInputStream oin = new ObjectInputStream(
Channels.newInputStream(readChannel))) {
return oin.readObject();
}

I get an error in the ObjectInputStream everytime ,please help I need to be able to create an InputStream object . However If I serve to a HTTPRESPONSE object it work ,But I also want to use it tis way

Support for pip install for Python3

The Google Cloud Storage docs (here) show how to read and write to Cloud Storage using this library. I tried to install this with pip, but when I try to use python3, it fails as in this SO post, because the changes in #33 have not been uploaded to PyPI.

I saw here (#33), that they suggest using the google-cloud-python library. If that is indeed the recommend library, then the GCS docs should be updated. If it is not, the version number in setup.py needs to be bumped up and the new version needs to be uploaded to PyPI.

Blobstore filename field set to None

I've run the demo code because I was unable to access an uploaded file by Filename. It looks like the Blobstore on dev saves every created file with a Filename of None, including files created in the demo code.

Is there a fix for this?

NOT COMPATIBLE WITH gcloud SDK maven plugin

@GoogleCloudPlatform/cloud-tools-for-java This is not compatible with the gcloud SDK based Maven Plugin.

How to do chunked upload using this library?

I slice large file as several pieces and upload them over ajax since GAE has limited 32MB per request and will cause OOM. But I can not open the existing file with append mode, or seeking the specific position of file to write the chunked data.

I would like to know how to do it using this library or I only can do it using JSON API?
Thank you!

Incompatibility with createUploadUrl and dev app server

Using App Engine SDK 1.9.48 with Maven project and this library

<dependency>
    <groupId>com.google.appengine.tools</groupId>
    <artifactId>appengine-gcs-client</artifactId>
    <version>0.6</version>
</dependency>

I implementing a flow which will use an upload-url generated like this

import com.google.appengine.api.blobstore.BlobstoreService;
import com.google.appengine.api.blobstore.BlobstoreServiceFactory;
import com.google.appengine.api.blobstore.UploadOptions;

String bucket = ...;
String callback = ...;
UploadOptions uo = UploadOptions.Builder.withGoogleStorageBucketName(bucket);

BlobstoreService bs = BlobstoreServiceFactory.getBlobstoreService();
return bs.createUploadUrl(callback, uo)

The generated url is something like this
http://localhost:8888/_ah/upload/aghhcGUtYmFzZXIiCxIVX19CbG9iVXBsb2FkU2Vzc2lvbl9fGICAgICAgMAJDA

The callback is implemented like this

import com.google.appengine.api.blobstore.BlobstoreServiceFactory;
import com.google.appengine.api.blobstore.FileInfo;
import com.google.appengine.tools.cloudstorage.GcsFileMetadata;
import com.google.appengine.tools.cloudstorage.GcsFilename;
import com.google.appengine.tools.cloudstorage.GcsInputChannel;
import com.google.appengine.tools.cloudstorage.GcsService;
import com.google.appengine.tools.cloudstorage.GcsServiceFactory;

Map<String, List<FileInfo>> fileInfos = BlobstoreServiceFactory.getBlobstoreService().getFileInfos(request);
FileInfo file = ... get fileinfo from map ...;

String gsObjectName = file.getGsObjectName();
GcsFilename uploadedFile = ... parse gsObjectName  to GcsFilename object ...;

The toString() of the file object, is like this

{file=[<FileInfo: contentType = image/jpeg, creation = Fri Dec 23 10:42:59 CET 2016, filename = animals_030.jpg, size = 84015, md5Hash = ad40a7e4a6028bb4192e8189a7f09ab1, gsObjectName = /gs/my_bucket/hmgtHXafqTDbYkY6XkfWNw>]}

I tried reading the file in memory with openReadChannel method, but I get this error

java.io.FileNotFoundException: com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService@1dc2b580: No such file: GcsFilename(my_bucket, hmgtHXafqTDbYkY6XkfWNw)

I also tried getting the file metadata with getMetadata method, but the GcsFileMetadata object is null

Is the uploadUrl flow fully integrated with the local GcsService object?
Does it work only on online environment?

Release 1.0 version for Java

The Java version of this library seems unlikely to get further updates. As best I can tell, it's for the java7 runtime only which is scheduled to be turned off on January 19. I suggest bumping it to 1.0 and releasing it.

Or deprecate it.

Or both.

Could you add an example for storing image files in the demo?

Avoid using dependency version ranges

Due to the issue with google-http-java-client googleapis/google-http-java-client#438 our production build just started to fail today morning. Such an issue shouldn't threaten anyone's build, but such errors can escalate quickly is dependency ranges are used in released dependencies.
The version <version>[1.19,2.0)</version> used in the pom should be replaced to a fix, release version to ensure build stability.

Bug: cloudstorage.open() writes to the options dictionary passed as parameter

I use a dict datastructure to store permission sets that should be CONSTANT and pass it to cloudstorage.* functions. Some cloudstorage functions write to this permission set and modify it.
This changes my constant GCS_ACL_OPTIONS['public'] and adds options that cause the ValueError.

GCS_ACL_OPTIONS = {
    "public": {'x-goog-acl': 'public-read'},
    "private": {'x-goog-acl': 'private'},

# Fails
ostream = cloudstorage.open(filename, content_type='application/json', mode='w',
                                options=GCS_ACL_OPTIONS[permission])
# Works
ostream = cloudstorage.open(filename, content_type='application/json', mode='w',
                                options={'x-goog-acl': 'public-read'})

Alternates:
ValueError: option x-goog-api-version is not supported.
ValueError: option x-goog-metadata-directive is not supported.

File "distlib/cloudstorage/cloudstorage_api.py", line 94, in open
common.validate_options(options)
File "distlib/cloudstorage/common.py", line 308, in validate_options
raise ValueError('option %s is not supported.' % k)
ValueError: option x-goog-copy-source is not supported.

Readme should point to java as well as python

AppEngine Google Cloud Storage clients

Documentation can be found here:
https://cloud.google.com/appengine/docs/python/googlecloudstorageclient/

Local blobstore upload test

Hi,
you have provided an example how to test blobstore serving locally.
https://github.com/GoogleCloudPlatform/appengine-gcs-client/blob/master/java/src/test/java/com/google/appengine/tools/cloudstorage/TestServingViaBlobstoreApi.java
Is it also possible to test blobstore upload locally? The url generated by the following code is not reachable:

UploadOptions options = UploadOptions.Builder.withGoogleStorageBucketName("param");
String urlRoot = "/blob/upload/" + String.valueOf(param);
String blobUploadUrl = BlobstoreServiceFactory.getBlobstoreService()
        .createUploadUrl(urlRoot, options);