Storage and url encoding about java-storage HOT 15 CLOSED

Can you reproduce this without scala?

from java-storage.

With a single file names 1 1

import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;

import java.nio.file.Path;

public class Yava {
  public static void main(String[] args) {
    Storage storage = StorageOptions.getDefaultInstance().getService();
    storage.list("eli-test").iterateAll().forEach(blob -> blob.downloadTo(Path.of("target", blob.getName())));
  }
}

Exception in thread "main" com.google.cloud.RetryHelper$RetryHelperException: com.google.cloud.storage.StorageException: 404 Not Found
No such object: eli-test/1+1
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54)
	at com.google.cloud.storage.Blob.downloadTo(Blob.java:233)
	at com.google.cloud.storage.Blob.downloadTo(Blob.java:217)
	at com.google.cloud.storage.Blob.downloadTo(Blob.java:260)
	at Yava.lambda$main$0(Yava.java:9)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at Yava.main(Yava.java:9)
Caused by: com.google.cloud.storage.StorageException: 404 Not Found
No such object: eli-test/1+1
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:229)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:675)
	at com.google.cloud.storage.Blob$2.run(Blob.java:238)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
	... 6 more
Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
No such object: eli-test/1+1
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
	at com.google.api.client.googleapis.media.MediaHttpDownloader.executeCurrentRequest(MediaHttpDownloader.java:255)
	at com.google.api.client.googleapis.media.MediaHttpDownloader.download(MediaHttpDownloader.java:185)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMediaAndDownloadTo(AbstractGoogleClientRequest.java:685)
	at com.google.api.services.storage.Storage$Objects$Get.executeMediaAndDownloadTo(Storage.java:6996)
	at com.google.cloud.storage.spi.v1.HttpStorageRpc.read(HttpStorageRpc.java:671)
	... 11 more

from java-storage.

Do you have a pom.xml or build.gradle for this?

from java-storage.

I have sbt: "com.google.cloud" % "google-cloud-storage" % "1.103.1"

from java-storage.

the only thing needed is the gcloud storage dependency

from java-storage.

@soujiro32167 Thanks for pointing out this bug, but it's related to google-http-java-client.

FYI, A string of characters that do not need to be encoded when used in URI query strings, as
specified in RFC 3986. Note that some of these characters do need to be escaped when used in
other parts of the URI and space is one of them which needs to be escape.

you can check list of rules for an encoding a string which is parts of the URI.

Rules :
https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/util/escape/PercentEscaper.java#L32

Also look at the below links which replacse the space with {'+'}

https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/util/escape/PercentEscaper.java#L218

https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/util/escape/PercentEscaper.java#L93

from java-storage.

The problem is not that the space is not being encoded. It's that it's being encoded incorrectly as a plus sign. This is wrong. The plus sign is the plus sign. It is not a space and does not represent one in URLs. This is a common confusion between URL encoding and application-x-www-formurl-encoding. The latter only applies in HTML forms.

This is complicated because there's a lot of broken code out there that does not follow the spec. To make matters worse, some of the GCS code is trying to be bug-for-bug compatible with other non-Google code.

from java-storage.

I thinks this is a duplicate of #57
it's still reproducible with google-http-client 1.34.1

Workaround to add this dependency:

        <dependency>
            <groupId>com.google.http-client</groupId>
            <artifactId>google-http-client</artifactId>
            <version>1.33.0</version>
        </dependency>

from java-storage.

The problem is not that the space is not being encoded. It's that it's being encoded incorrectly as a plus sign. This is wrong. The plus sign is the plus sign. It is not a space and does not represent one in URLs. This is a common confusion between URL encoding and application-x-www-formurl-encoding. The latter only applies in HTML forms.

This is complicated because there's a lot of broken code out there that does not follow the spec. To make matters worse, some of the GCS code is trying to be bug-for-bug compatible with other non-Google code.

Correct. Spaces should be encoded as %20 instead of a plus sign.

from java-storage.

The bug is in com.google.api.client.http.UriTemplate#expand or something it calls. This is where the + sign is incorrectly introduced.

from java-storage.

I built google-http-client 1.34.2-SNAPSHOT, the problem with spaces is gone!

from java-storage.

FYI if you import com.google.cloud:libraries-bom:4.1.0 or later the bug is fixed. We still need to release a version of the google-cloud-storage artifact that updates its dependencies.

  <dependencyManagement>
	 <dependencies>
	  <dependency>
	    <groupId>com.google.cloud</groupId>
	    <artifactId>libraries-bom</artifactId>
	    <version>4.1.0</version>
	    <type>pom</type>
	    <scope>import</scope>
	   </dependency>
	 </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
  	  <groupId>com.google.cloud</groupId>
  	  <artifactId>google-cloud-storage</artifactId>
  	</dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.13</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

from java-storage.

@soujiro32167 could you try using the latest release:
https://search.maven.org/artifact/com.google.cloud/google-cloud-storage-parent/1.104.0/pom

Thank you for your patience folks.

from java-storage.

Verified the latest version has resolved the issue (GoogleCloudPlatform/java-docs-samples#2064). Closing issue, please reopen if the issue persists.

from java-storage.

@frankyn thanks! I'll try it out

from java-storage.