Comments (26)
This was broken by the following commit:
googleapis/google-http-java-client@7d4a048
The problem disappeared when I changed GenericUrl.java as it was before:
result.add(verbatim ? sub : CharEscapers.decodeUriPath(sub));
=>
result.add(verbatim ? sub : CharEscapers.decodeUri(sub));
from java-storage.
I do confirm: it works with the libraries-bom 3.2.0, but no longer works with 3.3.0.
I will investigate the root couse.
from java-storage.
Workaround:
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.33.0</version>
</dependency>
from java-storage.
Thanks for the detailed report. I think you've nailed the root cause. We're going to have to dig into this one and figure out exactly how to fix it. The old http java client behavior was a common but incorrect handling of plus signs that caused other problems. Something in the GCS stack--I'm not sure what yet--was depending on the buggy behavior. If we're lucky, we can fix it here. If not, we might have to fix this somewhere on the server.
from java-storage.
googleapis/google-http-java-client#398
from java-storage.
Encoding spaces as "%20" not as "+" if met in the path component of URL should probably help.
from java-storage.
Yes, that's exactly what we should be doing. The question is what piece of code in which project is doing this encoding. It probably isn't in this repo.
from java-storage.
The easiest way to fix:
com.google.api.client.util.escape.PercentEscaper
private static final char[] URI_ESCAPED_SPACE = {'+'};
=>
private static final char[] URI_ESCAPED_SPACE = "%20".toCharArray();
It fixes problem with the storage. But it may have some side effects.
from java-storage.
Where do you propose making that change?
from java-storage.
I guess the source of the com.google.api.client.util.escape.PercentEscaper class exists only here:
https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/util/escape/PercentEscaper.java
from java-storage.
I don't think we're going to change http-java-client. 1.33.0 had a bug here and 1.34.0 fixed it. The problem is some other code somewhere has a complementary bug.
from java-storage.
We should dig into the actual HTTP calls that are happening here to understand what the server is sending us. That might narrow this down. See
https://cloud.google.com/storage/docs/xml-api/overview
and
https://cloud.google.com/storage/docs/json_api/
I'm not sure which of these two the client library is wrap[ping.
from java-storage.
https://cloud.google.com/storage/docs/request-endpoints#encoding
from java-storage.
Per the GCS API docs %20 is correct and + is wrong. This is consistent with the HTTP spec as well. Something is not following the docs, and I now think it might actually be in this repo. let me look
from java-storage.
As I can see, the storage passes the names of bucket and object to the http client as is. The client is responsible for URL encoding/decoding. It encodes spaces as '+', but since 1.33.0 '+' is no longer decoded as space. Encoding as '%20' allows to decode it properly.
from java-storage.
The client should not encode spaces as +. Just need to figure out exactly where that happens so I can fix it.
from java-storage.
This might be it:
from java-storage.
This works only to obtain a signed URL
from java-storage.
This is stack trace from the point where the storage parameters (bucket name, object name) are starting to be encoded:
at com.google.api.client.http.UriTemplate$CompositeOutput.getEncodedValue(UriTemplate.java:160)
at com.google.api.client.http.UriTemplate.getSimpleValue(UriTemplate.java:330)
at com.google.api.client.http.UriTemplate.expand(UriTemplate.java:314)
at com.google.api.client.http.UriTemplate.expand(UriTemplate.java:229)
at com.google.api.services.storage.Storage$Objects$Get.buildHttpRequestUrl(Storage.java:7014)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:422)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeMedia(AbstractGoogleClientRequest.java:502)
at com.google.api.services.storage.Storage$Objects$Get.executeMedia(Storage.java:7006)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.load(HttpStorageRpc.java:631)
at com.google.cloud.storage.StorageImpl$16.call(StorageImpl.java:589)
at com.google.cloud.storage.StorageImpl$16.call(StorageImpl.java:586)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.storage.StorageImpl.readAllBytes(StorageImpl.java:585)
at com.google.cloud.storage.StorageImpl.readAllBytes(StorageImpl.java:577)
at storage.Generic.main(Generic.java:36)
from java-storage.
This is quite a huge bug, preventing anyone from reading files containing spaces in their name, which is quite frequent.
Apparently, it exists in the latest version 1.103.0, but not in version 1.102.0. Wouldn't it be possible to release a new version identical to 1.102.0 for those upgrading to the latest version?
I had the luck to have caught this issue by a test before going to production, but I guess this could break a lot of applications.
from java-storage.
This might be it:
This only applies to signing objects. The code path is not used when reading objects.
from java-storage.
This was broken by the following commit:
googleapis/google-http-java-client@7d4a048The problem disappeared when I changed GenericUrl.java as it was before:
result.add(verbatim ? sub : CharEscapers.decodeUriPath(sub));
=>
result.add(verbatim ? sub : CharEscapers.decodeUri(sub));
I can confirm this finding.
from java-storage.
@elharo Have you considered update com.google.api.client.util.escape.PercentEscaper
private static final char[] URI_ESCAPED_SPACE = {'+'};
=>
private static final char[] URI_ESCAPED_SPACE = "%20".toCharArray();
from java-storage.
closing this issue as its duplicate of #121.
from java-storage.
@athakor I don't know why you choose to close this old bug and keep the new one which it seems has less priority as well, but any way we are still waiting for a fix, we can just use the work around and make sure google-http-client stays with none problematic version, but we use many Google client java libraries, not sure we want to take the risk of updating them but keeping one of their transitive dependencies.
from java-storage.
@montss this issue has been fixed. wait until next version of google-http-client
.
from java-storage.
Related Issues (20)
- Cloud strorage v2 API, writeObject missing header x-goog-request-params for client streaming call HOT 9
- Cloud storage v2 API, add support for batch operations HOT 2
- Retrying idempotent requests that return 503 backendError HOT 1
- com.google.cloud.storage.StorageException: java.lang.IllegalStateException: Connection pool shut down when trying to check if bucket exists HOT 5
- CVE-2023-33953 through io.grpc.* (1.56.1) HOT 3
- com.example.storage.QuickstartSampleIT: testQuickstartGrpc failed HOT 1
- com.example.storage.QuickstartSampleIT: testQuickstartGrpcDp failed HOT 1
- Google Cloud Storage library using deprecated javax package HOT 4
- Async / streaming support
- com.example.storage.bucket.PrintPubSubNotificationTest: testPrintBucketPubSubNotification failed HOT 2
- Warning: a recent release failed HOT 2
- Add support for Autoclass v2.1 HOT 1
- grpc compatibility with storage 2.27.x HOT 3
- Google Cloud Storage: unexpected exception after successfully writing to a Google Cloud Storage object HOT 2
- grpc compatibility issue 2.27.1 HOT 9
- GCP cloud storage downloaded file corruption HOT 4
- Improve Virtual Thread compatibility
- Unable to initiate resumable uploads from java sdk: Cloud storage v2 API. HOT 4
- Unable to Get or Refresh access token from Google Credentials instance in java sdk HOT 1
- Novel transient error introduced between 2.27.1 and 2.30.1? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from java-storage.