Comments (9)
From @stephenplusplus on September 5, 2017 21:14
The apiResponse pattern is library-wide and could certainly be re-opened for discussion. Many of our API calls are not autopagination methods, and because of this, it's easy to return our simplified response alongside the raw API response. However, the autopagination methods involve multiple raw API responses, which is why we didn't include them; it wouldn't be clear which of many objects were relevant to the user.
Do you think we should return an array of responses?
A couple notes;
For list operations, we assign the relevant section of the raw API response to our custom object that's returned. E.g., for storage.getFiles, each returned File object has a metadata property that includes many details from the API response. I'm pointing this out just to note that we don't discard the entire API response, just the outer shell.
The other way to fix this problem (and ones like it) is to add a new method to get what you're after, e.g. storage.getDirectories()
@callmehiphop @lukesneeringer for thoughts.
from nodejs-storage.
From @esprehn on September 5, 2017 21:4
Looks like this is paginator.run_:
if (autoPaginate) {
this.runAsStream_(parsedArguments, originalMethod)
.on('error', callback)
.pipe(concat(function(results) {
callback(null, results);
}));
It's super confusing and not at all clear that to list the sub-directories you need to do getFiles with delimiter: '/', paginate turned off, and pass a callback. It'd be really nice this API could be made more ergonomic. :)
from nodejs-storage.
From @esprehn on September 5, 2017 21:57
There's two problems here, the first is that the docs are confusing and show examples with apiResponse but never explain that you need to pass autoPaginate: false:
https://googlecloudplatform.github.io/google-cloud-node/#/docs/storage/1.2.0/storage/bucket?method=getFiles
The second is that getting the list of sub-directories is hard and has some confusing surprises. Google Storage has files, prefixes and sometimes prefixes are also files...
ex. I'm seeing:
getFiles({ prefix: 'foo', delimiter: '/', autoPaginate: false })
returns the prefixes 'foo/' and no files.
getFiles({ prefix: 'foo/', delimiter: '/', autoPaginate: false })
returns prefixes 'foo/bar' and 'foo/baz' and a single file with the name 'foo/'.
I assume this is because creating a folder in the console generates a _$folder$ file, while just uploading a file with a prefix doesn't, and getFiles() will return the magic
from nodejs-storage.
From @stephenplusplus on September 5, 2017 22:2
@frankyn might be able to help with those questions about the Storage API.
from nodejs-storage.
From @esprehn on September 5, 2017 22:7
The docs seem to contradict the results here, I mostly want the GoogleStorage node API to behave similarly to what gsutil does. :)
ex. creating a folder in the console does not create a file with _$folder$ anymore, I just get a file with a / on the end and application/x-www-form-urlencoded;charset=UTF-8 as the Content-Type.
I'm happy to migrate that part of this discussion elsewhere if you tell me where to file those bugs.
Separately a new getDirectories() API would be super useful. Lots of folks all over are trying to figure out how to make the Google Storage API behave like gsutil, and needing to remember to pass delimiter: '/', prefix, autoPaginate, and then potentially filter out yourself (since the prefix can both be a prefix and a file of the same name, but not always) is a lot of ceremony to list some files.
from nodejs-storage.
@esprehn, I think there are 2 things we can do:
-
Remove the documentation for the
apiResponse
argument, since it doesn't exist by default. The examples showing how to useautoPaginate
will point out that the callback will receive thenextQuery
andapiResponse
arguments. -
We should add a
bucket.getDirectores()
method.
I'll get started on fixing the docs, and put a help wanted
label to the issue for the bucket.getDirectories()
feature request. If anyone from the community is willing to contribute their time for that, please do!
from nodejs-storage.
I think the docs might still be wrong or at least confusing. The docs for GetFilesRequest say about delimiter
Results will contain only objects whose names, aside from the prefix, do not contain delimiter. Objects whose names, aside from the prefix, contain delimiter will have their name truncated after the delimiter, returned in apiResponse.prefixes. Duplicate prefixes are omitted.
Based on this I would expect
bucket.getFilesStream({delimiter: '/'})
.on('data', function(f) { console.log(f) })
to return objects for each root prefix in the bucket. Say the bucket had these objects
a/foo.txt
b/foo.txt
b/foo/bar.txt
I would expect to see File
objects emitted for a/
and b/
, but nothing is ever passed to .on('data',...)
. Yet if you try exactly this say from https://cloud.google.com/storage/docs/json_api/v1/objects/list?apix=true you indeed get just the root prefixes in the bucket. I can find no combination of adding autoPaginate: false
or prefix: ''
that will work either.
Aside, from the behavior being different, what I really need is to know how to work around this. I need to be able to discover (I can't know them a priori) the root prefixed in a bucket
from nodejs-storage.
It seems to me that the API is saying: we'll give you files that do not contain your delimiter. But, so you know more about the files that we're excluding, we'll give you some accessory information in the response's "prefixes" property.
To get that information, you need the apiResponse
argument, which is not supported in stream mode. You can use bucket.getFiles()
with autoPaginate: false
:
var prefixes = []
function callback(err, files, nextQuery, apiResponse) {
prefixes = prefixes.concat(apiResponse.prefixes)
if (nextQuery) {
bucket.getFiles(nextQuery, callback)
} else {
// prefixes = array of prefixes
}
}
bucket.getFiles({ autoPaginate: false, delimiter: '/' }, callback)
from nodejs-storage.
I sent a PR to accept a directory
option to getFiles()
: #162
At this point, it's basically just an alias for prefix
. I wasn't sure what would be the most useful way to add the concept of a directory, so feel free to comment with suggestions. Thanks!
from nodejs-storage.
Related Issues (20)
- FileMetadata type missing null on metadata for delete HOT 1
- The build failed HOT 2
- GetServiceAccountOptions missing projectIdentifier field
- Unable to generate signedUrl when using WIF under GKE HOT 11
- File.from picks up incorrect bucket name from passed URL HOT 3
- FR: Support HNS enablement in bucket metadata
- CreateBucketRequest type is misaligned with JSON API
- storage HMAC keys: should get metadata for an HMAC key failed HOT 2
- storage HMAC keys: should make the key INACTIVE failed HOT 2
- tests: move HMAC system tests to use emulator
- storage bucket hierarchical namespace: should create a bucket without hierarchical namespace enabled (explicit) failed HOT 1
- Missing interface's property HOT 4
- storage HMAC keys: should delete the key failed
- Signed URL prefix HOT 1
- preconditions not working HOT 2
- File.save sometimes returns a Promise that never rejects or resolves HOT 9
- storage HMAC keys: should delete the key failed HOT 2
- File.save() with a Uint8Array fails HOT 9
- Methods that perform a fetch should throw any socket errors directly (ECONNRESET) HOT 9
- storage HMAC keys: should make the key INACTIVE failed HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nodejs-storage.