Comments (12)
listTree
has a conceptual problem since it produces a promise for an array. My hope is to replace that with a promise for a Q stream in a future release—perhaps under a different name to retain backward compatibility—or a promise for completion and just use the guard emitter for streaming purposes.
from q-io.
That sounds pretty cool @ Q streams. In a particular case that I am working on, I need stat results on all files to check their mtimes, and so it is no problem if I get the entire array all at once; (The only problem is that after I listTree(), then I have to run stat() on all the paths. It sounds like Q streams or the guard emitter might help with that... but that's a separate issue from this.)
Edit: I should say: streaming isn't a problem yet here for me, and probably won't be in this application. My directory trees and associated data aren't big enough to eat up all the RAM - they can happily be all resident in memory for now. But I can see benefit to streaming though otherwise.
from q-io.
The guard function can already do what you need, I think.
FS.listTree(path, function (name, stat) {
// return true to include
// return false to exclude
// return null to exclude and skip subtree
}))
.then(function () {
})
from q-io.
That's a cool trick with guard. Looks like this would do the trick of snarfing the stats:
var stats = [];
FS.listTree(path, function(name, stat) {
stats.push([name, stat])
return true; // edit: don't forget, this is a guard function afterall
})
.then(function() {
return stats;
})
.then(function(stats) {
// awesome, got the stats I need!
});
But this doesn't address the speed gap of using Q.
The issue here is how long it takes listTree() to finish. The fact that it collects a list isn't really a problem. Using Q on stat and readdir inside of listTree is the primary cause of the slow down. I can rewrite listTree() to internally not use promises, and then it works like a little speed demon. Obviously, then it loses the clarity that promises provide which is undesirable. So, I'm wondering if there is potential to identify a specific bottleneck that for some reason causes Q-code to be slower than Node.js callback-style code (and by a large factor: 10x).
from q-io.
@dhbaird I presume that the problem is recursive array aggregation. It might be faster to pass a single array forward and push results onto it instead of concatenating results from all
. Of course, listTree
may lose some of the clarity in the process.
from q-io.
I thought recursive array aggregation might have been a problem too, but take a look at asyncFindTree()
. It uses recursive array aggregation (see its nested checkCompletion()
function). And it's fast.
When you get a moment, could you take a look at the following: asyncFindTree()
and asyncFindTreeQstat()
. The difference in code is very small: a single Q.nfcall()
is added, and that results in a substantial performance deviation. The run time goes from 3 seconds to 18 seconds.
Edit: Here's the diff (to make it easier to compare the two):
--- a.js 2013-08-12 17:58:04.111106000 -0600
+++ b.js 2013-08-12 17:58:07.262917000 -0600
@@ -1,7 +1,9 @@
+var Q = require('q');
var path = require('path');
var FS = require('fs');
-function asyncFindTree(basePath, cb) {
- FS.stat(basePath, function(error, stat) {
+function asyncFindTreeQstat(basePath, cb) {
+ return Q.nfcall(FS.stat, basePath)
+ .then(function(stat) {
if (stat.isDirectory()) {
FS.readdir(basePath, function(error, children) {
children = children.map(function(child) { return path.join(basePath, child); });
from q-io.
(Note - I edited previous comment to include diff)
from q-io.
@dhbaird That is very curious. nfcall
does do some extra work in this case, coercing FS.stat
to a promise for FS.stat
, manipulating the arguments, sending the apply
message to that promise, but I would not expect it to dominate the performance like that. You might try using Q.nfbind instead of Q.nfcall to cache the node-ified function in the parent scope to see whether that makes a difference.
from q-io.
@kriskowal something like this?
var stat2 = Q.nfbind(FS.stat); // aka denodeify
function asyncFindTreeQstat(basePath, cb) {
return stat2(basePath)
.then(function(stat) {
...
Good idea, but performance remains about the same.
from q-io.
Weirder and weirder.
I am going to have to let this issue idle for a while, but if you send a PR that helps the issue without impacting the API, I can get a patch out quickly. If it requires an API change, I can entertain a proposal as well.
from q-io.
Sounds good to me. I just wanted to get the word out, and I'm happy if you want to close this and re-open it later. I'll keep looking into it.
from q-io.
I will probably introduce listStream() and listTreeStream() in the v2 branch.
from q-io.
Related Issues (20)
- Reader().map() fails to limit properly the number of parallel instances of a function. HOT 3
- proxy requests HOT 2
- join() doesn't support network shares HOT 4
- not able to do x-www-unencoded body HOT 5
- Crash on reading large objects HOT 1
- problems by copying / moving data-files with special characters within path-variable HOT 7
- HTTP.read resolves with buffer instead of string HOT 1
- q-io downloading or writing too much data in version 1.13 HOT 14
- Mock: fs.copy() is buggy HOT 1
- removeTree() thinks it fails with ENOTEMPTY, but works on second try HOT 2
- `write()` resolves before content is written HOT 5
- Bump version of collections to avoid conflict with ES6 HOT 3
- filetree compatible with jstree
- `fs.copyTree` does not copy a large amount of files HOT 1
- is v2 up to date? HOT 3
- [v2] Last written file is always empty HOT 1
- Drop usage of collections, it breaks globals HOT 15
- Fixing a Prototype Override Protection Bypass Vulnerability in qs HOT 3
- Connect Timeout setting HOT 1
- Need 'qs' version updated to latest version to resolve Security Vulnerabilities HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from q-io.