Comments (9)
I think I have managed to add the funcionality with this
export async function asyncPool(poolLimit, array, iteratorFn, exception) {
const ret = [];
const executing = [];
for (const [index, item] of array.entries()) {
process.stdout.write(`\rProcessing ${index + 1}/${array.length} ...`);
const p = Promise.resolve().then(() => iteratorFn(item, array));
ret.push(p);
if (poolLimit <= array.length) {
const e = p.then(() => executing.splice(executing.indexOf(e), 1)).catch(() => executing.splice(executing.indexOf(e), 1))
executing.push(e)
if (executing.length >= poolLimit) {
await Promise.race(executing)
}
}
}
const results = await Promise.allSettled(ret)
const rejected = results
.filter(({ status, reason }) => status === 'rejected' && reason.name === exception)
.map(({ reason }) => reason.error)
if (rejected.length) {
await asyncPool(poolLimit, rejected, iteratorFn, exception)
}
}
from async-pool.
Hi @alvaro-escalante thanks for your request.
I am afraid the change would break the original behavior of this library: "It rejects immediately as soon as one of the promises rejects. It resolves when all the promises completes".
Can't your use case be solved at app side? For example, by having your iteratorFn
to take care of retries before actually rejecting? Such as this #26 (comment)
Also note Promise.allSettled()
is fairly new, so the change would impose a significant difference in browser support (probably requiring polyfill)
from async-pool.
Hi Rafael, thanks for you response.
I can deal with the errors at app level and in fact in the code I presented I actually do that by throwing
the item I want to run again on the iteratorFn
, I am not so invested in Promise.allSettled()
as much as I just want a way to recursively run a failing request.
Like I do on both my functions, I think it would be an interesting proposal to under some circustances, controlled at app level, re-run/ push the 'failing' item. This could be a url that just timesout and can be safetly push
back in to the array to re-run,.
Whatever the error is, it could be modified and thrown at app level to the asyncPool
in my example I run asyncPool
rescursively.
Many thanks.
Alvaro.
from async-pool.
I think it would be an interesting proposal to under some circustances, controlled at app level, re-run/ push the 'failing' item
Please, what it would look like to use it if asyncPool had such feature implemented?
from async-pool.
I already did up on the previous code snippet, just checking with you, if you think it makes sense
from async-pool.
Above I see the potential asyncPool
implementation changes/updates. I am wondering what it would look like at the app level to use the modified-asyncPool. How simple/easy it would be to use it in the case you mention. Thanks
from async-pool.
FWIW, my understanding is that you believe this proposal would be a good addition to the library and the purpose of my question is to understand your point of view.
from async-pool.
This is my how I implemented it at the app level.
So say I have an array of urls, on this example, I am just removing the https://
part, and if one of the urls matches a condition I reject it and throw it so the asyncPool
can detect it and re run, obviously in this example, I then modify the url so it does not go on an infinite loop.
If for example the error was a 500 and I know the API just timed out, then I could just send the same url again, this is what I mean the control is on the app level, so the app decides when to send the error item to the asyncPool
again.
This is the whole thing working recursively
asyncPools.js Modified to re run cretain items
export async function asyncPool(poolLimit, array, iteratorFn, exception) {
const ret = [];
const executing = [];
for (const [index, item] of array.entries()) {
process.stdout.write(`\rProcessing ${index + 1}/${array.length} ...`);
const p = Promise.resolve().then(() => iteratorFn(item, array));
ret.push(p);
if (poolLimit <= array.length) {
// I know this is horrible but is the only way I know to let Promise.race ignore the failing url
const e = p.then(() => executing.splice(executing.indexOf(e), 1)).catch(() => executing.splice(executing.indexOf(e), 1))
executing.push(e)
if (executing.length >= poolLimit) {
await Promise.race(executing)
}
}
}
const results = await Promise.allSettled(ret)
const rejected = results
.filter(({ status, reason }) => status === 'rejected' && reason.name === exception)
.map(({ reason }) => reason.error)
if (rejected.length) {
await asyncPool(poolLimit, rejected, iteratorFn, exception)
}
}
index.js App level control
import { asyncPool } from './asyncPool.js'
const data = ['many urls', '....'];
const concurrent = 3;
const res = [];
// Mock API response - Let's suppose is a list of urls and if one of then is apple.co.uk add foo and re run by rejecting
const API = (item) =>
new Promise((resolve, reject) => {
if (item === 'https://www.apple.co.uk/') reject(item + 'foo')
setTimeout(() => resolve(item), 1000)
});
// Iterator Function
const fetchAPI = async (item) => {
try {
const request = await API(item);
res.push(request.replace(/^https?:\/\//, ''));
} catch (error) {
// The rejection is picked up by asyncPool with the name timeout, our `expection`
throw { name: 'timeout', error };
}
};
// Call function
(async () => {
await asyncPool(concurrent, data, fetchAPI, 'timeout');
console.log(res);
})();
This is just a solution I came up with cause I have this problem, I am not saying is the most elegant solution or the best, it just works for what I need, and just checking with you, maybe you see a better way.
from async-pool.
The new 2.x API doesn't use Promise.all
anymore, so I am closing this issue. If you still believe there's anything to be done in the new API please feel free to reopen. Thanks
from async-pool.
Related Issues (20)
- how to kill this XD HOT 3
- don't throw on empty array HOT 5
- Readability issue HOT 2
- Deno support HOT 1
- Typescript bindings HOT 4
- Chrome fails (Uncaught (in promise) TypeError: Failed to fetch) HOT 1
- another args to function HOT 1
- Failed to process the request HOT 3
- why using "array" as an input for iteratorFn() HOT 4
- question) How can i execute function with multiple arguments? HOT 1
- Benchmarks, overhead information HOT 1
- Unhandled rejections in some corner case HOT 2
- your code amazing!
- Allow for 1.x style await, without having to use for await HOT 9
- Feature: guarantee order of results HOT 3
- Does not "yield" reliably
- README.md usage issue HOT 2
- Allow iterable to be an async-iterable / async-generator HOT 2
- Add documentation about ordering no longer being preserved HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from async-pool.