Allow for 1.x style await, without having to use for await about async-pool HOT 9 CLOSED

My solution:

export async function asyncPoolAll<IN, OUT>(
	poolLimit: number,
	array: readonly IN[],
	iteratorFn: (generator: IN) => Promise<OUT>
): Promise<OUT[]> {
	const results: OUT[] = [];
	for await (const result of asyncPool(poolLimit, array, iteratorFn)) {
		results.push(result);
	}
	return results;
}

Then using the above:

const timeout = ms => new Promise(resolve => setTimeout(() => resolve(ms), ms));

// if we just want to run the callback as a series of jobs to complete
await asyncPoolAll(2, [1000, 5000, 3000, 2000], timeout);

// if we need to do something with the result
const timeoutResults = await asyncPoolAll(2, [1000, 5000, 3000, 2000], timeout);

from async-pool.

Hi @jpike88,

Thank you for providing your use case feedback and for also providing a solution.

Considering the solution is derived from the 2.x API, I'd like to avoid including it in the library to keep it minimal. That being said, I am open to improve the README documentation and include such function in the "Migrating from 1.x" section. Would you like to create a PR?

Thank you

from async-pool.

#39

from async-pool.

Thank you!

from async-pool.

IMO this should be available as a method in the package, maybe by calling asyncPool.all.

Like @jpike88, I only need it to "just run the jobs", and if I have to define my own function to make it do that, then it is more efficient to just copy-paste the source code for the 1.x package into that function.

from async-pool.

@atjn can you post examples of how you use it?

from async-pool.

Absofruitly:

let completedFileProcesses = 0;

await asyncPool(
  4,
  filesToProcess
  async item => {
    await processFile(item);
    completedFileProcesses++;
    console.log(`${(completedFileProcesses / filesToProcess.length)*100}%`);
  },
);

This is part of a build tool. It takes a list of files that need to be minified (filesToProcess), it then minifies them with processFile(), and keeps a counter running to see how far it has come.

I want code execution to stop here and wait until all minification has ended. The next piece of code handles something else, and requires that the minified files are done.

from async-pool.

Please, where does (a potential) asyncPool.all fit in your example?

PS: On 2.x API, your code looks like this:

let completedFileProcesses = 0;

for await (const result of asyncPool(4, filesToProcess, processFile)) {
  completedFileProcesses++;
  console.log(`${(completedFileProcesses / filesToProcess.length)*100}%`);
}

from async-pool.

Please, where does (a potential) asyncPool.all fit in your example?

The above code is how I use the package today (I am still using the 1.x API). My proposal would be to replace line 3:

await asyncPool(

..with a new function:

await asyncPool.all(

..which does the same thing as the 1.x API did. So in this case, it returns a promise that resolves only when all the files have been minified.

PS: On 2.x API, your code looks like this:

let completedFileProcesses = 0;

for await (const result of asyncPool(4, filesToProcess, processFile)) {
  completedFileProcesses++;
  console.log(`${(completedFileProcesses / filesToProcess.length)*100}%`);
}

Thank you for the suggestion, but I am not a big fan of the readability of that code. In the 1.x API it is very clear that there is a function containing three lines of code, which will be run for each file. I think it is readable because it looks a lot like a for loop.

In the 2.x example you show here, it is very hard to figure out that processFile is the main code step. It seems like only the 2 other lines are being run, and looks like processFile is just a variable that is used to set up the pool.

You can rewrite it to make it more readable:

let completedFileProcesses = 0;

for await (const result of asyncPool(
  4,
  filesToProcess,
  async item => {
    await processFile(item);
    completedFileProcesses++;
    console.log(`${(completedFileProcesses / filesToProcess.length)*100}%`);
)) {};

..but now you have a for loop which creates a result variable and injects it into an empty loop body. It works, but it is adding a lot of extra steps that seem wholly unnecessary, and make it harder to read and understand what the code is doing.

A general rule of thumb is that if you are creating variables and loops just to make things work, but without actually needing them, then you have structured your code incorrectly, and there will be a better way to do it where you don't need the unused variables and loops. And in this case there is, because that is exactly what the 1.x API provides.

from async-pool.