tldr; async-batch is a great way to manage complex task queues.
Promises and Async/Await have made asynchronous programming in Javascript / Node.js simpler and cleaner. But I frequently find myself processes sets of Promises concurrently and having to decide the best approach.
Async/await is a syntactic wrapper around Promises. So I will refer to Promises and Async/await interchangeably.
Below are several strategies with their pros and cons.
Strategy 1 - Simple For Loop
Pros:
- Simple
- Run promises sequentially, avoid potential concurrency issues
- Control load on external resources – ie. don't make too many queries at the same time
- Easy debugging / monitoring
Cons:
- Slower if you want concurrent execution
Strategy 2 - Promise.all with Array.map
Pros:
- Concise code that can run all functions concurrently
Cons:
- Assumes the number of concurrent Promises is small and there are sufficient resources to run concurrently.
- Debugging becomes more complicated that a simple for loop if one of the promises fails
Strategy 3 - Async Batch
Async-Batch (GitHub, NPM) is an excellent dependency free library for managing concurrency in a set of promises. It allows you to specify the number of parallel promises running. As one completes, it starts the next in the queue.
Note: this library using ES6 syntax, so until Node.js fully supports it, you may need to use require().default.
Pros:
- Efficiently manages large sets of async processes
- Relatively simple way to run complex task queues
- Dependency free
- Can scale the Parallelism from 1 to x dynamically, adjusting to available resources.
- Debugging is simpler than Promise.all since you can set parallelism to 1.
Cons:
- ? Possibly more complex to read than the For loop.
Conclusion:
Each method has its advantages, but I find myself using async-batch regularly for simple or complex scenarios.