Skip to content

Many implmentations of promise-concurrency in the wild does Promise.all in batches, which means the next batch doesn't start until all promises are resolved, so let's say you're running 10 parallel and one of them takes forever, then you could have been running 9 other jobs, this implementation will dequeue as soon as anything is resolved, tryin…

Notifications You must be signed in to change notification settings

davidcheung/big-cheungus

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

big-cheungus

                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣧⠀⠀⠀⠀⠀⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣿⣧⠀⠀⠀⢰⡿⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⡟⡆⠀⠀⣿⡇⢻⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠀⣿⠀⢰⣿⡇⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⡄⢸⠀⢸⣿⡇⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⣿⡇⢸⡄⠸⣿⡇⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢿⣿⢸⡅⠀⣿⢠⡏⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⣿⣿⣥⣾⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⣿⣿⣿⣿⣿⣆⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⣿⣿⡿⡿⣿⣿⡿⡅⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠉⠀⠉⡙⢔⠛⣟⢋⠦⢵⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣾⣄⠀⠀⠁⣿⣯⡥⠃⠀⢳⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣴⣿⡇⠀⠀⠀⠐⠠⠊⢀⠀⢸⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠀⠀⢀⣴⣿⣿⣿⡿⠀⠀⠀⠀⠀⠈⠁⠀⠀⠘⣿⣄⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⣠⣿⣿⣿⣿⣿⡟⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⣿⣷⡀⠀⠀⠀
                   ⠀⠀⠀⠀⣾⣿⣿⣿⣿⣿⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⣿⣿⣧⠀⠀
                   ⠀⠀⠀⡜⣭⠤⢍⣿⡟⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⢛⢭⣗⠀
                   ⠀⠀⠀⠁⠈⠀⠀⣀⠝⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠄⠠⠀⠀⠰⡅
                   ⠀⠀⠀⢀⠀⠀⡀⠡⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠁⠔⠠⡕⠀
                   ⠀⠀⠀⠀⣿⣷⣶⠒⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢰⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠘⣿⣿⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠰⠀⠀⠀⠀⠀
                   ⠀⠀⠀⠀⠀⠈⢿⣿⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⠊⠉⢆⠀⠀⠀⠀
                   ⠀⢀⠤⠀⠀⢤⣤⣽⣿⣿⣦⣀⢀⡠⢤⡤⠄⠀⠒⠀⠁⠀⠀⠀⢘⠔⠀⠀⠀⠀
                   ⠀⠀⠀⡐⠈⠁⠈⠛⣛⠿⠟⠑⠈⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
                   ⠀⠀⠉⠑⠒⠀⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀

Many implmentations of promise-concurrency in the wild does Promise.all in batches, which means the next batch doesn't start until all promises are resolved, so let's say you're running 10 parallel and one of them takes forever, then you could have been running 9 other jobs, this implementation will dequeue as soon as anything is resolved, trying to keep the amount of pending promises at the max at all time.

Note: This is meant for node 10+ only.

How to use

PromiseFulfillAll

const request = require('request');
const { PromiseFulfillAll } = require('big-cheungus');

const paginate = (page)=> `http://example.com/${page}`;
const urls = [1, 2, 3, 4, 5].map(paginate);
const executables = urls.map((url) => async () => request(url));
const results = await PromiseFulfillAll(executables});

normally in a Promise.all you would do the following

const request = require('request');

const paginate = (page)=> `http://example.com/${page}`;
const urls = [1, 2, 3, 4, 5].map(paginate);
const pendingPromises = urls.map((url) => request(url));
const results = await Promise.all(pendingPromises);

Appending to queue

const request = require('request');
const PromiseConcurrency = require('big-cheungus');

const promiseQueue = new PromiseConcurrency({ concurrency: 5 });
const paginate = (page)=> `http://example.com/${page}`;
const urls = [1, 2, 3, 4, 5].map(paginate);
urls.forEach((url) => {
  promiseQueue.push(async () => request(url, {json : true}))
});
const results = await promiseQueue.results();

Map(ish) signiature

  const { promiseMap } = require('big-cheungus');

  const paginate = (page)=> `http://example.com/${page}`;
  const urls = [1, 2, 3, 4, 5].map(paginate);
  const results = await promiseMap(urls, async (url) => request(url, {json : true}));

Extending the Array.prototype

  const { promiseMap } = require('big-cheungus');
  Array.prototype.bigCheungus = function(...args) {
    return promiseMap.apply(this, [this, ...args]);
  };

  const paginate = (page)=> `http://example.com/${page}`;
  const urls = [1, 2, 3, 4, 5].map(paginate);
  const results = await urls.bigCheungus(async (url) => request(url, {json : true}));

  // or if you want concurrency
  const concurrencyOptions = {concurrency: 2};
  const results = await urls.bigCheungus(
    async (url) => request(url, {json : true}),
    concurrencyOptions
  );

About

Many implmentations of promise-concurrency in the wild does Promise.all in batches, which means the next batch doesn't start until all promises are resolved, so let's say you're running 10 parallel and one of them takes forever, then you could have been running 9 other jobs, this implementation will dequeue as soon as anything is resolved, tryin…

Topics

Resources

Stars

Watchers

Forks

Packages

No packages published