feat(async): async functions in parallel are now called with additional item index and queue index arguments

[UnKnoWn-Consortium]

Description

feat(async): added item index as the second argument in the async callback functions called in parallel while queue index becomes the third argument.
test(async): added tests for item index and queue index
doc(async): updated description and usage for the two new arguments

Checklist

• Changes are covered by tests if behavior has been changed or added
• Tests have 100% coverage
• If code changes were made, the version in package.json has been bumped (matching semver)
• If code changes were made, the yarn build command has been run and to update the cdn directory
• If code changes were made, the documentation (in the /docs directory) has been updated

Resolves

Resolves #288

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
radash-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 2, 2023 9:48pm

[sodiray]

Hey @UnKnoWn-Consortium thanks for the PR 🙏 and sorry for the delay 🙃

This is interesting, I'm just not sure it's going to be very helpful. My concern is that the index will be misunderstood and/or misused by devs.

Would love to here you're thoughts, I'm happy to be convinced!

[UnKnoWn-Consortium]

Hello @rayepps my use case for this is rather simple. It is for building UI, specifically progress bars showing the status for each queue. May I know more about your concern on misuse?

[sodiray]

@UnKnoWn-Consortium My concern is that it feels a little like we're exposing an internal implementation detail that users won't understand unless they look at the code.

If an index is going to pass I would expect it to be the index of the item being worked on but this is passing the index of the queue that is doing the processing.

I could be wrong BTW, just hesitant

[UnKnoWn-Consortium]

@rayepps I do agree we are exposing a bit of the internal implementation but it is a useful bit in my opinion.

If an index is going to pass I would expect it to be the index of the item being worked on but this is passing the index of the queue that is doing the processing.

I think I can implement that. So we will have

const users = await parallel(3, userIds, async (userId, index, queueIndex) => {
  return await api.users.find(userId)
})

What do you think?

[sodiray]

@UnKnoWn-Consortium you think queueIndex is still useful if you have the individual index?

[UnKnoWn-Consortium]

@rayepps I believe they two serve different purposes. The former refers to the position of the item that is being processed in the original array while the latter refers to which queue or parallel pipeline the item is being processed in. Yes the latter can be considered an internal construct of the function but exposing the latter read-only does not hurt. It may be useful to somebody (myself included lol). (P.S. Maybe I should add it to the returned WorkItemResult too.) Also, to be frank, I cannot quite wrap my head around how item index can substitute queue index in my progress bar scenario....

Let me elaborate my use case further. I am using parallel to run multiple detached long-running tasks (e.g. video encoding and object recognition) in parallel. The number of tasks is limited by the number of CPU (or in some cases GPU) cores available. So effectively I am using it like a work queue. That is why I want to have a small peek into the internals of the queues and have something to monitor and report to the users. Before encountering radash, I was using a custom implementation that bore a lot of resemblance to parallel albeit being way less elegant and way more bloated.

[sodiray]

albeit being way less elegant and way more bloated

stop, I'm blushing

The former refers to the position of the item that is being processed in the original array while the latter refers to which queue or parallel pipeline the item is being processed in.

This makes sense. If you can bare with me a bit I'm curious if something like this would work. It's a naive example where you track the state without an index.

const results = await parallel(gpuCount, items, async (item) => { 
  incrementProgress()
  // do work on item
  incrementComplete()
}

[UnKnoWn-Consortium]

const results = await parallel(gpuCount, items, async (item) => { 
  incrementProgress()
  // do work on item
  incrementComplete()
}

Yes this is how I increment the overall progress bar. But without the queue number I cannot have by-queue (or as in this example by-GPU) progress bars.

[UnKnoWn-Consortium]

@rayepps I have taken the liberty to add the item index as the second arguments (and the queue index becomes the third) supplied to the async functions called in parallel in case I may not have time to work on it later on.

[rd-xx]

A similar PR (#291) was merged last year so is there any reason this change is not accepted @rayepps ?

I really need this and would like to avoid copy/pasting the implementation into my own codebase