Add concurrency (parallel runs) support

[duddu]

This is to enable concurrent runs for a collection.

Premise
I love Postman, and the work that you guys have being doing with newman and this library is outstanding. I hate the fact that I have to use other horrible and obsolete software to run my performance/load tests, when I already would have everything I need with newman.
I appreciate that maybe if you never put concurrency in this library it may be a decision, but I thought I might try anyway.

Description
At the moment there is no real support for concurrency, and the workaround proposed in the newman example for parallel runs puts javascript out of memory with a very small number of concurrent runs. Also, it makes impossible to aggregate the results in a proper way. It looks like there is no way to get anything really working while instantiating multiple runs.
I decided to try and keep everything within the single run start. There are very few changes of design; the most relevant is making cursor and queue into a collection rather than single object, and using the cursor reference as the identifier of the process (aka virtual user). This id gets propagated to all the downstream instructions scheduled, allowing to run concurrent processes on the same collection. In order to achieve this in a clean way, I moved the initialisation of cursors and queues from the waterfall init method into the run start.

Notes

• From a performance point of view, I can't see any relevant decrement using the memory-check, and I was able to run without any problem a collection even with 10000 concurrent users against a local running server.
• The main concern is around the event handling: I decided to trigger always only once the start and done events, passing as cursor the first and the last respectively, also in order to avoid breaking compatibilities with the existing implementations. I avoided passing a cursor into the pause and abort triggers, since choosing one from the possible collection would have been quite arbitrary. Please advice on what is the best approach in your opinion. All the other triggers (e.g. request) get invoked for each real occurrence (per iteration, per concurrent process).
• I'm not sure what's the best approach to test this: we might duplicate all the existent integration tests adding concurrency options, or just add the concurrency in the existing ones, adjusting the expectation on the called counts of the various triggers. Clearly the test are green atm since they alway run with concurrency set at 1.
• If this approach sounds reasonable, adding ramp-up and hold-for features on top of concurrency would be quite easy, and would make possible a fully fledged load testing with newman (I kept it out of scope from this PR to make it easier to review and discuss).

There are still some possible optimisations to do, I didn't want to dig too deep without getting some feedback first. Please let me know what do you think; if this is something that even remotely you would be interested in supporting, I'm more than happy to adjust the changes following your advice, or even change completely the approach.

Todo

• Decide strategy for integration test concurrency
• Set concurrency default in newman run options
• Update newman reporters to print out number of concurrent users
• Update readme

[halfzebra]

I hope this PR gets some attention! 👍

Newman would benefit so much from running the tests concurrently!

[codenirvana]

@halfzebra I already reviewed this PR once, great work by @duddu 👏

Changes made are in the right direction but currently, we haven't thought of how parallel runs will be baked into the app.

Got busy with other stuff, will review this PR soon! 😅

[halfzebra]

Thanks for your reply! 👍

I feel like newman would really benefit from being able to run requests from a single collection concurrently. We're easily hitting 5 minutes mark for some collections, which is longer than some e2e tests.

Using Postman App for load testing would be super cool, but I've been assuming this wasn't the use-case @duddu had in mind. Looking forward to this feature!

[duddu]

@halfzebra thanks for your support
To be fair, the main use case for me is definitely load testing 😁 (have another look at the PR description! ☺️)
But, nevertheless, this is just to add parallel runs capability, how you play with iterations and concurrency together to address your use case is up to you!

[duddu]

@codenirvana really happy to know you are looking at this!
I am well conscious of the fact that handling concurrency on the app side would not be trivial, and might have a huge impact on the existing mechanics and presentational layer; and I appreciate that Postman might not be super keen to invest on this without figuring out first the possible functional and commercial implications 😢

Though, in my opinion working on the Newman side of the thing and getting the code properly just from a cli point of view, would still be a precondition for any work on the app. And after that we can take all the time needed to understand how (and whether) to plug this in the client ui; it doesn’t need to be be done all together!

Clearly, this makes sense ONLY if there is no other current plan to handle concurrency outside the runtime code (e.g. with partnerships with cloud services providers... 🤭).
That’s the only doubt that is stopping me to refine the proposed solution (which is waaay less performant then it could be) - if you could be so kind if clearing out this doubt for me (without getting fired), I am more than happy to keep this going!

[alexandre-g]

@codenirvana Any updates?

Looking for an API testing solution for our company, thought Postman would cut it but it seems like

1. Parallel running is not supported (hate being negative as I bet you are all working hard, but.. really?)
2. People are having issues running multiple newman instances
   Just wanting to say this may be a dealbreaker for many people wanting to go on your paid plans

[henningms]

Any news on doing load testing with Postman / newman? 😅

[halfzebra]

I'm not a maintainer of this project, just wanted to share some of my findings hoping they could help.

If there is a need for running multiple collections concurrently, I'd advise looking into implementing a test script on top of newman, this is what I did for our testing setup at work and it worked well.
For load testing, I'd recommend looking into k6 as it provides a more realistic simulation of traffic than just sending the number of requests per second.

The intended workflow assumes that users want to run requests in a collection sequentially. As far as I understand, Runscope competes with Postman and they also focus on sequential runs for requests, so it looks like it's not a part of "intended" functionality and it might take a while to adopt it.

Still hope this gets picked up, there's plenty of situations where requests inside collections can be executed concurrently, which would speed up a bunch of collections a lot!