(core) Support for multi source dependency

[awspbade]

Some common scenarios needs to wait for more than one file to go for the next step in the pipeline, common examples are joining different tables to enrich the data

I think there're two ways to make this.

1. Make WaitingStage X time to let all files are ready. Simpler method I guess.
2. I read some forums that propose: a manifest file as an input of the process that needs N files, also to control a full cycle of execution all files needs a prefix per cycle. Every time you receive a new file you check the prefix and look for all files needed using the same prefix, if all are uploaded you run the next step.

Both alternatives could be used in different scenarios.

[malachi-constant]

Agreed that this is a valuable feature add. Let's groom and add to the roadmap.

[awspbade]

is this feature going to be developed?

[malachi-constant]

@awspbade I'm hoping to look into this soon. We are working on an upcoming major version beta and this will be addressed after all essential features and enhancements are in.

It will be an interesting ask though given that we handle all stage orchestration with event rules, maybe it warrants a statemachine or some orchestrator to handle logical conditions.