Resources in parallel unless dependson

[mgreenegit]

Summary of the new feature / enhancement

There was strong feedback at PS & DevOps Summit that rather than having a Parallel group type, resources should just always run in parallel unless "DependsOn" expresses a relationship so resource 1 must complete before resource 2 can start. The intention is to have configuration behave in the same way that Azure deployments behave, hoping that new users might have intuition that the two could work the same way.

After sleeping on the question, my main concern is that we don't really know which resources will work well when run in parallel, and some might cause problems. We could express it as a capability but that would create work for the community.

Tagging a few people to invite them to continue the discussion. Thank you!
@alexandair, @Jaykul, @brwilkinson

[jeremytbrun]

My first thought is that resource conflicts and issues can happen in Azure deployments as well can't they? But most of the time they can be resolved by configuring dependsOn, changing references, or implementing modules (Bicep) or nested templates (json).

[brwilkinson]

I will add that I mainly commented based on the presentation layer and syntax to reference other resources. Relative to the Dependson capability.

At least on Windows Systems, things should nearly always run in series, however it's important to have the dependency chain. I feel like maintaining the current dsc v1 behavior would be the safest way to not introduce unexpected collision between resources.

It's just having to use resourceid() method directly that makes me feel like I am time traveling back in time 10 years.

Unless there is a definite plan to generate those reference Id's automatically with another layer on top for abstraction, then I don't see this as an inviting language to use. And is bordering on imperative in nature.

Yes I keep referencing Bicep, based on the clarity and concise nature of the language through the use of Symbols that compile to resource id's etc.

So for me it's more about knowing if this is an intermediate language, or what the authoring, debugging, linting will be like. Not specific to parallel verses serial execution by default.

Docs for reference to 'symbolic-names' examples.

• https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/resource-declaration?tabs=azure-powershell#declaration

Plus bicep explicit and implicit dependency.

• https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/resource-dependencies

Both of above cross reference using symbolic-names. Both have linting etc.

So I realize this is off topic for this thread, however this is significant for me to say, early on in the development lifecycle... Than to wait and see something get created that isn't inviting for me to use to "describe my environments", since at the end the day, that is what I use this for.

Also I know there are layers of development and baby steps, however so long as there is a shared vision or destination, then it makes it worthwhile to spend time to test the early iterations of the product.

Radius have taken to extend Bicep in a way that would exactly meet the need that I am interested for this core tool of DSC to be extended.

• https://github.com/radius-project/radius
• https://github.com/radius-project/bicep

Perhaps there are some learnings/conversations that could be seeded now to ensure that any cross-team requirements could occur up front. 🙏

[michaeltlombardi]

Regarding parallel-by-default: I think this is actually a recipe for disastrously difficult-to-diagnose-and-reproduce bugs. DSC itself can't safely determine whether and which resource instances can be processed in parallel, and any of them that conflict for any reason can lead to unreproducible bugs, race conditions, and other errors.

In my experience both in operations and working as an engineer at a configuration management software vendor, the speed of an individual configuration operation is much less important in the aggregate than reliability, reproducibility, and clarity. I think opting into parallel execution is much safer than defaulting to parallel execution and less surprising to users given how other configuration management software behaves. Parallel-by-default also drastically complicates any rollback-on-failure operation. It introduces more problems than it solves (speed of execution).

Adding parallel-by-default would also invert the existing problem: I would need to specify, somehow, which resources to run in serial and when, even if those resources advertise themselves as being parallel-friendly - resource authors can make mistakes, and I want control over how the node graph is processed. I could solve that by keeping a long chain of dependencies for my resource instances, rather than a serial processing group, but that also complicates and obfuscates the node graph. I wouldn't be able to tell the difference between "run X, then Y, because they might have a race condition or other problematic side effect" and "run X, then Y, because Y depends on the successful configuration of X intrinsically."

[SteveL-MSFT]

I think ARM can get away with parallel-by-default because infrastructure can be deployed in parallel, but node configuration is much more likely to have conflicts on the same system. Also, since ARM resources are written by Microsoft, they can expect this, but DSC resources will come from many different sources and much more likely won't be tested or expected to be run in parallel with another resource or another instance of the same resource.