Workflow fails when using withParam field with Inline templates

[yogendra-mck]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened/what did you expect to happen?

I am currently utilizing inline templates in my workflow configuration. I’ve encountered an issue when using the withParam field with a task. The subsequent task fails to start with below error:

level=info msg="Updated message  -> error in entry template execution: failed to get a template" namespace=default

In the provided image, you can see that task3 fails with the error I mentioned earlier.

Please take a look at it, as I need to use inline templates. Since I'm generating the workflow configuration dynamically, it makes more sense to stick with inline templates.

Version

v3.5.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: test-fan-out
  namespace: default
  labels:
    example: 'true'
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: task1
            inline:
              container:
                image: docker/whalesay:latest
                command:
                  - cowsay
                args:
                  - test template
          - name: task2
            withParam: '["test", "list"]'
            dependencies:
              - task1
            inline:
              container:
                image: docker/whalesay:latest
                command:
                  - cowsay
                args:
                  - test template
          - name: task3
            dependencies:
              - task2
            inline:
              container:
                image: docker/whalesay:latest
                command:
                  - cowsay
                args:
                  - test template

Logs from the workflow controller

2024-04-29 18:33:26 time="2024-04-29T13:03:26.546Z" level=info msg="Processing workflow" Phase= ResourceVersion=2916428 namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.550Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.550Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
2024-04-29 18:33:26 time="2024-04-29T13:03:26.550Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.550Z" level=info msg="Updated phase  -> Running" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.550Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=info msg="DAG node test-fan-out-r6xsm initialized Running" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1129662630, taskName task2"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1112885011, taskName task1"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1112885011, taskName task1"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=info msg="All of node test-fan-out-r6xsm.task1 dependencies [] completed" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.551Z" level=info msg="Pod node test-fan-out-r6xsm-1112885011 initialized Pending" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.560Z" level=info msg="Created pod: test-fan-out-r6xsm.task1 (test-fan-out-r6xsm--1112885011)" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.560Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1129662630, taskName task2"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.560Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.560Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1129662630, taskName task2"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.560Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:26 time="2024-04-29T13:03:26.560Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.560Z" level=info msg=reconcileAgentPod namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:26 time="2024-04-29T13:03:26.560Z" level=info msg="Workflow to be dehydrated" Workflow Size=1776
2024-04-29 18:33:26 time="2024-04-29T13:03:26.570Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=2916432 workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.561Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2916432 namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.561Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.561Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=test-fan-out-r6xsm-1112885011 old.message= old.phase=Pending old.progress=0/1 workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.561Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:36 time="2024-04-29T13:03:36.561Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1129662630, taskName task2"
2024-04-29 18:33:36 time="2024-04-29T13:03:36.561Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1129662630, taskName task2"
2024-04-29 18:33:36 time="2024-04-29T13:03:36.562Z" level=info msg="TaskGroup node test-fan-out-r6xsm-1129662630 initialized Running (message: )" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.562Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-3080733731, taskName task2(0:test)"
2024-04-29 18:33:36 time="2024-04-29T13:03:36.562Z" level=info msg="All of node test-fan-out-r6xsm.task2(0:test) dependencies [task1] completed" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.562Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.562Z" level=info msg="Pod node test-fan-out-r6xsm-3080733731 initialized Pending" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.566Z" level=info msg="Created pod: test-fan-out-r6xsm.task2(0:test) (test-fan-out-r6xsm--3080733731)" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.566Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-2696649548, taskName task2(1:list)"
2024-04-29 18:33:36 time="2024-04-29T13:03:36.566Z" level=info msg="All of node test-fan-out-r6xsm.task2(1:list) dependencies [task1] completed" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.566Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.566Z" level=info msg="Pod node test-fan-out-r6xsm-2696649548 initialized Pending" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.572Z" level=info msg="Created pod: test-fan-out-r6xsm.task2(1:list) (test-fan-out-r6xsm--2696649548)" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.572Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:36 time="2024-04-29T13:03:36.572Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:36 time="2024-04-29T13:03:36.572Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.572Z" level=info msg=reconcileAgentPod namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:36 time="2024-04-29T13:03:36.572Z" level=info msg="Workflow to be dehydrated" Workflow Size=2773
2024-04-29 18:33:36 time="2024-04-29T13:03:36.577Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=2916475 workflow=test-fan-out-r6xsm
2024-04-29 18:33:41 time="2024-04-29T13:03:41.579Z" level=info msg="cleaning up pod" action=deletePod key=default/test-fan-out-r6xsm--1112885011/deletePod
2024-04-29 18:33:46 time="2024-04-29T13:03:46.568Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2916475 namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:46 time="2024-04-29T13:03:46.568Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=test-fan-out-r6xsm
2024-04-29 18:33:46 time="2024-04-29T13:03:46.568Z" level=info msg="node changed" namespace=default new.message= new.phase=Running new.progress=0/1 nodeID=test-fan-out-r6xsm-2696649548 old.message= old.phase=Pending old.progress=0/1 workflow=test-fan-out-r6xsm
2024-04-29 18:33:46 time="2024-04-29T13:03:46.568Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=test-fan-out-r6xsm-3080733731 old.message= old.phase=Pending old.progress=0/1 workflow=test-fan-out-r6xsm
2024-04-29 18:33:46 time="2024-04-29T13:03:46.569Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:46 time="2024-04-29T13:03:46.569Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:46 time="2024-04-29T13:03:46.569Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:46 time="2024-04-29T13:03:46.569Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:46 time="2024-04-29T13:03:46.569Z" level=info msg=reconcileAgentPod namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:46 time="2024-04-29T13:03:46.569Z" level=info msg="Workflow to be dehydrated" Workflow Size=2843
2024-04-29 18:33:46 time="2024-04-29T13:03:46.575Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=2916530 workflow=test-fan-out-r6xsm
2024-04-29 18:33:51 time="2024-04-29T13:03:51.576Z" level=info msg="cleaning up pod" action=deletePod key=default/test-fan-out-r6xsm--3080733731/deletePod
2024-04-29 18:33:56 W0429 13:03:56.926722       1 listers.go:79] can not retrieve list of objects using index : Index with name namespace does not exist
2024-04-29 18:33:57 time="2024-04-29T13:03:57.457Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=2916530 namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.457Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.457Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=test-fan-out-r6xsm-2696649548 old.message= old.phase=Running old.progress=0/1 workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.457Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="node test-fan-out-r6xsm-1129662630 phase Running -> Succeeded" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="node test-fan-out-r6xsm-1129662630 finished: 2024-04-29 13:03:57.458103339 +0000 UTC" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=warning msg="was unable to obtain the node for test-fan-out-r6xsm-1146440249, taskName task3"
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="Skipped node test-fan-out-r6xsm-1146440249 initialized Error (message: failed to get a template)" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=error msg="Mark error node" error="failed to get a template" namespace=default nodeName=test-fan-out-r6xsm workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="node test-fan-out-r6xsm phase Running -> Error" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="node test-fan-out-r6xsm message: failed to get a template" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="node test-fan-out-r6xsm finished: 2024-04-29 13:03:57.458224005 +0000 UTC" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=error msg="Mark error node" error="failed to get a template" namespace=default nodeName=test-fan-out-r6xsm workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=error msg="error in entry template execution" error="failed to get a template" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=warning msg="Non-transient error: failed to get a template"
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="Updated message  -> error in entry template execution: failed to get a template" namespace=default workflow=test-fan-out-r6xsm
2024-04-29 18:33:57 time="2024-04-29T13:03:57.458Z" level=info msg="Marking workflow completed" namespace=default workflow=test-fan-out-r6xsm

Logs from in your workflow's wait container

2024-04-29 18:33:38 time="2024-04-29T13:03:38.004Z" level=info msg="Starting Workflow Executor" version=v3.4.8
2024-04-29 18:33:38 time="2024-04-29T13:03:38.005Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
2024-04-29 18:33:38 time="2024-04-29T13:03:38.005Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=test-fan-out-r6xsm--3080733731 template="{\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay:latest\",\"command\":[\"cowsay\"],\"args\":[\"test template\"],\"resources\":{}}}" version="&Version{Version:v3.4.8,BuildDate:2023-05-25T22:21:53Z,GitCommit:9e27baee4b3be78bb662ffa5e3a06f8a6c28fb53,GitTag:v3.4.8,GitTreeState:clean,GoVersion:go1.20.4,Compiler:gc,Platform:linux/arm64,}"
2024-04-29 18:33:38 time="2024-04-29T13:03:38.005Z" level=info msg="Starting deadline monitor"
2024-04-29 18:33:43 time="2024-04-29T13:03:43.009Z" level=info msg="Main container completed" error="<nil>"
2024-04-29 18:33:43 time="2024-04-29T13:03:43.009Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
2024-04-29 18:33:43 time="2024-04-29T13:03:43.009Z" level=info msg="No output parameters"
2024-04-29 18:33:43 time="2024-04-29T13:03:43.009Z" level=info msg="No output artifacts"
2024-04-29 18:33:43 time="2024-04-29T13:03:43.010Z" level=info msg="Alloc=8529 TotalAlloc=15949 Sys=24173 NumGC=4 Goroutines=7"
2024-04-29 18:33:43 time="2024-04-29T13:03:43.010Z" level=info msg="Deadline monitor stopped"

[shuangkun]

Reproduced it and will take a look.

[agilgur5]

Follow-up from this Slack thread.

Per there, workaround would be to use named templates

Please take a look at it, as I need to use inline templates. Since I'm generating the workflow configuration dynamically, it makes more sense to stick with inline templates.

You can dynamically generate names for the templates. We may do something similar in the implementation of #12694