Setting final outputs on active tasks: should kill first

[hjoliver]

The new(ish) Cylc 8 interventions proposal allows:

• setting a final output on task to about a retry chain
  • (note this can always be done on a waiting task, between retries)
• setting a final output on an active task to abandon a "stuck job submission"
  • (presumably the job platform or job runner has gone AWOL)

However, the proposal neglects to cover setting final outputs on active tasks that are not "stuck".

Hence, #6025 (don't allow automatically set (i.e., clock) expire on an active task); and this issue.

If an active task is not "stuck" we should require the user to kill the job before setting a final output, because orphaned jobs are:

• confusing to users who look at their batch queues
• dangerous - if the task gets retriggered you end up with duplicate jobs

We can easily avoid this by simply not allowing manual setting of final outputs on active tasks.

If a user tries to do it, refer them to cylc kill. Note #5981 could handle both normal and "stuck" tasks: if the job kill fails, set the requested output anyway.

[oliver-sanders]

Considerations:

• The task might not be killable.
• The user may or may not want to try to kill the task.

Possible options to consider, when setting a final output on an active task:

1. Try to kill by default, if the kill fails, continue anyway.
2. Kill by default, if the kill fails, reject the set. Provide a --no-kill option.
3. Don't kill by default, if the kill fails, reject the set. Provide a --kill option.
4. Other?

[hjoliver]

Yes I addressed Consideration 1. above:

Note #5981 could handle both normal and "stuck" tasks: if the job kill fails, set the requested output anyway.

Consideration 2. is interesting - can you think of a valid reason for taking that view?

Note I'm not suggesting we extend cylc set to do the (attempted) job kill - see #5981

[hjoliver]

Ah, I wonder if I misinterpreted "stuck job submission" in the proposal.

I took it to mean a task that is "stuck" as submitted or running, because the the job was hard-killed, or the host or job-runner down, or network issues.

I suppose it could also mean job stuck in a batch queue indefinitely? (or seemingly indefinitely). Mind you, that's killable. I can see no good reason to just orphan it, and risk a duplicate.

[oliver-sanders]

To the best of my memory:

• Jobs which poll as submitted but will never run (e.g. "held" in PBS?) but cannot be removed by the user. We've had some strange issues of this ilk.
• Jobs which cannot be either polled or killed (e.g. network issues).
• Jobs the batch system reincarnated?