[WIP] Attempt to use single submit file template with dag

[jrbourbeau]

Want to see if it's possible to use a single Job submit file when using a Dagman

[ChromeHearts]

@jrbourbeau How is the progress of this PR? We are interested using pycondor to build a dag with over hundreds of thousands nodes. Many nodes will have the same executable with different arguments. So it makes sense to generate one submit file and specify different arguments in the dagman submit file.
We are happy to help implement this feature. I wonder if you can shed some lights where we should make changes in.

[jrbourbeau]

@ChromeHearts Thanks for asking about this, I really appreciate your interest!

Unfortunately I've been distracted with other things and haven't worked seriously on this PR in a couple months. However, as you've pointed out, I think having a single job submit file could be really useful when a dagman has many nodes.

We are happy to help implement this feature. I wonder if you can shed some lights where we should make changes in.

That's great! For now I'd like to have a separate dagman object, perhaps a TemplatedDagman (suggestions for a better name are welcome : ) ) to avoid breaking changes for users who might rely on having a submit file for each Job. The TemplatedDagman.build() method needs to create a single Job submit file with variables for each Job attribute. For example, something like

# Job submit file
executable = $(executable)
arguments = $(arguments)
request_memory = $(request_memory)
request_disk = $(request_disk)
...

and then create a dagman submit file that fills in those variable values for each Job it is managing.

# Dagman submit file
JOB job1_name /path/to/job_template.submit
VARS job1_name executable="my_script.py"
VARS job1_name request_memory="3GB"
JOB job2_name /path/to/job_template.submit
VARS job2_name executable="my_other_script.py"
VARS job2_name arguments="--some-option"

The construction of the Dagman submit file lines happens here

https://github.com/jrbourbeau/pycondor/blob/80f8a6cf5f60397c0d3766283c34464bae2f8c5f/pycondor/dagman.py#L306-L322

Specifically the _get_job_arg_lines method would need to be changed for the TemplatedDagman.

[ChromeHearts]

@jrbourbeau Thanks for your prompt reply and help!

I started from your branch and found it's needed one change to fully work. When a job had no arguments, no VAR_ATTR would be generated. I made the code changes. Then I was able to generate a DAG with 20k+ individual jobs in a single submit file and run successfully in condor.

Of course, it needs to be backward compatible. Another way to do this is to take another argument in build, eg. dagman.build(single_file=True) or dagman.build(file_mode=SINGLE). This argument will eventually transverse to _get_job_arg_lines(). I prefer this way because there will be less client APIs (less documentation)

[jrbourbeau]

Thanks for your work on this @ChromeHearts! Would you like to open up a new PR with these changes, or are you okay with me adding them to this existing PR?

Another way to do this is to take another argument in build, eg. dagman.build(single_file=True) or dagman.build(file_mode=SINGLE)... I prefer this way because there will be less client APIs (less documentation)

Good point. I like that way too : )