Can we let --playlist be used with more than one APSIM file?

[BrianCollinss]

Describe the new feature

Can we let --playlist be used with more than one APSIM file?

[ric394]

@BrianCollinss if this functionality was to be created, how would you like it to function?

1. One way would be to run a matching playlist in every apsim file referenced in the command.
2. Another way would be to have the --playlist option take multiple playlist name values, where any matching playlist in any apsim file is run.

I'm wondering what seems better to you?

[BrianCollinss]

I think the second option is too complicated. Personally I think the first option is sufficient.

[hol353]

Before we implement anything, what is the use case here?

[BrianCollinss]

In order to avoid ending up with huge DB files (which significantly increase the running time of the sims on HPC), I divide sims into more than 1 apsim files. I then use a similar approach to what Bundle.exe does in classic to extract simulations and create the PBS script with multiple jobs. This requires running multiple apsimx files that have Playlists of the same name. Currently what I have to do is to have one singularity exec command for each file.

[hol353]

In order to avoid ending up with huge DB files (which significantly increase the running time of the sims on HPC), I divide sims into more than 1 apsim files

I do this as well (not on HPC though)

I then use a similar approach to what Bundle.exe does in classic to extract simulations and create the PBS script with multiple jobs.

I'm not familiar with Bundle.exe, nor PBS scripts.

This requires running multiple apsimx files that have Playlists of the same name. Currently what I have to do is to have one singularity exec command for each file.

I'm not sure what you're getting at. Here is what I do:

I break up the job (milllions of simulations) into a collection (1000s) of .apsimx files. Each of these .apsimx files are either a collection of simulations or an experiment node that runs a factorial of simulations. I then pass each .apsimx file to a compute node and tell it to run it (e.g. dotnet Models.dll --csv Task00001.apsimx). I use the csv switch to tell apsim to write outputs to a csv file. I then grab the .csv and store on shared storage, appending the contents of the csv file to the existing contents from other compute nodes. Once the job is finished, I have a single .csv file that is ready for analysis. I have avoided the .db file completely.

I don't use playlists and I don't understand why you need them.

[BrianCollinss]

That's OK. I'll manage it some other way.