Restart File

[Peku995]

I have an idea for the upcoming release: At the moment, it is not possible to use a completely finished simulation as a starting point to carry out further iterations from here (at least to my knowledge). At the moment, this is only possible for simulations that are not completely finished. I think it would improve the restart capabilities of Paramonte considerably if you could also start from simulations that have already ended and then simply save the whole thing under a different name.

[shahmoradi]

This very useful idea has also caught my attention in the past.
There are, however, a few ambiguities that need to be resolved for such functionality.
Currently, one can achieve this behavior by manually deleting the *_sample.txt file and increasing the chainSize simulation specification.
But this will continue with the old simulation. Is the goal to create an entirely new simulation that merely starts from the last state of the past simulation? Or is the goal to continue with the old simulation files? The latter would be more interesting because all simulations related to a single problem would be concatenated into a single set of output files. Which one is in your mind?
One way to allow both possibilities is to replace the existing option overwriteRequested with a new specification status that can take multiple values:

1. append: (default behavior) Start a new simulation and attach the new output to the old existing files. If the simulation is incomplete, complete it. If there is no old simulation, create the output files as if it is a new simulation.
2. duplicate: same as append except when the existing simulation is complete, the new simulation will be output to separate new files.
3. replace: same as append except when the existing simulation is complete, the new simulation will start from the last state of the old simulation and the results will completely overwrite (replace) the old files.
4. fresh: same as append except when the existing simulation is complete, the sampler will overwrite the existing files without reling on any information from the past simulation as if the old simulation and its files did not exist.
5. new: same as append except when the existing simulation is complete, the sampler will raise an error because the files must not exist as it is a new simulation.
   The last two are the existing behavior dictated by overwriteRequested being True or False, respectively.

[Peku995]

I had imagined it like duplicate, where the new output file should then also contain the data of the old simulation and the new simulation simply continues where the old one left off (I think that's how duplicate was meant). I think it would be handy if you could then specify the name of the old simulation as the start file and directly a new name for the new simulation.
In general, I think all the suggested specifications are useful and would improve Paramonte!

[shahmoradi]

This is doable. Can you clarify why you'd want a copy of the old simulation in the new files? Unless there is a good reason, such duplication behavior can quickly exhaust all computer memory by arithmetically increasing the output files sizes from one simulation to the next.

[Peku995]

I would like to have it like that because I think it is much easier to work with that, otherwise I always have to import multiple files if i would like to see all iterations at once.
Also not all problems are that huge in terms of memory that this behaviour would ne a problem, so I think the decision might be up to the user if he or she would like to have that behaviour.

[arwelHughes]

I also think this would be very useful! A particular use-case would be if a simulation has not been given sufficient iterations to fully converge. Then, rather than repeating the whole thing with more iterations, it should be possible to just continue from the point where the last one ended, as if a complete longer simulations had been done (Presumably, this would need some information from the end of the previous chain and the covariance matrix in whatever form it was in when the last simulation ended?)

[shahmoradi]

This functionality is much needed. I am glad it was brought up in this discussion.
Aside from simulation continuation, a more systematic approach to ending a simulation is needed. This is doable at the cost of a small performance penalty.