Elaborate NS run using a loop #644

fnovak42 · 2023-06-07T09:36:32Z

Use a loop to run a nested sampler internally as outlined here: https://dynesty.readthedocs.io/en/stable/dynamic.html
This is related to one or several points from issue #573.

dvandyk · 2023-06-07T11:47:09Z

python/eos/analysis.py

-        sampler.run_nested(dlogz_init=dlogz, maxiter=maxiter, print_progress=print_progress)
+        #sampler.run_nested(dlogz_init=dlogz, maxiter=maxiter)
+
+        for results in sampler.sample_initial(dlogz=dlogz, maxiter=maxiter):


I would enumerate this and run a log message every few seconds/every few samples. @mreboud ?

Every few samples is probably simpler to implement.

dvandyk · 2023-06-26T09:05:49Z

python/eos/analysis.py

@@ -607,7 +607,7 @@ def _prior_transform(self, u):
        return self._u_to_par(u)


-    def sample_nested(self, bound='multi', nlive=250, dlogz=1.0, maxiter=None, seed=10, print_progress=True):
+    def sample_nested(self, bound='multi', nlive=250, dlogz=1.0, maxiter=None, seed=10, print_progress=True, save_intermediate=False, base_directory='./', posterior=None):


That's a reasonable idea to begin with, but I do not like it very much. So far, the design was to separate the tasks from the lower-level operations. I think we should and can keep doing that.

I would suggest to use a yield_intermediate keyword argument. Instead of using return sampler.results once at the end, we rather yield sampler.results periodically. The task can then run a for loop over this function and save the results itself.

fnovak42 · 2023-07-17T19:34:06Z

The main change in the implementation right now is that the intermediate results are stored every few loop, specified by checkpoint_interval. I don't think that storing them every few samples would be practical, since the results of several batches (with differing numbers of samples) are combined within an iteration. It would be easier, and perhaps more useful, to save the results every few seconds using dynesty.utils.DelayTimer -- this is also used in dynesty for the checkpoint_every parameter in run_nested.

fnovak42 · 2023-09-05T09:49:35Z

To tick some of the boxes in issue #573 off:

the log file now contains the messages generated by eos.info and is placed in the nested directory as discussed. So I would tick the third box off
the fourth point is also done; we've replaced the loop
the log file will now tell you if sampling was stopped due the maximum number of iterations or the dlogz limit being reached

Regarding the last point, currently we use the same variables maxiter and dlogz in the initial sampling (line 641 in analysis.py) and the batches (line 667) as well as the overall loop (lines 656, 661). Is it feasible to use different variables for those or should they adhere to the same limit?

I also thought about adding something like this if maxiter is not given:

import sys
if maxiter is None: maxiter = sys.maxsize

dvandyk

Looks very reasonable! Some comments inline.

dvandyk · 2023-10-04T10:37:11Z

python/eos/analysis.py

            for p in self.varied_parameters:
-                eos.error(' - {n}: {v}'.format(n=p.name(), v=p.evaluate()))
+                eos.error(f' - {p.name()}: {p.evaluate()}')


All of the changes above are due to pyupgrade and should be moved to a separate commit.

dvandyk · 2023-10-04T10:37:50Z

python/eos/analysis.py

@@ -653,14 +653,55 @@ def sample_nested(self, bound='multi', nlive=250, dlogz=1.0, maxiter=None, seed=
        :type maxiter: int, optional
        :param seed: The seed used to initialize the Mersenne Twister pseudo-random number generator.
        :type seed: {None, int, array_like[ints], SeedSequence}, optional
+        :param save_intermediate: If set to True, the intemediate dynesty sampler results are stored in each loop iteration.
+        :type save_intermediate: bool, optional
+        :param checkpoint_every: The number of seconds between checkpoints at which the intermediate dynesty sampler results are stored.


I like the checkpointing. Not so sure that I like the naming.

What about backup and backup_frequency?

How about just a checkpoint_interval (float, larger zero)? If None, no intermediate results are saved.

dvandyk · 2023-10-04T10:41:22Z

python/eos/analysis.py

        sampler = dynesty.DynamicNestedSampler(self.log_likelihood, self._prior_transform, len(self.varied_parameters), bound=bound, nlive=nlive, rstate = np.random.Generator(np.random.MT19937(seed)))
-        sampler.run_nested(dlogz_init=dlogz, maxiter=maxiter, print_progress=print_progress)
-        return sampler.results
+        #sampler.run_nested(dlogz_init=dlogz, maxiter=maxiter)


This comment needs to be removed.

dvandyk · 2023-10-04T10:43:47Z

python/eos/tasks.py

+        posterior_values = results.logwt - results.logz[-1]
+        weights = _np.exp(posterior_values)
+        eos.data.DynestyResults.create(os.path.join(base_directory, posterior, f'dynesty_results-{iter:04}'), analysis.varied_parameters, results)
+        eos.data.ImportanceSamples.create(os.path.join(base_directory, posterior, f'samples-{iter:04}'), analysis.varied_parameters,


I like the labelling of the dynesty result by the iteration number. However, doing so for the samples will break backward compatibility with e.g. the plotting framework. Please undo this labelling for samples.

fnovak42 · 2023-11-06T18:34:22Z

For comparison I calculated the chi**2 values from test_multivariate_priors_1 in python/eos/analysis_TEST.py. The values using the code from the main branch (unrolled loop) are:

chi 1: 0.002682592060968833
chi 2: 5.803923701460194e-07
chi 3: 2.52241831039844e-05
chi 4: 0.2764884467520001

and using this branch:

chi 1: 0.017353753382656843
chi 2: 0.00031026173335609383
chi 3: 0.0009632358001228368
chi 4: 0.5324848524622118

I haven't yet spotted a significant difference to the dynesty version that would tell me where the difference comes from: https://github.com/joshspeagle/dynesty/blob/f59d963fad80301e5d28bc5a6e3718c467c1bf94/py/dynesty/dynamicsampler.py#L1824

mreboud

Can you clean up line 176 of analysis_TEST.py? There is a missing sqrt, but sigma is not used anyway ...

dvandyk reviewed Jun 7, 2023

View reviewed changes

fnovak42 force-pushed the ns-loop branch from b32e4ac to f027887 Compare June 19, 2023 20:04

dvandyk reviewed Jun 26, 2023

View reviewed changes

fnovak42 force-pushed the ns-loop branch from f027887 to 360ae24 Compare July 3, 2023 20:07

fnovak42 force-pushed the ns-loop branch from 360ae24 to 7d68799 Compare July 17, 2023 19:26

fnovak42 force-pushed the ns-loop branch from 7d68799 to 4028a30 Compare August 28, 2023 20:32

fnovak42 force-pushed the ns-loop branch from e0765dc to 94da4c4 Compare October 4, 2023 08:47

dvandyk requested changes Oct 4, 2023

View reviewed changes

fnovak42 force-pushed the ns-loop branch from 94da4c4 to bd18a7e Compare October 23, 2023 19:47

fnovak42 added 4 commits October 24, 2023 11:20

[python] Run pyupgrade on tasks.py

2ec51b0

[python] Run pyupgrade on analysis.py

c84f637

[python] Modify NS task to save results of multiple iterations

1f49a86

[python] Unrolled NS loop using generators

6ed6e3b

fnovak42 force-pushed the ns-loop branch from bd18a7e to 6ed6e3b Compare October 24, 2023 09:22

mreboud reviewed Nov 22, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elaborate NS run using a loop #644

Elaborate NS run using a loop #644

fnovak42 commented Jun 7, 2023

dvandyk Jun 7, 2023

mreboud Jun 20, 2023

dvandyk Jun 26, 2023

fnovak42 commented Jul 17, 2023

fnovak42 commented Sep 5, 2023

dvandyk left a comment

dvandyk Oct 4, 2023

dvandyk Oct 4, 2023

mreboud Oct 4, 2023

pluegh Oct 4, 2023

dvandyk Oct 4, 2023

dvandyk Oct 4, 2023

fnovak42 commented Nov 6, 2023

mreboud left a comment

Elaborate NS run using a loop #644

Are you sure you want to change the base?

Elaborate NS run using a loop #644

Conversation

fnovak42 commented Jun 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fnovak42 commented Jul 17, 2023

fnovak42 commented Sep 5, 2023

dvandyk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fnovak42 commented Nov 6, 2023

mreboud left a comment

Choose a reason for hiding this comment