Merging the dev-bbob-noisy to the development branch #2233
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added the f_sphere_gaussian.c file, which implements the sphere function with gaussian noise (and customable noise variance, for either moderate level or strong level of noise). The file contains the implementations of f_sphere_gaussian_raw, f_sphere_gaussian_evaluate, f_sphere_gaussian_evaluate_gradient,f_sphere_gaussian_allocate and f_sphere_gaussian_bbob_problem_allocate. In order to implement these functions, we slightly modified the coco_problem_s structure and added three new attributes: random_seed (uint_32, the random seed for generating samples from given distributions), distribution_theta (*double, the parameters of the distributions from which the noise is sampled) and last_noise_value(double, the noise value obtained from the last function evaluation. Used to compute the gradient at a given point with the same noise as the function evaluation it follows. The correct functioning of this is based on the assumption that the gradient is computed after each function evaluation). We also added the coco_noisy_problem_allocate_from_scalars function in the coco_problem.c file, which is basically a wrapper around the coco_problem_allocate_from_scalars function, which also allocates the random_seed and distribution_theta arguments mentioned before.
Added the f_sphere_gaussian.c file, which implements the sphere function with gaussian noise (and customable noise variance, for either moderate level or strong level of noise). The file contains the implementations of f_sphere_gaussian_raw, f_sphere_gaussian_evaluate, f_sphere_gaussian_evaluate_gradient,f_sphere_gaussian_allocate and f_sphere_gaussian_bbob_problem_allocate. In order to implement these functions, we slightly modified the coco_problem_s structure and added three new attributes: random_seed (uint_32, the random seed for generating samples from given distributions), distribution_theta (*double, the parameters of the distributions from which the noise is sampled) and last_noise_value(double, the noise value obtained from the last function evaluation. Used to compute the gradient at a given point with the same noise as the function evaluation it follows. The correct functioning of this is based on the assumption that the gradient is computed after each function evaluation). We also added the coco_noisy_problem_allocate_from_scalars function in the coco_problem.c file, which is basically a wrapper around the coco_problem_allocate_from_scalars function, which also allocates the random_seed and distribution_theta arguments mentioned before. EDIT: Added some missing semi-columns that I forgot around...
…_noise_value and added coco_problem_sample_gaussian function to generate normal noises. * ```/coco/code-experiments/src/coco.h``` * ```/coco/code-experiments/src/coco_problem.c``` * ```/coco/code-experiments/src/f_sphere_gaussian.c``` Added all the new getters for the new `coco_problem_s` members (more specifically `coco_problem_get_random_seed`, `coco_problem_get_distribution_theta`, `coco_problem_get_last_noise_value`). Their signat>
…he sphere with uniform noise
The approach previously undertaken is highly unpractical and unfeasible. So I decided to change the approach and create a wrap around the standard problems and use it to implement the bbob-noisy test suite.
The present commit implements the wrappers around the f_<function_name>_evaluate, f_<function_name>__allocate and f_<function_name>_bbob_problem_allocate functions, needed to implement a noisy version of the given problem. In particular, this is handled by the three functions: * coco_problem_f_evaluate_wrap_noisy * coco_problem_allocate_f_wrap_noisy * coco_problem_allocate_bbob_wrap_noisy The definition of the just mentioned functions is written in the suite_bbob_noisy_uitilities file, while their declaration is in the coco.h header file. In order to make this idea of wrapping work, we defined the following structures * coco_noise_model_s * coco_noisy_problem_s The coco_noise_model_s is used to evaluate the noise model according to the paper <paper_name>. The coco_noisy_problem_s is a structure used as a wrapper around coco_problem_t (which is one of its member attributes, inner_problem), and is (should be) able to implement the noisy version of a given problem in the bbob test suite. The preceding approach of defining the noisy version of each function is, as we mentioned in the previous commit message, unfeasible. On the other hand, for respecting the project structure in terms of protection and isolation of certain files, we found the need to define, finally, the three functions used to allocate the noisy version of the problems in the bbob datasuite, according to the three aforementioned noise models. These functions are: * coco_problem_allocate_bbob_wrap_noisy_gaussian * coco_problem_allocate_bbob_wrap_noisy_uniform * coco_problem_allocate_bbob_wrap_noisy_cauchy
… bbob_noisy.c file containing the definition of the functions needed to create and allocate the bbob-noisy test suite
This is due to the fact that the functions f_schaffers_bbob_problem_allocate and f_gallagher_bbob_problem_allocate have a different signature than the other f_<function_name>_bbob_problem_allocate ones. For how I was thinking of implementing the noisy suite, as a wrap around the existing bbob suite, this fact created the need to define specific wrappers (and function types) for these two mentioned functions. In addition, we added some other function type in coco_internal.h for minimizing the number of unused parameters in the overall project
Apparently it kinda works now, it's time to do the needed tests
Generated data for regression test for the BBOB NOISY datasuite
Done the necessary modifications needed to be able to test the new implementation of the bbob-noisy test suite against the old code. Running now the full checks. I already checked if the fopt and the xopt are the same across the two implementations, and this seems to be the case. Writing now the scripts to be able to reproduce the results. One thing I observed regarding the fact that these tests were not passing at all, is that I realized that the function id needs to exactly match across the two implementations for the xopt and yopt to be computed in the same way. This is what I did and, as expected, it corrected this behaviour.
I am currently proceeding with the regression tests. I tested the new implementation (in the noise free version), against the legacy code (by taking the ftrue values instead of the fval ones, noise free again) The test consisted in randomly sampling a set of x coordinates from the legacy code and evaluate it both on the old and the new implementation. Before doing so I tested all the optimal values, which worked (the values are the same as for the previous implementation). For what concerns the results of the regression tests (still on the noise free version), all the tests passed except for F116 F117 F118 (ellipsoid) and F126 F127 F128 (griewank-rosenborg). This is probably due to some differences in the inner implementation of the functions across the two versions of the coco platform, but I am still trying to figure it out... The next step (once all test will pass for the noise-free versions), will be to check if the noisy version of the suite yields the same results as well, but now Im currently facing some problems, as the distributions parameters for sampling the noise are not correctly allocated (or correctly passed to the sampling function), but I dont know for sure yet.
Maybe I should use the rotated version, as the legACY code computes a rotation on it
Basically there were some parameters of the objective functions that were passed as variables or literals and were supposed to be different across the noisy and the noise free implementation, according to the legacy code Example.: Default conditioning for ellipsoid was 1.0e6, which is correct for the noise free functions but not for the noisy version, since for the legacy code the conditioning of noisy problem was just 1.0e4. There was a similar issue with the Griewank-Rosenbrock function. This modifications required slightly ones also in the other suites calling the allocators for those functions (to be able to pass them the correct value of the argument). Hopefully, these modifications should not alterate the already existing behaviour of the other suites, but we'll see when the tests done at commit time are done.
Passed the regression test against the legacy code and generated regression test data 1) Cleaning the code further. 1.1) Removed all the declarations that didn't need to be public from the `coco.h` file 1.2) Now the random seed is handles as a global variable (as in the legacy code), so there's no need to pass it as argument for the bbob_noisy problem allocator or as attribute of the noise model 1.3) Passing the noise model as argument to the constructor so that we'll only have one for the three different noise models 2) Modifying the test. The randomness of the bbob noisy test suite implied the necessity to think about some sort of reproducibility between the runs. Currently, the structure of the code emulates the legacy code-base, with the random seed defined as a global variable that is increased each time a noise term is sampled. Thus, in order to obtain the same results over different runs, the order of evaluation must remain unaltered. As the regression data didn't previously store any information about the order of evaluation (only for the functions, not for the x datapoints), this created the need to modify the way the regression data was created, in order to store information about the order of eevaluation of the datapoints. I then uploaded, in the modified format, the generated data (only for the bbob-noisy test suite) to the regression-data github repository of the numbbo organization. This required some modifications for the script actually running the test, but, if I didn't make any mistake, it should work as well for the older format.
I added the interface for getting the last noise value for debugging, but removed such function from the source code once not needed, but forgot to remove it from the inetrface
…rc code by writing functions for allocating the distributon theta array for each distribution
…commit where the results obtained are exactly the same as the one for the legacy code. Before I forgot to implement the boundary handling method that were adding a penalty on the decision vector.
…esults of the tests done at commit time
FMGS666
merged commit 9cfbf2d
into
merge-development-dev-bbob-noisy
23 of 25 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I created this third "instrumental" branch for safely merging the
dev-bbob-noisybranch into thedevelopmentone.There are some merge conflicts, so I thought that it would have been better to fix them in another branch before the final merge. This is probably because I am not really good with git, probably there are other ways to do this...
Thank you and good evening