-
Notifications
You must be signed in to change notification settings - Fork 63
Benchmarks
CCL has automated tests in place which compare CCL outputs to a set of benchmarks. These are documented in the CCL note. Here, we list the benchmarks used and the contacts responsible for producing them. All of the benchmark codes have been made available through the CCL repository in the folder: CCL/benchmarks/data/codes. A README in that same directory gives further details.
Benchmarks for the following quantities. The final files used specifically in the CCL checks were produced by David Alonso (@damonge). The codes used for producing these files are released with the CCL repository as indicated below.
a) Growth factor at z = 0,1,2,3,4,5. growth_lowz_bm.py
b) Comoving radial distance [Mpc/h] at z = 0,1,2,3,4,5, distances_bm.py
c) Comoving radial distance [Mpc/h] and growth factor at z=10, 20, 50, 100, 200, 500, 1000. distances_hiz_bm.py and growth_hiz_bm.py
d) Linear matter power spectrum (from BBKS) [(Mpc/h)^3] at z = 0,2; 1.e-3 h/Mpc <= k <= 10, 10 bins/decade. bbks_bm.py
e) \sigma(M,z=0) sigmaM_bm.py
f) Projected galaxy clustering tomography power spectra, density term only (no magnification, RSD, etc.) with non-evolving linear bias b(z) = 1, 10 < l < 10000. Folder cl_corr_bm.
g) Angular convergence tomography power spectra, leading order convergence term only (no magnification), 10 < l < 10000. Folder cl_corr_bm.
h) Angular galaxy clustering tomography correlation function, 0.01 deg < theta < 5 deg, 5 bins/decade. Folder cl_corr_bm.
i) Angular shear tomography correlation functions (\xi_+,\xi_-), 0.01 deg < theta < 5 deg, 5 bins/decade. Folder cl_corr_bm.
All of these originated from a code-comparison project in which several members of TJP collaborated, comparing several codes. The private webpage that gives further details can be accessed by DESC members here. For f)-i), both analytic and binned redshift distributions were used.
The benchmarks for distances and growth function for the whole redshift range from 0.01 to 1000 and for cosmologies with massive neutrinos were produced by Tilman Tröster (@tilmantroester):
- Growth function:
growth_allz.py - Comoving radial distances and distance moduli, including models with massive neutrinos:
create_CLASS_distance_benchmarks.ipynb
Independent distance benchmarks using astropy for cosmologies with massive neutrinos were created by Danielle Leonard (@c-d-leonard): multiple_neutrino_distances.ipynb
These benchmarks require relaxing the tolerance to 1e-3 because astropy relies on an approximation for the neutrino phase-space integral. The CLASS benchmarks do not suffer from this approximation and are used for the plots in the paper.
Produced by Sukhdeep Singh (@sukhdeep1989): cl_cmbl_bm.py
Produced by David Alonso (@damonge): bcm_bm.c
The automated test for power spectrum emulation within CCL compares the smoothed simulated power spectra provided by the paper authors (Lawrence et al. ,2017, arXiv version) to the CCL output of the power spectrum via the emulator.
There are two tests. The first, for cosmologies without massive neutrinos, corresponds to Figure 6 of the emulator paper, for a specific subset of the cosmologies: {1,3,5,6,8,10}. Other cosmologies are not allowed because CLASS fails when w(z) crosses -1 and we need the linear power spectrum from CLASS in general for sigma_8 computation.
In addition, the Cosmic Emulator also produces power spectra for cosmologies with neutrinos. The CCL test compares the smoothed simulated power spectra provided by the paper authors to the CCL output of the power spectrum via the emulator. This test corresponds to Figure 5 of the emulator paper, for a specific subset of the cosmologies: {38,39,40,42}.
The tables for the simulated power spectra were obtained directly from the authors of the emulator paper, and thus there is no benchmark code in this case.
Produced by David Alonso (@damonge): mfunc_bm.py
Produced by David Alonso (@damonge) with comparison against Jeremy Neveu's (@jeremyneveu) code: ehpk_bm.py
We use code from Tom McClintock (@tmcclintock) to do a precise numerical P(k) -> xi(r) transform: Uses cosmology models 1-3 CCL non-linear Halofit P(k) with z=0,1,2,3,4,5 (see CCL Note).
Produced by John Ellison (@ellison-j) and Zilong Du (@zdu863): 3dcorr_benchmark.ipynb
We benchmark the redshift-space correlation functions against the calculation from CosmoMAD (https://github.com/damonge/CosmoMAD) using the power spectrum P(k) from CCL which is fed into CosmoMAD.
Produced by John Ellison (@ellison-j) and Zilong Du (@zdu863): RSD_benchmark.py
The computation of angular power spectra beyond the Limber approximation has been benchmarked against CCL's own outputs at the high multipole limit. For sufficiently high multipoles, the two methods for computing angular power spectra are required to agree within a certain tolerance.
The Angpow computation of angular power spectra in the full non-Limber limit was benchmarked against an alternative non-Limber method present in a previous version of CCL (version tagged 0.9) which is now deprecated. Externally, Angpow was produced and benchmarked against CLASS.
We benchmark the computation of the halo model power spectrum using CCL to generate linear matter power spectra, and then using those power spectra to generate the halo model prediction for the non-linear power spectrum outside CCL.
Produced by David Alonso (@damonge): halomod_bm.py
We benchmark the computation of the linear and non-linear power spectrum with CLASS via CCL against the same quantities produced directly via CLASS. This serves to tests the accuracy of CCL's matter power spectrum splines in the case of multiple massive neutrinos and to ensure relevant parameters are being passed correctly to CLASS.
Produced by Danielle Leonard (@c-d-leonard): from CLASS ini files model1_pk_nu.ini, model2_pk_nu.ini, and model3_pk_nu.ini.
Benchmarks for galaxy clustering, galaxy-galaxy lensing, and cosmic shear angular power spectra and correlation functions including intrinsic alignments (3x2pt)
We self-consistently benchmark the computation of the angular power spectra and correlation functions for galaxy clustering, galaxy-galaxy lensing, and cosmic shear including intrinsic alignments for all tomographic bin combinations in the code comparison set-up, for both analytic and histogram redshift distributions. This includes the terms galaxy-galaxy (gg), galaxy-shear (gG), galaxy-intrinsic (gI), shear-shear (GG), shear-intrinsic (GI), intrinsic-intrinsic (II), galaxy-total (gG+gI), and total-total (GG+GI+II). For these benchmarks, we have assumed an IA amplitude A=1, and galaxy bias b=1.
To run these benchmarks, please download CosmoLSS from github:
https://github.com/sjoudaki/CosmoLSS_CCLbenchmarking
Please make sure that you have installed Intel Fortran 14 or higher. To compile the code, enter the directory "cosmolss_cclbenchmarking/source", type "make clean", then "make". Then go back to the main directory (i.e. "cd .."), type "./cosmomc testcclbenchmarking.ini". This will generate all of the benchmark files for the analytic redshift distributions. For the histogram case, enter "cosmolss_cclbenchmarking/source" again, and type "cp CosmoLSS_cclbenchmarking_histo.f90 CosmoLSS.f90". Now, type "make", then go back to the main directory ("cd .."), and type "./cosmomc testcclbenchmarking.ini". This now generates all of the benchmark files for the histogram redshift distributions. For completeness, the downloaded tarball contains all of the benchmark files used in the unit tests and python notebook.
Produced by Shahab Joudaki (@sjoudaki): cosmolss_cclbenchmarking.tar.gz.
We provide additional benchmarks for the computation of the galaxy-galaxy lensing angular power spectrum and correlation function for the combination of lenses in bin 1 and shapes in bin 2. We use one of the fiducial cosmologies described in the paper. Specifically, we compute the galaxy-galaxy lensing angular power spectrum for lenses in bin 1 and shapes in bin 2 of the code comparison set up, both in the analytic and binned cases. For the correlation function, we only benchmark the analytic case.
To run these benchmarks, unzip the files in ggl_benchmarks.zip. Within myggl.py, edit the string for the CCL directory and choose between the analytic and histogram cases. Then run by typing python myggl.py in the terminal. Notice this will also produce files for the corresponding angular power spectra.
Produced by Elisa Chisari (@elisachisari): ggl_benchmarks.zip.
We benchmark the computation of the CMB lensing angular cross-power spectrum with galaxy positions and shapes. We use one of the fiducial cosmologies described in the paper. Specifically, we compute these cross-angular power spectrum for lenses in bin 1 and bin 2 for both the analytic and binned N(z) cases.
To run these benchmarks, find the file cl_cmblx_bm.py and run it by typing python cl_cmblx_bm.py in the terminal.
Produced by Sukhdeep Singh (@sukhdeep1989) with small modifications from David Alonso (@damonge) and Elisa Chisari (@elisachisari): cl_cmblx_bm.py.
We benchmark the matter power spectrum and the real-space 2-pt correlation functions of cosmic shear, galaxy-galaxy lensing, and galaxy clustering under modifications to GR in the mu / Sigma quasistatic parameterisation.
For both power spectrum and correlation functions, we benchmark against outputs produced by the new version 3.0 of ISiTGR code (Integrated Software in Testing General Relativity (ISiTGR) which runs on the software CAMB and CosmoMC.) The code is publicly available from the repository https://github.com/mishakb/ISiTGR. Note that ISiTGR is based on CAMB, and therefore we do not expect agreement at 1e-4 level with CCL observables which are produced using CLASS. This is also consistent with other benchmarks for LCDM of power spectrum and correlation functions compared with codes based on CAMB.
In the case of the correlation functions, we use the same redshift distributions for lens and source bins as in the GR case above (that is, we have two redshift bins as use each as sources and lenses). We also use the forecast errors from the GR case as a guide as to the level at which we require agreement.
Produced by Mustapha Ishak (@mishak) and Cristhian Garcia Quintero (@Cristhian Garcia Quintero)
The real-space 2-pt correlation functions xi+/xi- of cosmic shear, galaxy-galaxy lensing, and galaxy clustering under modifications to GR in the mu / Sigma quasi-static parameterisation are benchmarked against outputs from the
modified GR DES 3X2pts likelihood code integrated into the most recent ISITGR version 3.1 which is built on modifications to most recent versions of CAMB and CosmoMC from 2019. The code is publicly available from the repository https://github.com/mishakb/ISiTGR. There is an agreement of 10^{-3} for the 3X2pts correlations and 10^{-5} for the matter power spectrum.
Produced by Mustapha Ishak (@mishak)