Validating the GNILC template: Redux

[brandonshensley]

In the course of using the d9 and d10 models, it has been noted that these models have more B-mode power at ell~80 in the Southern Hole than is measured by BICEP. I've put together a notebook with @delabrou investigating this issue, and my main takeaways are:

1. The difference comes from the Planck dust polarization templates used, not the post-processing work (e.g., adding small scales, log pol tens, etc.). I find roughly the same BB amplitudes in the raw GNILC templates as I do in the d9 and d10 models. @delabrou thinks it is possible that the excess power is residual noise not removed by GNILC.
2. It is unclear why the MKD model looks so different, other than a different template was used. It is not a priori clear that d12 (MKD) is any better than d10, but a posteriori whatever was used for MKD seems to agree better with BICEP measurements.

I'm not quite sure what to do here: the differences are quite large and have an impact on r forecasts. @zonca @bthorne93 @giuspugl @seclark @NicolettaK Thoughts?

Note also this old issue on validating these templates.

[zonca]

thanks @brandonshensley @delabrou, it looks pretty bad.
Is it just in the BICEP patch or is it an overall scale factor?

Skimming through the MKD paper, is seem like they also use GNILC, but with a custom postprocessing, @delabrou is it true?

[seclark]

Hrm. Do these have the same color correction?

[brandonshensley]

Hrm. Do these have the same color correction?

The 353 GHz color correction is ~10% (see this issue). It was definitely applied in creating the d9-d11 models, unsure about d12.

Is it just in the BICEP patch or is it an overall scale factor?

I believe just the BICEP patch and BB specifically. In fact, I am seeing that the d12 model has a higher polarized intensity on average over the full sky.

[b-thorne]

I made this plot a while ago for the paper draft, and it found the same discrepancy between the d9 model and the BK level:

At the time, I think we agreed that this was most likely due to residual noise in the template, and so not something it was clear we had an easy way to fix. I'm interested to understand why the MKD model seems to do better, as I had thought it was normalized to a similar GNILC input.

[brandonshensley]

Thanks @bthorne93! Would it be feasible to do a map-space comparison? I'm not sure the relevant BICEP maps are public, but happy to take that on if they are and I could be pointed to them. Other ideas for testing the noise hypothesis welcome.

[seclark]

What do we expect to be the noise level at these multipoles? We should be able to answer whether the additional power is consistent with an unlucky upward fluctuation in the BK patch.

[brandonshensley]

Computing a BB spectrum from 353 GHz splits is also on the agenda.

[brandonshensley]

I've updated the notebook with a BB spectrum based on half mission splits (scroll all the way down, sorry it's getting a bit ungainly). Agreement with the GNILC templates is very good, so I am now more confused.

[brandonshensley]

Update based on subsequent tests:

1. MKD (d12) and d9/d10/d11 certainly disagree on the dust BB power in the diffuse Southern sky (the "Southern Hole"). The raw GNILC QU template, the PR3 half mission splits, and the 353 GHz NPIPE splits all yield spectra in better agreement with d9/d10/d11. I do not understand the source of the discrepancy, but there seems to me no reason to mistrust the d9/d10/d11 models at these scales on this patch of sky.
2. In the BICEP patch specifically, and as illustrated nicely by @bthorne93 above, there does seem to be excess power in d9/d10/d11 that could be noise. The MKD model (d12) is in better agreement with the ell=80 amplitude measured by BICEP, but at least in my spectra there is a lot of bin-to-bin variance so not sure how much to read into that.

My assessment is that the d9/d10/d11 models are consistent with the best available Planck data products and therefore fine. That there are disagreements on relatively small, diffuse patches is not unexpected and also not easy to fix (as @bthorne93 says). I would still like to understand the systematic differences with MKD, but I am not seeing impetus to make any changes to the d9/d10/d11 models.

[delabrou]

Hi everyone. I checked my code and the generation of small scales is set to match the spectrum contraints from Planck, rather than being fixed according to the large scale template. This might explain why it differs from the other models.

[delabrou]

Hi Ben, all, The MKD model generates small scales at a level that matches the Planck Cl constraints from Planck 2018 results. XI. My understanding is that the other models extrapolate the Cls computed on the GNILC template. This might be the origin of the discrepancy. It is interesting that the Planck spectra match better with the BICEP measurements on the BICEP patch. Good sign! Best, Jacques

…

On 13 Sep 2022, at 09:03, Ben Thorne ***@***.***> wrote: I made this plot a while ago for the paper draft, and it found the same discrepancy between the d9 model and the BK level: <https://user-images.githubusercontent.com/16899444/189949520-862263f9-69a2-4d41-bfc2-f2e968dd11d5.png> At the time, I think we agreed that this was most likely due to residual noise in the template, and so not something it was clear we had an easy way to fix. I'm interested to understand why the MKD model seems to do better, as I had thought it was normalized to a similar GNILC input. — Reply to this email directly, view it on GitHub <#126 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXLM5KBFBLSWUEMNXURACTV6CQUXANCNFSM6AAAAAAQK7F4SY>. You are receiving this because you were mentioned.

[brandonshensley]

Thanks @delabrou! At what ell does the MKD model transition from being template-dominated to being dominated by the added fluctuations? It is 110 for d9/d10, and I confirm that our BB amplitude agrees well with the GNILC template at ell ~ 80, so discrepancies are not due to how small scales are added in the d9/d10 models. There is also the issue of PR2 vs PR3 polarization data, but I haven't yet had a chance to quantify how big an impact that is.

[delabrou]

There is no sharp transition. I generate a random map with EE and BB spectra matching the Planck ell dependence. Specifically I use a spectral index of -2.41 for the E modes and -2.52 for the B modes. Then I merge the observed a_lm with the random a_lm with a smooth transition: I co-add the original map filtered by some beam bl and the small scales map filtered by SQRT(1-bl^2) so the weights add-up to 1 when we compute the total spectrum. I use a beam of 2.5 deg in the first 3 layers (those which count most at high galactic latitude, and 2 deg. in the next 3 (more distant, with power close or in the galactic plane). I chose those "beam sizes" by visual inspection of the spectra of the input maps for the 6 layers. A bit of trial and error was necessary to get final total spectra consistent with the Planck fits. For a 2.5 or 2 degree beam, the simulated small scales take over slightly below or slightly above ell=50, but around ell=80 there still is substantial contribution from the input map, in particular in regions of filamentary dust structure. I also post-process the final co-added map with a multivariate filter to force again the total BB and EE spectra to match the Planck fit, with a normalisation set by the power in the ell=20-70 band in the layer. This procedure is specific to the new implementation of the multilayer model in the PSM, which was used to generate the recent PySM KMD model. I admit it is a bit "ad hoc", but the real data is noisy enough that it was hard to do much better on a short timescale, while being reasonably consistent with constraints -- and I feel lucky that it turns out to be in reasonable agreement with BICEP while it was not implemented specifically for that. I should add that what was done for the other two models in the latest release of the PySM looks reasonable to me, and discrepancies illustrate the uncertainties in this whole process. Constraining further those two models so that the output power matches BICEP/Keck does not guarantee that it will be consistent with observations in other regions of the sky.

[brandonshensley]

Thanks so much Jacques! This makes sense and consistent with the differences we are seeing among the models. It will be useful to include a summary of these points in the new model paper.

discrepancies illustrate the uncertainties in this whole process

I want to emphasize this important point! One of the reasons it is so critical to make multiple models available.

Constraining further those two models so that the output power matches BICEP/Keck does not guarantee that it will be consistent with observations in other regions of the sky.

Agreed, and consistent with the fact that those models seem to reproduce Planck dust BB on larger patches without any apparent deficit.

[jdborrill]

While it is obviously true that fixing the B/K patch (if it is indeed broken) won't guarantee the accuracy of the rest of the sky, it is a very special patch. Experiments drilling deep on it don't get to compensate for discrepancies between a model and reality by averaging over a large enough sky area.

[delabrou]

There is no sharp transition. I generate a random map with the proper EE and BB and TT and EE spectrum, and then co-add the original map filtered by some beam bl and the small scales map filtered by SQRT(1-bl^2) so the weights add-up to 1 when we compute the total spectrum. The bl depend on which field is being considered (T, E or B) and on which layer is being considered (from 1 to 6). I use 5 arcmin for T, 150 arcmin (2.5 degrees) for the first three layers (most of the high galactic latitude signal), and 120 arcmin for the next 3 layers (mostly close to and in the galactic plane. I chose that by visual inspection of the spectra of the input maps for the 6 layers. This process suppresses some of the power of the input map, according to a 2 or 2.5 degree beam, and inputs the power of a random map, which is generated with the ell dependancy found in the Planck paper (spectral index of -2.41 for the E modes and -2.52 for the B modes). For a 2.5 or 2 degree beam, the simulated small scales take over slightly below or slightly above ell=50, but around ell=80 there still is substantial contribution from the input map, in particular in regions of filamentary dust structure. This process is a bit "ad hoc", and is also specific to the multilayer model. It leaves margin for improvement, but I did some checking over various sky regions that the output spectra of the total map agree reasonably well with the Planck fit.

…

On 15 Sep 2022, at 07:52, brandonshensley ***@***.***> wrote: Thanks @delabrou <https://github.com/delabrou>! At what ell does the MKD model transition from being template-dominated to being dominated by the added fluctuations? It is 110 for d9/d10, and I confirm that our BB amplitude agrees well with the GNILC template at ell ~ 80, so discrepancies are not due to how small scales are added in the d9/d10 models. There is also the issue of PR2 vs PR3 polarization data, but I haven't yet had a chance to quantify how big an impact that is. — Reply to this email directly, view it on GitHub <#126 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXLM5PI4JRCVYRQNLMRQ6DV6MZ3JANCNFSM6AAAAAAQK7F4SY>. You are receiving this because you were mentioned.

[brandonshensley]

Update: I reran my analysis on the BICEP patch specifically, notebook here. I used \Delta\ell = 20 to avoid large wiggles at low \ell. My takeaways are:

1. As expected, the PySM models agree very well with the GNILC template at low \ell, modulo the color correction.
2. There is some discrepancy between the GNILC template and the power spectrum computed from the NPIPE splits. Perhaps this gives some indication of noise levels?
3. Both the GNILC template and the NPIPE splits have more BB power at \ell=80 than the value quoted by BK.

@bthorne93, would it be informative to add the NPIPE splits to your plot?