Self calibration

[iancze]

Is your feature request related to a problem or opportunity? Please describe.

Accurately characterizing the antenna-based complex gains is part of the calibration process for interferometric visibilities, and is frequently carried out by observing a known calibrator source, like a quasar, interleaved with source observations. The complex gains derived from the calibrator observations are then interpolated and applied to the source observations.

Though this calibration is usually adequate for imaging, it can leave residual gain calibration errors, which manifest in poor dynamic range for otherwise high signal to noise sources (i.e., the image noise is significantly larger than one might expect from thermal noise alone).

Self-calibration is the process of iteratively solving for the time and antenna-dependent complex gains using a model of the target source and the data visibilities of the source themselves. As shown in Brogan et al. 2018, we can construct an objective function for this process

V are the data visibilities and w are the data weights, both of which remain fixed throughout the problem. g_i and g_j are the antenna-based complex gains and M are the model visibilities. The goal is to minimize the objective function S, which encapsulates the weighted deviations between the model visibilities and (possibly recalibrated) data visibilities for all times, for all antenna pairs.

In this formulation, we solve for g_i and g_j for all antennas i,j, for all times t_k. At first glance, the problem might seem degenerate because, in addition to the gains, we also need to solve for the (unknown) source model. In practice, however, for any interferometric observation using a large array like ALMA or the VLA, the many pairs of cross-antenna baselines break this degeneracy, and the gains and model can be solved for in an iterative process.

Self calibration with CASA is typically carried out in the manner described in Brogan et al. 2018:

1. a rough source model (i.e. CLEAN components) is constructed by tcleaning to a shallow level
2. the phase components of the complex gains are solved for using a wide-running time average
3. the phase calibration perturbations are applied to the dataset
4. the dataset is reimaged to make a new (presumably more accurate) source model. The synthesized images are monitored for peak S/N

and the loop repeats (gradually using finer-grained averaging windows) until the peak S/N in the image stops increasing. Sometimes, additional amplitude calibration loops are carried out.

Describe the solution you'd like
Successive rounds of self-calibration can be more closely interleaved with the source model optimization itself. This has been successfully implemented for other non-parametric modeling applications (e.g., see Hezaveh et al. 2013, Appendix A).

To accomplish this within the MPoL framework, we first need to implement #17 (loose visibilities) to gain access to the model visibilities as a function of antenna pair and time (this information is destroyed during the gridding process). Depending on how we end up solving #17, we will also want a mechanism to store the complex gain perturbations for all antennas, for all times. This could be as a set of parameters on the loose visibility connector itself, or an additional layer.

If we have multiple execution blocks, most likely we will want a different "connector" for each block, to keep the antenna parameters straight. Different execution blocks are also good candidates for "batching" during the training process.

We can interleave optimization steps for the full source model loss functions with solutions for the self-calibration loss function S. It might even be possible to just combine them all into a single loop, but it might be easier to organize the process if we split them into separate loops.

Rather than averaging gain settings over successively finer time windows, we can apply a Gaussian process or other type of regularizer to the time-series gains and solve that.

Describe alternatives you've considered
There probably is some overlap with imaging routines that use the bispectrum directly, like Chael et al. 2016, and we can probably add yet another iteration loop including these imaging adjustments.

For now, we are just starting with visibilities that have been self-calibrated using a CLEAN model produced by tclean. As described in Yamaguchi et al. 20, if the self-calibration model is decoupled from the true source model, it's possible for this self-calibration to imprint features into the residuals.

In truth, these residual effects are pretty minor so this process probably only makes sense for very high S/N datasets.