Calibration framework for telescope observations
This project proposes a set of tools to integrate image processing algorithms into a pipeline for the calibration of astronomical data. The project functions operate as a layer between the user input/output parameters and the mathematical operations. The latter may be developed by the user or imported from python astronomical packages such as astropy. Since the library tools store all the user steps, it can repeat the whole calibration procedure automatically. This functionality will facility the teaching and publishing of the astronomical data calibrations. Furthermore, it will also improve the error propagation into the final scientific data.
Unlike other scientific disciplines, astrophysical theories and models depend exclusively on observational data. The diverse instruments on terrestrial or spatial telescopes convert the electromagnetic spectrum into a signal, whose strength depends on the number of incoming photons. These scientific observations, however, only provide a qualitative measurement on their own. Additional, non-scientific observations are required to calibrate the astronomical signal into physical units. This calibration process is commonly known as data reduction.
Most telescope archives and virtual observatories provide access to their raw scientific observations, as well as, their corresponding calibration files. The data reduction, however, must be accomplished by each user. For the past 30 years, IRAF [1] has been the main tool available to the community for this purpose. Furthermore, thanks to its documentation and argument-based functions, it provides enough flexibility to treat data from any instrument. Despite these unmatched features, this library application remains very challenging for both novice and experienced users. In “The Zen of IRAF” guide by A. Charles Pullen we can read in the introduction: “The [IRAF] user who doesn’t [see every error message, as a worthy challenge] will likely be found naked, screaming, in the rain, on a dark night, in their local city park prior to being institutionalized”. Given the current situation, those seeking to use the available astronomical data have two options: Develop their own solutions via image processing algorithms or relaying on the patronage of a previous user pipeline (either in IRAF [2] or, if available, the tools provided by the telescope instrument [3]). Both of these scenarios contribute to a segmentation on the development, due to different scientific objectives, instead of encouraging a convergence based on the algorithm purpose.
The python community has attempted to provide a more universal solution to the astronomical data calibration. The STScI (Space Telescope Science Institute, an american university association working for NASA) developed PyRaf . This library gives the ability to run IRAF functions in an environment that has all the power and flexibility of Python. Despite this significant upgrade, its complexity has limited its support: The STScI has constrained the installation to its legacy coanda package, which only supports python2.7. Another “pythonic” approach can be found within the Astropy project. The “ccdproc” package provides basic astronomical calibration treatments, which make use of python numerical libraries, while also including third party algorithms, such as the “lacosmic” task [4].
This project seeks to relay on these works, while also introducing a new layer of features to improve the astronomical data calibration by making it easier to share and also by improving its scientific value. The diagram on Fig.1 displays a basic workflow structure for this calibration framework. At the heart of this design we find a data container, which stores the order of the calibration procedures, as well as, their configuration. This container guarantees the first benefit from this project: Since all the inputs are saved, any user can repeat the calibration treatment, as long as, they have access to the raw observations and the data on this container. The secondary advantage from this design lies in the potential to accomplish an error propagation from the original data to the final calibrated output. Even though, the pixel noise can be easily calculated from the instrument manufacturer documentation, most published research does not include this uncertainty source due to the complexity of the calibration process. Instead, researchers commonly take a 5% error on the pixel value, which is the upper limit in high quality observations of standard stars. This is accepted as a constant value even though the mathematical foundation in the error propagation is well establish [5]. As an improvement from this situation, the library functions would provide a basic error propagation scheme: For each input file the code would generate a pixel noise image from the data on the fits file header. Afterwards, at each calibration step the library functions would repeat the operation on this uncertainty frame, hence, propagating the error. Additionally, the user would also have the option to declare an error propagation algorithm in their own functions.
Finally, an essential aspect in the astronomical data reduction procedure is the capability to visualise the outputs from each calibration step. Using the tools available from the python community, this library will include functions towards this purpose. In the proposed project scope, this represents a log, in which the user parameters are stored and the outputs printed. The user, however, would have the tools to define a plotting style in their algorithm so it is included in the log. In the long term aims, a basic graphical interface to run the tasks could be generated using jupyter notebooks and widgets.
The main class functions enable the user to link the image processing algorithms, which may be imported from PyRaf, ccdproc or third party user libraries, with the desired configuration. The algorithm sequence and their configuration are stored in the data container, for which we can use a pandas dataframe. Once the current algorithm is executed the output parameters are also stored within the data container. Finally, using this information the same algorithm is executed on the uncertainty frame to propagate the error. Within this milestone, no feedback is provided to the user. However, using tools such as pylatex, it is possible to generate a log to print the output, and hence, check its quality. For example, in the case of the lacosmic task, an algorithm developed to remove cosmic rays, we may plot the before and after frames, as well as, the cosmic rays mask. This is displayed in Fig.2:
At this design layer, the project includes functions to bring some feedback to the user. Going back to the previous example, the plot in Fig.2 would be displayed in a matplotlib window. The algorithm would require the user input at this point: For a valid treatment, the output would be saved, otherwise a new configuration has to be introduce to repeat the algorithm on the observational frame.
The complete design includes an user interface to link input files and to launch the calibration algorithms. The sketch in Fig.3 displays the widgets on the main window: A region, which lists the treatments so far performed and provides the option to select them, relaunching, adding or deleting. In the central region, we can adjust the configuration of the treatment currently selected. The number of arguments depend on the task selected. Below there is an inbuilt terminal to display output information from the executed task. Finally, on the right there is a window with all files belonging to the current observation. In this region shortcuts may be added to display these files using tools such as matplotlib or pyds9.
Introduction to telescope data calibration (April 23 — April 27):
We will use the first week to familiarize the students with the instrument allocation time procedure, telescope observation and the data calibration. For the later case, I propose to follow the examples in my video tutorials.
Abril 30 — May 4 (1 week):
The second week is employed to lay out the workflow for the first milestone: The calibration framework. The tasks to consider include defining the inputs and outputs configuration for the calibration data container, defining the structure of the configuration files and the requirements for third party algorithms integration. Furthermore, there will be an introduction to telescope observational files using astropy and an introduction to pandas dataframes. Finally, we will discuss the best code design so that the community can share and install the code.
May 7 — May 11 (1 week):
The third week is employed to lay out the workflow for the second milestone: The user feedback framework. The tasks to consider are the functions, which will be provided to the developer to design a quality check-up after each algorithm and establish a label system for output files in order to easily distinguish the outputs from each algorithm. Finally, a set of functions will be declared in order to plot the output data for the calibration log and the graphs displayed at each iteration if required. Furthermore there will be an introduction to matplotlib display and widgets and python terminal I/O.
May 14 — May 25 (2 week):
At this point we will start to write the project code focusing on the first milestone.
May 28 — June 8 (2 weeks):
At this point we will start to write the project code focusing on the second milestone.
June 11 — June 15 (1 week):
The core functions of the project are expected to be ready at this deadline. This week will be employed to run a basic data reduction from the point of view of an astronomer seeking to write their own reduction pipeline using this project functions. Additionally, we will start to write the documentation.
June 18 — June 22 (1 week):
A week will be spent discussing the basic GUI to employ over the project functions. The design will encourage the installation of new image procesing algorithms and instruments configuration. Furthermore, there will be tutorials on PyQt and QTdesigner.
June 25 — July 6 (2 weeks):
At this point, we will start to write the project code for the GUI.
July 9 — July 13 (1 weeks):
Once the complete project code is ready, along with its GUI, we will run tests during one week seeking bugs and the input from colleagues from both the python and astronomical communities. Additional work will be performed on the documentation to include the GUI description.
July 16 (suggested pencils down date) — July 20 (final evaluation deadline):
Finish the documentation and write some tutorials. Look for any code issues which may exist. Confirm the installation on different operative systems.
[1] Doug Tody. The IRAF data reduction and analysis system. page 733–748, 1986.
[2] Philip Massey and Margaret M. Hanson. Astronomical spectroscopy. arXiv:1010.5270 [astro-ph], October 2010.
[3] W. Freudling, M. Romaniello, D. M. Bramich, P. Ballester, V. Forchi, C. E. García-Dabló, S. Moehler, and M. J. Neeser. Automated data reduction workflows for astronomy - the ESO reflex environment. Astronomy & Astrophysics, 559:A96, 2013.
[4] Pieter G. van Dokkum. Cosmic-Ray rejection by laplacian edge detection. pages 1420–1427, November 2001.
[5] Nicolas Cardiel, Javier Gorgas, Jess Gallego, Angel Serrano, Jaime Zamorano, Maria-Luisa Garcia-Vargas, Pedro Gomez-Cambronero, and Jose M. Filgueira. Proper handling of random errors and distortions in astronomical data analysis. volume 4847, pages 297–304, December 2002.