Image generation pipeline sanity test

[EiffL]

The purpose of this test will be to check the consistency of the instance catalog creation and image generation with the input quantities in the protoDC2 catalog. This can be done by generating a small instance catalog and one exposure with ImSim and checking the measured sizes and ellipticities of the galaxies in the field. The statistics will not be very high but it will catch obvious failures.
This test will also be able to check for obvious failures in the model for complex morphologies slated to be included in the ImSim version of DC2

[EiffL]

@yymao @fjaviersanchez If I understand what we just said in the SSim meeting, this issue was opened with essentially the same intention of testing the generated images.
I have played around with that a little bit, I can generate instance catalogs and produce an eimage with imsim, but then I got stuck when trying to analyse the images with the DM stack. The first issue is that I'm very new to the DM stack, the second was that I manage to get some code working with v13, but it's python2 and wasn't compatible with the rest of my python3 environment.

So, I got frustrated and decided to do something a little simpler, that is, generating galaxies in individual postage stamps with galsim (bypassing most of imsim), for basic morphology tests.

Let me know what you guys had in mind, this issue can be repurposed, and I can open another one for simple tests on individual galaxy postage stamps.

[fjaviersanchez]

@EiffL what I was thinking for DC2 was to add a GCR reader for the e-images and add some tests in DESCQA for those images. For the validation of the DM processed catalogs/images there are some tools already in DM that @jchiang87 was testing and on which he'll report tomorrow. However, more validation tools will probably be needed. I have a notebook that reads an image and processes through DM, I can share it with you if this is useful. Also, we have a python3 shared version of the DM stack at NERSC here: /global/common/software/lsst/cori-haswell-gcc/stack/setup_w_2017_46_py3_gcc6.sh

[yymao]

Agreed that there can be multiple validation tools, even for the same data set. I think DESCQA would be particularly useful for users who are not familiar with the technical details of the data sets, since they can just browse the validation outputs to get a sense of what's going on. It's also useful for quick sanity checks.

[fjaviersanchez]

@EiffL do you already have any code in this repo to ingest and check images?

[EiffL]

@yymao Sorry, I got seriously sidetracked but I now have a draft of a test that does the following:

• uses the sims_GCRCatSimInterface to create a small phosim instance catalog
• uses ImSim functionalities to parse the instance catalog
• uses sims_GalSimInterface to create a list of galsim objects
• uses galsim to draw each galaxy in a postage stamp at the COSMOS resolution and at the same time draws a random COSMOS galaxy
• measures moments from both real and simulated galaxies using galsim.hsm

It lives for now on branch issue/62/image_test. The good thing about this approach is that it is a non-trivial and fast way to check the galaxy image simulation without generating full exposures and running the DM stack.

The only difficulty that I'm having is that it looks like some of the required packages are not accessible in the default environment (or maybe they are but need to be activated) like:

    from desc.sims.GCRCatSimInterface import *
ModuleNotFoundError: No module named 'desc'

For now I can manually run it in my own python environment but I'm not sure how to add these softwares in default descqa

Here is a small example of the results https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-22_8

And some postage stamps comparing cosmos images (left) to simulated images (right):

It kind of looks like the simulated galaxies are fainter than their COSMOS counterparts... This may be due to some subtlety on how I select the 2 samples (imag <24) or on how I rescale the fluxes to match the COSMOS setting...

@rmandelb any particular other statistics and/or splits of the data you think could be interesting to look at ?

[yymao]

@EiffL thanks! This is awesome. I'm cc'ing @fjaviersanchez as he'll be interested in this progress.

Are you ready to submit a PR? Or do you think there are still outstanding issues?

As for the desc packages, I do think they are installed in the same environment as descqa but I'll double check with Heather.