WCS sampling & FITS to TOAST FITS #86
Conversation
…2) get all ancestors of a set of tiles
…default way to tile images with a large FoV
…ges with large angular size
Codecov Report
@@ Coverage Diff @@
## master #86 +/- ##
==========================================
+ Coverage 75.42% 77.34% +1.91%
==========================================
Files 23 23
Lines 3520 3703 +183
==========================================
+ Hits 2655 2864 +209
+ Misses 865 839 -26
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
…um things like TilingMethod.TOAST actually equals TilingMethod.TOAST. Also added assertion for the resulting imageset's projection type after creation by tests.
|
Now that the tests are properly creating a TOAST from FITS, they are failing because they are not yet using the latest wwt_data_formats WorldWideTelescope/wwt_data_formats#50 |
|
wwt_data_formats 0.15.1 is now out with the new code. Closing and opening to re-trigger the CI. |
There was a problem hiding this comment.
This looks really good! Lots of the infrastructure comes together in a nice way.
I've pushed an update to the branch to merge in master and honor parallelism flags throughout the FITS tiling framework, since that couldn't actually be controlled before.
I've run into a serious problem with the _image_bounds() function with the DASCH test image, which (by sheer luck) is a pretty pathological test case. The maximum Dec of the image is in its middle (δ = 90°), not one of its corners, and this also means that it in fact spans all RAs. The np.min() and np.max() approach taken in this function has problems when dealing with longitudes because if, say, the min and max values are 30° and 60°, that could mean an image that spans longitudes 30–60°, or it could mean an image that spans 60–180–360–30°. It could be pretty tricky to make sure that we capture the two possibilities correctly.
…180 degrees of longitude
|
OK, I have pushed a couple of commits that I think fix the computation of image bounds, and it turns out that the lat/lon sampler also needed fixing to deal with images spanning very large amounts of latitude. I now can fully TOAST the DASCH test image (which I'm doing after reprojecting it to a straight TAN projection to avoid the Astropy WCS bug). They're not fully tested, but I wanted to get them uploaded before I go away for my travel. Unfortunately the positioning of the image appears to be off after these fixes. I need to look into why that's happening. |
|
Good catch on the min/max mistake, thanks! |
In one test image I have, hipsgen produces a "BAD PIXEL CUT" warning and doesn't emit this information. (I think it's because the file has an integer data type.) Avoid the crash if this is the case.
|
OK, I've pushed one more commit that avoids a crash in hipsgen with my DASCH test file. The "good" news is that my test image is landing in the wrong place because the edit: the reproject problem seems to be a consequence of astropy/astropy#13509 . The image shape looks reasonable, but the image is half-empty. However, the less-good news is that if I compare the TOAST and hipsgen output, there's still a disagreement. It looks like there's a negated rotation term somewhere. But I wouldn't worry about this too much since the image coordinate system may be generally messed up. |
|
For posterity, here's what I'm seeing. The
|
|
Here's hipsgen vs. TOAST of the
|
|
I had planned to add handling of FITS collections to a new PR, but it turned out this functionality could be added with minimal changes (thanks past toasty developers!). This may not be the most optimal way of toasting multiple FITS files, but it seems to work quite well. Let's discuss here or on Monday @pkgw |
…de the other if it exists
As noted in the comments, my DASCH sample file that crosses the pole has WCS headers that seem to have another problem beyond their specification of the TPV polynomial distortions. I need to set LONPOLE explicitly to get the correct orientation on the sky.
|
I figured out the problem with the DASCH test image, and it appears to be something outside of all of the TOAST infrastructure: the file should have a LONPOLE header. Based on my reading of Calabretta & Greisen 2002, the headers in the file are not correct. Forcing LONPOLE to 180 fixes the celestial alignment for this test image. |
The "initiate readiness state" algorithm repeatedly tests whether positions are in the tiles_to_process set. If large numbers of tiles (tens of millions) are being processed, the algorithm basically fails because such tests have O(n) scaling with lists. With a set, things become tractable. There may well be further room for improvement here, but this one change makes an enormous difference for big problems.
In processing a FITS-to-TOAST dataset going down to level 14, with about 24 million tiles to process, I discovered that the tile filter function was a non-trivial bottleneck -- it took about 10 minutes just to count the tiles that needed to be processed! Some profiling showed that most of the time was actually in the Numpy min/max functions. Here we Cythonize the function and avoid the Numpy functions. My measurements indicate a speedup of a factor of around 20.
|
OK, there is still some more work to do here — in the DECaPS2 test processing I am running into some scalability issues since there are about 28 million tiles to process. But the code here seems to basically be working reliably, and I think it makes the most sense to close out this PR and continue evolving in separate ones. Docs build is failing here due to an expired cert for |
Mostly addresses #81, but does nothing about masked pixels and does not yet support a collection of FITS input.
The main feature is based around the WCS sampler & filter, which allows us to create a TOAST imageset from any array/wcs input. The
tile_fitsmethod now has the ability to create TOAST imagesets, which also has replaced HiPS as the default tiling method for images covering a large area of the sky.Upgrades to the general process:
toast_basedoes now accept tile filters to allow processing of a specific area, and skipping every tile outside input boundsforce_tanetc. with aTilingMethodenumRequires WorldWideTelescope/wwt_data_formats#50 to work properly.