Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor WFAU image retrieval to reject deprecated images #2809

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
Draft

Refactor WFAU image retrieval to reject deprecated images #2809

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
d070bdf
refactor wfau to retrieve metadata about images and reject deprecated…
keflavich Aug 13, 2023
2fa5050
codestyle
keflavich Aug 13, 2023
6d16328
WIP: trying to fix radius=not None queries
keflavich Aug 13, 2023
bfd412e
update reader for radius!=None searches, and correspondingly update t…
keflavich Aug 13, 2023
e3e4905
restore extract_urls - used by other things
keflavich Aug 15, 2023
3a7b298
tests of different image result queries
keflavich Aug 15, 2023
c4db9fb
update test data
keflavich Aug 15, 2023
79fd880
update tests to match new data
keflavich Aug 15, 2023
75e4726
change remote test
keflavich Aug 15, 2023
5ac89b0
avoid reusing variable
keflavich Aug 15, 2023
ed94efe
add changelog entry
keflavich Aug 15, 2023
8733870
privatize _extract_urls & try to fix a broken old test
keflavich Aug 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGES.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,12 @@ xmatch
- Minor internal change to use VOTable as the response format that include
units, too. [#1375]

wfau
^^^^

- Minor enhancement to enable getting tables of images to download instead
of just raw URLs. Tables include metadata about deprecation [#2809]


Infrastructure, Utility and Other Changes and Additions
-------------------------------------------------------
Expand Down
22 changes: 0 additions & 22 deletions astroquery/ukidss/tests/data/image_results.html
View file
Open in desktop

This file was deleted.

43 changes: 43 additions & 0 deletions astroquery/ukidss/tests/data/image_results_noradius.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
<HTML><HEAD>&nbsp;<script language="JavaScript" type="text/javascript">
<!--
window.focus();
//--> </script>
</HEAD><h3>GetImage cut-out results</h3>
<br><b>J2000 coords:</b> RA: 83.633083 Dec:22.0145
<br><b>Programme:</b> UKIDSS Galactic Clusters Survey, GCS
<br><b>Filter:</b> K
<br>Processing ...
<br>Connecting to database: UKIDSSDR11PLUS<p>
<table border="1"><tr bgcolor="#FFFFCC"><td><b>Link</b></td><td><b>multiframeID</b></td><td><b>frametype</b></td><td><b>obstype</b></td><td><b>filterid</b></td><td><b>shortname</b></td><td><b>dateObs</b></td><td><b>extNum</b></td><td><b>deprecated</b></td>
<tr bgcolor=#FFDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/getImage.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01818_sf_st.fit&mfid=1737581&extNo=4&lx=1339&hx=1638&ly=1953&hy=2252&rf=0&flip=1&uniq=4917_976_57_87624_1&xpos=150.7&ypos=150.3&band=K&ra=83.633083&dec=22.0145" target=show>show</a></td>
<td nowrap>1737581</td>
<td nowrap>leavstack</td>
<td nowrap>OBJECT</td>
<td nowrap>5</td>
<td nowrap>K</td>
<td nowrap>2007-10-11 13:12:05.5</td>
<td nowrap>5</td>
<td nowrap>0</td>
<tr bgcolor=#DDDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/getImage.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01818_sf.fit&mfid=1737579&extNo=4&lx=1339&hx=1638&ly=1953&hy=2252&rf=0&flip=1&uniq=4917_976_57_87624_2&xpos=150.6&ypos=150.3&band=K&ra=83.633083&dec=22.0145" target=show>show</a></td>
<td nowrap>1737579</td>
<td nowrap>leav</td>
<td nowrap>OBJECT</td>
<td nowrap>5</td>
<td nowrap>K</td>
<td nowrap>2007-10-11 13:12:05.5</td>
<td nowrap>5</td>
<td nowrap>0</td>
<tr bgcolor=#FFDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/getImage.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01822_sf.fit&mfid=1737587&extNo=4&lx=1323&hx=1622&ly=1937&hy=2236&rf=0&flip=1&uniq=4917_976_57_87624_3&xpos=150.7&ypos=150.6&band=K&ra=83.633083&dec=22.0145" target=show>show</a></td>
<td nowrap>1737587</td>
<td nowrap>leav</td>
<td nowrap>OBJECT</td>
<td nowrap>5</td>
<td nowrap>K</td>
<td nowrap>2007-10-11 13:12:53.8</td>
<td nowrap>5</td>
<td nowrap>0</td>
</table>
3 rows returned.
147 changes: 147 additions & 0 deletions astroquery/ukidss/tests/data/image_results_radius.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<title>WSA ImageList</title>

<script type="text/javascript" src="http://wsa.roe.ac.uk/imSwap.js">
</script>
</head>

<body bgcolor="#FFFFFF" text="#000000" link="#0000FF" vlink="#000080" alink="#FF0000" onLoad="swap(0);" >
<table cellpadding="0" cellspacing="0" border="0"><tr><td><b>WSA ImageList</b></td><td>&nbsp;&nbsp;&nbsp;</td>
<td><img name="imgMain" src="http://wsa.roe.ac.uk/static.gif" border="1"></td></table><p>
Not logged in: links will only be returned for frames that are publicly accessible<p>
Archive Listing<p>Searching...<br>
<b>Survey: </b>UKIDSS Galactic Clusters Survey, GCS<br>
<b>Waveband: </b>K<br>
<b>Minimum RA:</b> 5.551333333333333 hours <b>Maximum RA:</b> 5.599333333333333 hours <br>
<b>Minimum Dec:</b> 21.68116666666667 degrees <b>Maximum Dec:</b> 22.347833333333334 degrees <br>
<script language="JavaScript" type="text/javascript"> <!--
swap(1);
--> </script>
<br><b>Using database:</b> UKIDSSDR11PLUS
<p><table><tr><td align="right"><b>View</b> column link</td><td align="left">shows jpeg images of multiframe in a new window plus links to download file(s)</td>
<tr><td align="right"><b>Img</b> column link</td><td align="left">download the <a href="http://wsa.roe.ac.uk/qa.html#compress">RICE compressed</a> FITS image file. Use <b>View</b> column link to retrieve uncompressed images.</td>
<tr><td align="right"><b>Cat</b> column link</td><td align="left">download the FITS catalogue file.</td>
</table>
<br>begin row 1<br><table border="1"><tr bgcolor="#FFFFCC"><td align="center">View</td><td align="center">Img</td><td align="center">Cat</td>
<td align="middle"><b>multiframeID</b></td>
<td align="middle"><b>frameType</a></b></td>
<td align="middle"><b>obstype</a></b></td>
<td align="middle"><b>raBase</a></b></td>
<td align="middle"><b>decBase</a></b></td>
<td align="middle"><b>shortname</a></b></td>
<td align="middle"><b>exptime</a></b></td>
<td align="middle"><b>dateObs</a></b></td>
<td align="middle"><b>project</a></b></td>
<td align="middle"><b>numDetectors</b></td>
<td align="middle"><b>ukirtRunNo</a></b></td>
</tr>
<tr bgcolor=#FFDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/display.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01802_sf_st.fit&cat=djoser:/disk05/wsa/ingest/fits/20071011_v1/w20071011_01802_sf_st_cat.fits&comp=djoser:/disk04/wsa/products/jpgs/20071011_v1/w20071011_01802_sf_st&noExt=4&MFID=1737553&rID=50" target=display>view</a></td>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/fits_download.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01802_sf_st.fit&MFID=1737553&rID=50">FITS</a></td>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/fits_download.cgi?file=djoser:/disk05/wsa/ingest/fits/20071011_v1/w20071011_01802_sf_st_cat.fits&MFID=1737553&rID=50">FITS</a></td>
<td nowrap align="right">1737553</td>
<td nowrap align="right">leavstack</td>
<td nowrap align="right">OBJECT</td>
<td nowrap align="right">+5.5777306</td>
<td nowrap align="right">+21.7913333</td>
<td nowrap align="right">K</td>
<td nowrap align="right">+10.000000</td>
<td nowrap align="right">2007-10-11 13:08:30.0</td>
<td nowrap align="right">U/UKIDSS/GCS21</td>
<td nowrap align="right">4</td>
<td nowrap align="right">1802</td>
</tr>
<tr bgcolor=#DDDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/display.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01802_sf.fit&cat=NONE&comp=djoser:/disk04/wsa/products/jpgs/20071011_v1/w20071011_01802_sf&noExt=4&MFID=1737551&rID=50" target=display>view</a></td>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/fits_download.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01802_sf.fit&MFID=1737551&rID=50">FITS</a></td>
<td>&nbsp;</td>
<td nowrap align="right">1737551</td>
<td nowrap align="right">leav</td>
<td nowrap align="right">OBJECT</td>
<td nowrap align="right">+5.5777306</td>
<td nowrap align="right">+21.7913333</td>
<td nowrap align="right">K</td>
<td nowrap align="right">+10.000000</td>
<td nowrap align="right">2007-10-11 13:08:30.0</td>
<td nowrap align="right">U/UKIDSS/GCS21</td>
<td nowrap align="right">4</td>
<td nowrap align="right">1802</td>
</tr>
<tr bgcolor=#FFDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/display.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01806_sf.fit&cat=NONE&comp=djoser:/disk04/wsa/products/jpgs/20071011_v1/w20071011_01806_sf&noExt=4&MFID=1737559&rID=50" target=display>view</a></td>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/fits_download.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01806_sf.fit&MFID=1737559&rID=50">FITS</a></td>
<td>&nbsp;</td>
<td nowrap align="right">1737559</td>
<td nowrap align="right">leav</td>
<td nowrap align="right">OBJECT</td>
<td nowrap align="right">+5.5777306</td>
<td nowrap align="right">+21.7913333</td>
<td nowrap align="right">K</td>
<td nowrap align="right">+10.000000</td>
<td nowrap align="right">2007-10-11 13:09:18.6</td>
<td nowrap align="right">U/UKIDSS/GCS21</td>
<td nowrap align="right">4</td>
<td nowrap align="right">1806</td>
</tr>
<tr bgcolor=#DDDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/display.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01818_sf_st.fit&cat=djoser:/disk05/wsa/ingest/fits/20071011_v1/w20071011_01818_sf_st_cat.fits&comp=djoser:/disk04/wsa/products/jpgs/20071011_v1/w20071011_01818_sf_st&noExt=4&MFID=1737581&rID=50" target=display>view</a></td>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/fits_download.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01818_sf_st.fit&MFID=1737581&rID=50">FITS</a></td>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/fits_download.cgi?file=djoser:/disk05/wsa/ingest/fits/20071011_v1/w20071011_01818_sf_st_cat.fits&MFID=1737581&rID=50">FITS</a></td>
<td nowrap align="right">1737581</td>
<td nowrap align="right">leavstack</td>
<td nowrap align="right">OBJECT</td>
<td nowrap align="right">+5.5935528</td>
<td nowrap align="right">+21.7913333</td>
<td nowrap align="right">K</td>
<td nowrap align="right">+10.000000</td>
<td nowrap align="right">2007-10-11 13:12:05.5</td>
<td nowrap align="right">U/UKIDSS/GCS21</td>
<td nowrap align="right">4</td>
<td nowrap align="right">1818</td>
</tr>
<tr bgcolor=#FFDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/display.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01818_sf.fit&cat=NONE&comp=djoser:/disk04/wsa/products/jpgs/20071011_v1/w20071011_01818_sf&noExt=4&MFID=1737579&rID=50" target=display>view</a></td>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/fits_download.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01818_sf.fit&MFID=1737579&rID=50">FITS</a></td>
<td>&nbsp;</td>
<td nowrap align="right">1737579</td>
<td nowrap align="right">leav</td>
<td nowrap align="right">OBJECT</td>
<td nowrap align="right">+5.5935528</td>
<td nowrap align="right">+21.7913333</td>
<td nowrap align="right">K</td>
<td nowrap align="right">+10.000000</td>
<td nowrap align="right">2007-10-11 13:12:05.5</td>
<td nowrap align="right">U/UKIDSS/GCS21</td>
<td nowrap align="right">4</td>
<td nowrap align="right">1818</td>
</tr>
<tr bgcolor=#DDDDDD>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/display.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01822_sf.fit&cat=NONE&comp=djoser:/disk04/wsa/products/jpgs/20071011_v1/w20071011_01822_sf&noExt=4&MFID=1737587&rID=50" target=display>view</a></td>
<td><a href="http://wsa.roe.ac.uk/cgi-bin/fits_download.cgi?file=/disk05/wsa/ingest/fits/20071011_v1/w20071011_01822_sf.fit&MFID=1737587&rID=50">FITS</a></td>
<td>&nbsp;</td>
<td nowrap align="right">1737587</td>
<td nowrap align="right">leav</td>
<td nowrap align="right">OBJECT</td>
<td nowrap align="right">+5.5935528</td>
<td nowrap align="right">+21.7913333</td>
<td nowrap align="right">K</td>
<td nowrap align="right">+10.000000</td>
<td nowrap align="right">2007-10-11 13:12:53.8</td>
<td nowrap align="right">U/UKIDSS/GCS21</td>
<td nowrap align="right">4</td>
<td nowrap align="right">1822</td>
</tr>
</table><br> row(s) 1 to 6 displayed.<br>
<pre></pre><a href="javascript:history.go(-1)">Back to form</a> (uses Javascript)<script language="JavaScript" type="text/javascript">
<!--
swap(0)
-->
</script>
</body></html>
26 changes: 17 additions & 9 deletions astroquery/ukidss/tests/test_ukidss.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@
from ...exceptions import InvalidQueryError

DATA_FILES = {"vo_results": "vo_results.html",
"image_results": "image_results.html",
"image_results_noradius": "image_results_noradius.html",
"image_results_radius": "image_results_radius.html",
"image": "image.fits",
"votable": "votable.xml",
"error": "error.html"
Expand Down Expand Up @@ -74,9 +75,12 @@ def parse_coordinates_mock_return(c):

def get_mockreturn(method='GET', url='default_url',
params=None, timeout=10, **kwargs):
if "Image" in url:
filename = DATA_FILES["image_results"]
url = "Image_URL"
if "GetImage" in url:
filename = DATA_FILES["image_results_noradius"]
url = "GetImage"
elif "ImageList" in url:
filename = DATA_FILES["image_results_radius"]
url = "ImageList"
elif "SQL" in url:
filename = DATA_FILES["vo_results"]
url = "SQL_URL"
Expand Down Expand Up @@ -114,23 +118,27 @@ def test_get_images_async_1():

def test_get_images_async_2(patch_get, patch_get_readable_fileobj):

# debug check: get the table first & make sure it has 'deprecated' column as expected
tbl = ukidss.core.Ukidss.get_image_table(icrs_skycoord, programme_id="GPS")
assert "deprecated" in tbl.colnames

image_urls = ukidss.core.Ukidss.get_images_async(icrs_skycoord, programme_id="GPS")

assert len(image_urls) == 1
assert len(image_urls) == 3


def test_get_image_list(patch_get, patch_get_readable_fileobj):
urls = ukidss.core.Ukidss.get_image_list(
icrs_skycoord, frame_type="all", waveband="all", programme_id="GPS")
print(urls)
assert len(urls) == 1
assert len(urls) == 3


def test_extract_urls():
with open(data_path(DATA_FILES["image_results"]), 'r') as infile:
with open(data_path(DATA_FILES["image_results_radius"]), 'r') as infile:
html_in = infile.read()
urls = ukidss.core.Ukidss.extract_urls(html_in)
assert len(urls) == 1
urls = ukidss.core.Ukidss._extract_urls(html_in)
assert len(urls) == 14


def test_query_region(patch_get, patch_get_readable_fileobj):
Expand Down
17 changes: 17 additions & 0 deletions astroquery/ukidss/tests/test_ukidss_remote.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,20 @@ def test_query_region_constraints(self):

assert isinstance(table_constraint, Table)
assert len(table_noconstraint) >= len(table_constraint)

def test_deprecated_image_list(self):
"""
Regression test for Issue 2808
"""
crd = SkyCoord(ra=211.3194905, dec=54.413845, unit=(u.deg, u.deg))
uk = ukidss.core.UkidssClass()
uk.database = 'UHSDR2'
result = uk.get_image_list(crd, waveband='all', ignore_deprecated=True)

# this image is not deprecated (deprecated==0)
# can't check for exact URL match because URLs include generated 'uniq' strings
assert any("file=/disk73/wsa/ingest/fits/20190614_v5/w20190614_00626_st.fit"
in x for x in result)
# this image is deprecated (deprecated==80)
assert not any("file=/disk53/wsa/ingest/fits/20150129_v5/w20150129_02901_st.fit"
in x for x in result)