BUG: fetch_sdss_galaxy_colors 404 error

[crhea93]

Hello!
I am trying to collect the SDSS galaxy colors using the astroML.datasets.fetch_sdss_galaxy_colors() method. Unfortunately, I am obtaining the following error:

HTTPError: HTTP Error 404: Not Found

I am including a minimum working example here:

from astroML.datasets import fetch_sdss_galaxy_colors

data = fetch_sdss_galaxy_colors()

I am using astroML==1.0.2.post1 and astroquery==0.4.

Here is the full error log:

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Input In [2], in <module>
      1 # Download data
----> 2 data = fetch_sdss_galaxy_colors()
      3 data = data[::10]

File ~/miniconda3/envs/astro/lib/python3.9/site-packages/astroML/datasets/sdss_galaxy_colors.py:59, in fetch_sdss_galaxy_colors(data_home, download_if_missing)
     57 print("querying for %i objects" % NOBJECTS)
     58 print(query_text)
---> 59 output = sql_query(query_text)
     60 print("finished.")
     62 kwargs = {'delimiter': ',', 'skip_header': 2,
     63           'names': GAL_COLORS_NAMES, 'dtype': None,
     64           'encoding': 'ascii',
     65           }

File ~/miniconda3/envs/astro/lib/python3.9/site-packages/astroML/datasets/tools/sql_query.py:38, in sql_query(sql_str, url, format)
     36 sql_str = remove_sql_comments(sql_str)
     37 params = urlencode(dict(cmd=sql_str, format=format))
---> 38 return urlopen(url + '?%s' % params)

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:214, in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    212 else:
    213     opener = _opener
--> 214 return opener.open(url, data, timeout)

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:523, in OpenerDirector.open(self, fullurl, data, timeout)
    521 for processor in self.process_response.get(protocol, []):
    522     meth = getattr(processor, meth_name)
--> 523     response = meth(req, response)
    525 return response

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:632, in HTTPErrorProcessor.http_response(self, request, response)
    629 # According to RFC 2616, "2xx" code indicates that the client's
    630 # request was successfully received, understood, and accepted.
    631 if not (200 <= code < 300):
--> 632     response = self.parent.error(
    633         'http', request, response, code, msg, hdrs)
    635 return response

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:555, in OpenerDirector.error(self, proto, *args)
    553     http_err = 0
    554 args = (dict, proto, meth_name) + args
--> 555 result = self._call_chain(*args)
    556 if result:
    557     return result

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:494, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
    492 for handler in handlers:
    493     func = getattr(handler, meth_name)
--> 494     result = func(*args)
    495     if result is not None:
    496         return result

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:747, in HTTPRedirectHandler.http_error_302(self, req, fp, code, msg, headers)
    744 fp.read()
    745 fp.close()
--> 747 return self.parent.open(new, timeout=req.timeout)

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:523, in OpenerDirector.open(self, fullurl, data, timeout)
    521 for processor in self.process_response.get(protocol, []):
    522     meth = getattr(processor, meth_name)
--> 523     response = meth(req, response)
    525 return response

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:632, in HTTPErrorProcessor.http_response(self, request, response)
    629 # According to RFC 2616, "2xx" code indicates that the client's
    630 # request was successfully received, understood, and accepted.
    631 if not (200 <= code < 300):
--> 632     response = self.parent.error(
    633         'http', request, response, code, msg, hdrs)
    635 return response

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:561, in OpenerDirector.error(self, proto, *args)
    559 if http_err:
    560     args = (dict, 'default', 'http_error_default') + orig_args
--> 561     return self._call_chain(*args)

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:494, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
    492 for handler in handlers:
    493     func = getattr(handler, meth_name)
--> 494     result = func(*args)
    495     if result is not None:
    496         return result

File ~/miniconda3/envs/astro/lib/python3.9/urllib/request.py:641, in HTTPDefaultErrorHandler.http_error_default(self, req, fp, code, msg, hdrs)
    640 def http_error_default(self, req, fp, code, msg, hdrs):
--> 641     raise HTTPError(req.full_url, code, msg, hdrs, fp)

HTTPError: HTTP Error 404: Not Found

[crhea93]

I believe the error is in the URL which is defined in datasets.tools.sql_query.py. It is set as PUBLIC_URL = 'http://cas.sdss.org/public/en/tools/search/x_sql.aspx'. However, that site does not exist. On the contrary, the following similar site does exist: PUBLIC_URL = 'http://cas.sdss.org/public/en/tools/search/sql.aspx'

[bsipocz]

@crhea93 - thanks for reporting this. I can reproduce it locally, and in fact wonder how this is not picked up by our cron tests.

[bsipocz]

Some of these base URLs have changed with more recent data releases, and it's not sufficient anymore to rely on the default DR, which atm resolves to dr17.

The easy fix is to hard-wire an earlier DR number in the URL, that would ensure we get the same dataset as before, or to finally get rid of the otherwise super stable direct API calls and rely on astroquery for these functions (so far we avoided that as everything was extremely stable with directly interacting with the SDSS).

[crhea93]

Hi there,
Are there still plans to fix this?

[jotavemonte]

Hi folks. I'm new to this repo, but I was investigating this issue and I was able to implement a fix to it.

Please let me know if it's compliant with the repo policies.

Cheers.

#279