sdw endpoints are restricted to results <= 100,000

[mps9506]

The effluent endpoint returns multipage data when more than 100,000 records are returned. The SDW endpoints return an error when more than 100,000 records are returned:

curl -X GET "https://echodata.epa.gov/echo/sdw_rest_services.get_systems?p_st=CA%2CGA%2CFL%2COH%2COR%2CTX%2CVA%2CWA" -H  "accept: application/json"

{
  "Results": {
    "Error": {
      "ErrorMessage": "Rows Returned would be 101380. Queryset Limit would be exceeded - please make search parameters more selective."
    }
  }
}

the get_download endpoint still returns a full csv for <100,000 rows. Would make sense return a message when queryset is exceeded and remove the code for parsing pages returned by get_qid endpoint.

[mps9506]

Notes:

• It appears EPA would prefer the *_get_facilities endpoint be used in conjunction with the get_download, get_qid endpoints.
• The "enhanced" *_get_facilities_info endpoints are supposed to be self contained, not to be used with get_qid even though they return a qid. On big queries the get_qid endpoint hangs and eventually returns a gateway error.

So to the point above,

• change the *_get_facilities_info endpoints to *_get_facilities;
• return invisible nulls and useful error messages from the server when queries are too large;
• possibly eliminate code related to get_qid and use get_download only.