WPS exports fail for all file formats except CSV - GeoServer #7932

DavidQuartz · 2022-03-09T09:39:34Z

Description

CSV file export is currently the only option which does not fail in wps downloads. For the other file formats, a response like this is returned from geoserver

<ows:ExceptionReport xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:ows="http://www.opengis.net/ows/1.1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.1.0" xsi:schemaLocation="http://www.opengis.net/ows/1.1 https://gs-stable.geo-solutions.it/geoserver/schemas/ows/1.1.0/owsAll.xsd">
<ows:Exception exceptionCode="MissingParameterValue"  locator="request">
<ows:ExceptionText>
Could not determine geoserver request from http request org.geoserver.monitor.MonitorServletRequest@7470866c
</ows:ExceptionText>
</ows:Exception>
</ows:ExceptionReport>

How to reproduce

Open this map
Open TOC
Select the 'province_toscanna' layer and click on the export button
Choose any file format apart from CSV
Click on export

Expected Result

The export is successful

Current Result

The export fails

Not browser related

Browser info

(use this site: https://www.whatsmybrowser.org/ for non expert users)

Browser Affected	Version
Internet Explorer
Edge
Chrome
Firefox
Safari

Other useful information

This issue may be from the backend.
Geoserver version: 2.20 SNAPSHOT

The text was updated successfully, but these errors were encountered:

tdipisa · 2022-03-09T11:21:19Z

@DavidQuartz did you already check MS is performing a correct request? Can we effectively exclude bugs MS side for this? If yes please explain why.

Assuming there is no bug MS side with the request, we have to provide to devops some sample of requests here to make them able to directly reproduce the issue by performing them (eg. some CURL requests with different formats that are not working along with the one using CSV that is working instead).

DavidQuartz · 2022-03-09T11:42:44Z

@tdipisa the bug on MS side related to download so far concerns exports of layers with spatial filters. The fix is provided here. However, downloading a layer without any filters applied should not be affected by the bug.

I will include 3 cURL examples (Shapefile, Geojson, and CSV)

cURL request for a shapefile -- unsuccessful

curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=DescribeProcess&IDENTIFIER=gs%3ADownloadEstimator%2Cgs%3ADownload&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/json' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --compressed ;
curl 'https://dev-mapstore.geosolutionsgroup.com/mapstore/rest/geostore/session/refresh/c8e01475-6548-42c9-87a4-237b5012661b/b52c152d-574e-4976-b781-6d9574bd3f4a' \
  -X 'POST' \
  -H 'Connection: keep-alive' \
  -H 'Content-Length: 0' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'DNT: 1' \
  -H 'Authorization: Bearer c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: https://dev-mapstore.geosolutionsgroup.com/mapstore/' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  -H 'Cookie: exportDataResults_38133=%7B%22results%22%3A%5B%7B%22id%22%3A%2255eae250-9f8b-11ec-82f8-0bd875b8da7a%22%2C%22layerName%22%3A%22geosolutions%3Aprovince_toscana%22%2C%22layerTitle%22%3A%22province_toscana%22%2C%22status%22%3A%22failed%22%2C%22startTime%22%3A1646818133844%2C%22result%22%3A%7B%22msgId%22%3A%22layerdownload.wpsExecuteError.badResponse%22%2C%22msgParams%22%3A%7B%22eCode%22%3A%22UnexpectedProcessStatus%22%7D%7D%7D%2C%7B%22id%22%3A%225d2948e0-9f8b-11ec-82f8-0bd875b8da7a%22%2C%22layerName%22%3A%22geosolutions%3Aprovince_toscana%22%2C%22layerTitle%22%3A%22province_toscana%22%2C%22status%22%3A%22completed%22%2C%22startTime%22%3A1646818145729%2C%22result%22%3A%22https%3A%2F%2Fgs-stable.geo-solutions.it%2Fgeoserver%2Fows%3Fservice%3DWPS%26version%3D1.0.0%26request%3DGetExecutionResult%26executionId%3Db000a9da-3a2d-43fb-a3e1-33e88bd69cb4%26outputId%3Dresult.csv%26mimetype%3Dtext%252Fcsv%26authkey%3Dc8e01475-6548-42c9-87a4-237b5012661b%22%7D%2C%7B%22id%22%3A%2264003b10-9f8b-11ec-82f8-0bd875b8da7a%22%2C%22layerName%22%3A%22geosolutions%3Aprovince_toscana%22%2C%22layerTitle%22%3A%22province_toscana%22%2C%22status%22%3A%22failed%22%2C%22startTime%22%3A1646818157191%2C%22result%22%3A%7B%22msgId%22%3A%22layerdownload.wpsExecuteError.badResponse%22%2C%22msgParams%22%3A%7B%22eCode%22%3A%22UnexpectedProcessStatus%22%7D%7D%7D%2C%7B%22id%22%3A%22244ff360-9f8c-11ec-82f8-0bd875b8da7a%22%2C%22layerName%22%3A%22geosolutions%3Aprovince_toscana%22%2C%22layerTitle%22%3A%22province_toscana%22%2C%22status%22%3A%22failed%22%2C%22startTime%22%3A1646818479851%2C%22result%22%3A%7B%22msgId%22%3A%22layerdownload.wpsExecuteError.badResponse%22%2C%22msgParams%22%3A%7B%22eCode%22%3A%22UnexpectedProcessStatus%22%7D%7D%7D%2C%7B%22id%22%3A%224b585240-9f8c-11ec-8016-c1b73c3ce0c4%22%2C%22layerName%22%3A%22geosolutions%3Aprovince_toscana%22%2C%22layerTitle%22%3A%22province_toscana%22%2C%22status%22%3A%22failed%22%2C%22startTime%22%3A1646818545364%2C%22result%22%3A%7B%22msgId%22%3A%22layerdownload.wpsExecuteError.badResponse%22%2C%22msgParams%22%3A%7B%22eCode%22%3A%22UnexpectedProcessStatus%22%7D%7D%7D%2C%7B%22id%22%3A%220af40990-9f9d-11ec-ab1f-990aebcd3db0%22%2C%22layerName%22%3A%22geosolutions%3Aprovince_toscana%22%2C%22layerTitle%22%3A%22province_toscana%22%2C%22status%22%3A%22failed%22%2C%22startTime%22%3A1646825738993%2C%22result%22%3A%7B%22msgId%22%3A%22layerdownload.wpsExecuteError.badResponse%22%2C%22msgParams%22%3A%7B%22eCode%22%3A%22UnexpectedProcessStatus%22%7D%7D%7D%2C%7B%22id%22%3A%2267f14630-9f9d-11ec-ab1f-990aebcd3db0%22%2C%22layerName%22%3A%22geosolutions%3Aprovince_toscana%22%2C%22layerTitle%22%3A%22province_toscana%22%2C%22status%22%3A%22completed%22%2C%22startTime%22%3A1646825895065%2C%22result%22%3A%22https%3A%2F%2Fgs-stable.geo-solutions.it%2Fgeoserver%2Fows%3Fservice%3DWPS%26version%3D1.0.0%26request%3DGetExecutionResult%26executionId%3D225ea51a-44a1-41cb-8803-5dd94ff99020%26outputId%3Dresult.csv%26mimetype%3Dtext%252Fcsv%26authkey%3Dc8e01475-6548-42c9-87a4-237b5012661b%22%7D%5D%7D; SL_C_23361dd035530_VID=ZbcEtUme1L; SL_C_23361dd035530_KEY=1e81b1058a277e08b89ce1a8afae7c273bf43218; _gcl_au=1.1.654306417.1644505303; _ga=GA1.2.1529684590.1644505305; SL_C_23361dd035530_SID=0I0fLiMUgX' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=Execute&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/json, text/plain, */*' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'content-type: application/xml' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:DownloadEstimator</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>layerName</ows:Identifier><wps:Data><wps:LiteralData>geosolutions:province_toscana</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs></wps:Execute>' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=Execute&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/xml, application/zip' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'content-type: application/xml' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:Download</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>layerName</ows:Identifier><wps:Data><wps:LiteralData>geosolutions:province_toscana</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>outputFormat</ows:Identifier><wps:Data><wps:LiteralData>application/zip</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>cropToROI</ows:Identifier><wps:Data><wps:LiteralData>false</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs><wps:ResponseForm><wps:ResponseDocument storeExecuteResponse="true" status="true"><wps:Output mimeType="application/zip" asReference="true"><ows:Identifier>result</ows:Identifier></wps:Output></wps:ResponseDocument></wps:ResponseForm></wps:Execute>' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=GetExecutionStatus&executionId=dc6e912a-53ad-4b75-94b7-76ba76648a2c&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/xml' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --compressed

cURL for Geojson --unsuccesful

  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/json' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=Execute&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/json, text/plain, */*' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'content-type: application/xml' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:DownloadEstimator</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>layerName</ows:Identifier><wps:Data><wps:LiteralData>geosolutions:province_toscana</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs></wps:Execute>' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=Execute&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/xml, application/json' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'content-type: application/xml' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:Download</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>layerName</ows:Identifier><wps:Data><wps:LiteralData>geosolutions:province_toscana</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>outputFormat</ows:Identifier><wps:Data><wps:LiteralData>application/json</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>cropToROI</ows:Identifier><wps:Data><wps:LiteralData>false</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs><wps:ResponseForm><wps:ResponseDocument storeExecuteResponse="true" status="true"><wps:Output mimeType="application/json" asReference="true"><ows:Identifier>result</ows:Identifier></wps:Output></wps:ResponseDocument></wps:ResponseForm></wps:Execute>' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=GetExecutionStatus&executionId=f3bb97f6-2237-4e27-b447-b572a8cb08b3&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/xml' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --compressed

cURL for CSV -- successful

curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=DescribeProcess&IDENTIFIER=gs%3ADownloadEstimator%2Cgs%3ADownload&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/json' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=Execute&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/json, text/plain, */*' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'content-type: application/xml' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:DownloadEstimator</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>layerName</ows:Identifier><wps:Data><wps:LiteralData>geosolutions:province_toscana</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs></wps:Execute>' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=Execute&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/xml, text/csv' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'content-type: application/xml' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:Download</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>layerName</ows:Identifier><wps:Data><wps:LiteralData>geosolutions:province_toscana</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>outputFormat</ows:Identifier><wps:Data><wps:LiteralData>text/csv</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>cropToROI</ows:Identifier><wps:Data><wps:LiteralData>false</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs><wps:ResponseForm><wps:ResponseDocument storeExecuteResponse="true" status="true"><wps:Output mimeType="text/csv" asReference="true"><ows:Identifier>result</ows:Identifier></wps:Output></wps:ResponseDocument></wps:ResponseForm></wps:Execute>' \
  --compressed ;
curl 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=GetExecutionStatus&executionId=225ea51a-44a1-41cb-8803-5dd94ff99020&authkey=c8e01475-6548-42c9-87a4-237b5012661b' \
  -H 'authority: gs-stable.geo-solutions.it' \
  -H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
  -H 'accept: application/xml' \
  -H 'dnt: 1' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36' \
  -H 'origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'sec-fetch-site: cross-site' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://dev-mapstore.geosolutionsgroup.com/' \
  -H 'accept-language: en-US,en;q=0.9' \
  --compressed

randomorder · 2022-03-09T13:48:33Z

I investigated a bit the shapefile download. It's succeding on the GeoServer side but MS is throwing an error. Not sure why @tdipisa @DavidQuartz

Looks like a problem with the Getstatus call. Manually copy-pasting the link in the browser works but in the developer tools the request is failing:

Copy-pastse of the result URL

GetExecutionStatus

Nothing besides the exception in tthe logs to my eyes
geoserver.log

allyoucanmap · 2022-03-15T16:20:29Z

@tdipisa the download process is failing randomly, in some cases also the CSV format is failing.

Test done client side:

increased interval time in between status requests
replaced the uppercase REQUEST to lowercase request param
used the statusLocation from the ExceptionReport tag

all test are still failing but if we try to open the same execution status in a different tab the response seems succeful.

tdipisa · 2022-03-15T17:04:49Z

@tdipisa the download process is failing randomly, in some cases also the CSV format is failing.

Test done client side:

increased interval time in between status requests

replaced the uppercase REQUEST to lowercase request param

used the statusLocation from the ExceptionReport tag

all test are still failing but if we try to open the same execution status in a different tab the response seems succeful.

@allyoucanmap can you effectively exclude a problem client side then? Should we ask the GeoServer team for an investigation WPS side? It seems yes to me, please confirm.

allyoucanmap · 2022-03-15T17:09:10Z

@allyoucanmap can you effectively exclude a problem client side then? Should we ask the GeoServer team for an investigation WPS side? It seems yes to me, please confirm.

yes, I confirm that a server side investigation is needed

tdipisa · 2022-03-15T17:12:32Z

@taba90 I will get in touch with you for this according to the current schedule, there's no immediate rush with this.

aaime · 2022-03-25T14:40:59Z

Could reproduce, and then asked @randomorder support to debug, modifying the web.xml so that we get HTTP request, as seen by GeoServer, logged.
On restart GeoFence on H2 started misbehaving, complaining about multiple connections attempt at the same H2 database, we tried to clean up the log files (with GEoServer stopped), re-create the H2 database, even migrating to postgis... nothing worked.

Unfortunately we cannot make progress on the actual topic of this issue one until the GeoFence issue is solved, because it prevents all access to data due to the rule database not being available.... @randomorder is still looking into it...

randomorder · 2022-03-25T15:12:51Z

Ok geoserver is starting now. We are still investigating
I've opened a dedicated issue for GeoFence dependencies here

aaime · 2022-03-25T15:15:00Z

@randomorder managed to get GetFence work, resuming investigation.

tdipisa · 2022-03-25T15:24:39Z

Thank you @randomorder @aaime. I'm moving the issue in Blocked for the MapStore board then. I'm looking forward to receiving news on this.

aaime · 2022-03-25T17:17:32Z

Found where the issue is... and it's been epic, required debugging Tomcat own source code!

Here is what happens:

An Asynch Execute request is sent, for the download. A thread is allocated to handle it, which receives a copy of all common thread locals, including Dispatcher.REQUEST, which in turn contains the HttpServletRequest
The HTTP request above finishes, and Tomcat prepares its internal Request object for reclying, resetting all flags and contents, including the parsed KVP map! (I would not have imagined)
The asynch execution terminates and WPS saves the final status of the response, which includes an XML document that has backlinks. The LanguageURLMangler is invoked, which peeks into Dispatcher.REQUEST.getHttpRequest(), causing the recycled request to do a partial initialization, against an empty query string (it was cleared) and decide it has no parameters
The client does the GET request for GetExecutionStatus, which guess what, gets the recycled request. Its parameter map is not parsed again, because the flags say it's parsed and empty. The dispatch fails because GeoServer sees a request with no KVP parameters at all!

General problem: we should not be using HTTPServletRequest in a background thread, its state is valid only as long as the original request is in flight.

I have searched in Tomcat for a setting that would make it stop recyling the requests, and found one, but it just stops recyling of the "request facade", and when enabled, caused the asynch process to just fail with:

java.lang.IllegalStateException: The request object has been recycled and is no longer associated with this facade
	at org.apache.catalina.connector.RequestFacade.getParameterNames(RequestFacade.java:391)
	at javax.servlet.ServletRequestWrapper.getParameterNames(ServletRequestWrapper.java:178)
	at javax.servlet.ServletRequestWrapper.getParameterNames(ServletRequestWrapper.java:178)
	at javax.servlet.ServletRequestWrapper.getParameterNames(ServletRequestWrapper.java:178)
	at javax.servlet.ServletRequestWrapper.getParameterNames(ServletRequestWrapper.java:178)
	at javax.servlet.ServletRequestWrapper.getParameterNames(ServletRequestWrapper.java:178)
	at org.geoserver.ows.LanguageURLMangler.mangleURL(LanguageURLMangler.java:27)
	at org.geoserver.ows.util.ResponseUtils.buildURL(ResponseUtils.java:392)
	at org.geoserver.wps.executor.ExecuteResponseBuilder.build(ExecuteResponseBuilder.java:113)
	at org.geoserver.wps.executor.WPSExecutionManager$Executor.execute(WPSExecutionManager.java:471)
	at org.geoserver.wps.executor.WPSExecutionManager$Executor.call(WPSExecutionManager.java:380)
	at org.geoserver.wps.executor.WPSExecutionManager$Executor.call(WPSExecutionManager.java:309)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

The quick fix is to make LanguageURLMangler (written by Yalin) to stop using the HttpServletRequest, and use the Request rawKvp instead (which is detached from the original HTTP request). However, mind, this could come up in the future in different shapes... to make sure we won't get bitten, we should be doing a deep copy of HttpServletRequest while we pass Dispatcher.REQUEST down to the asynch execution pool (which... is not trivial).

@simboss shall I go for the quick fix? The quick fix is a matter of hours, including backports, a fuller fix with deep cloning of the HTTP request is likely 1-2 days.

nmco · 2022-04-07T16:17:14Z

@taba90 | @fostash started to work on this today:

the quick fix should be ready tomorrow
the long term one mid next week

@taba90 | @fostash please update the related issues when you start working on them.

tdipisa · 2022-04-28T07:54:01Z

@ElenaGallo a fix have been provided for this issue in GS-STABLE. Can you please retest it with MS DEV and close the issue if all is working properly?

giohappy · 2022-05-26T08:20:09Z

@ElenaGallo a fix have been provided for this issue in GS-STABLE. Can you please retest it with MS DEV and close the issue if all is working properly?

@tdipisa it's not clear to me if, what and where the fix has been done. The epic is closed but the underlying (Geoserver) issues are still in backlog. What do you mean with "a fix has been provided for GS-STABLE"?

#8052 references geoserver/geoserver#5832 which reports that the fix has been backported to 2.21.x.

I guess both #8052 and #8053 can be closed in that case.

tdipisa · 2022-05-26T08:48:46Z

@giohappy I've missed the fact that the issue was converted into an epic by @nmco. Anyway, as far as I understand (correct me if I'm wrong @nmco), issues connected to this epic (as well as the overall problem) are related to GeoServer and not to MS so these issues should not be there. MS is not the good place to host GS issues :)

@nmco let me know anyway if I've missed something in the process.

DavidQuartz added the bug label Mar 9, 2022

DavidQuartz assigned tdipisa Mar 9, 2022

tdipisa assigned randomorder and unassigned tdipisa Mar 9, 2022

tdipisa added DevOps Priority: Medium labels Mar 9, 2022

tdipisa assigned allyoucanmap Mar 14, 2022

tdipisa assigned taba90 and unassigned randomorder and allyoucanmap Mar 15, 2022

DavidQuartz mentioned this issue Mar 17, 2022

"FeatureType Grid Widget" returns empty geometries when a spatial filter is set GeoNode/geonode-mapstore-client#852

Closed

nmco added the investigation label Mar 18, 2022

simboss assigned aaime and unassigned taba90 Mar 23, 2022

simboss added estimate needed Internal labels Mar 23, 2022

simboss changed the title ~~WPS exports fail for all file formats except CSV~~ WPS exports fail for all file formats except CSV - GeoServer Mar 23, 2022

aaime removed the investigation label Mar 25, 2022

aaime removed the estimate needed label Mar 25, 2022

nmco added the Epic label Apr 5, 2022

nmco mentioned this issue Apr 5, 2022

Implement fix for WPS exports fail for all file formats except CSV - GeoServer #8053

Open

simboss assigned nmco and unassigned aaime Apr 6, 2022

ale-cristofori added the BackportNeeded Commits provided for an issue need to be backported to the milestone's stable branch label Apr 8, 2022

tdipisa removed the BackportNeeded Commits provided for an issue need to be backported to the milestone's stable branch label Apr 8, 2022

ElenaGallo self-assigned this Apr 28, 2022

ElenaGallo added the Accepted label Apr 28, 2022

ElenaGallo closed this as completed Apr 28, 2022

MV88 added the current-release label May 3, 2022

MV88 added this to the v2022.01.01 milestone May 3, 2022

MV88 removed the current-release label May 3, 2022

ElenaGallo mentioned this issue May 5, 2022

#1945 Make FeatureGrid attributes visibility selector counted when user exports grid data #7845

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WPS exports fail for all file formats except CSV - GeoServer #7932

WPS exports fail for all file formats except CSV - GeoServer #7932

DavidQuartz commented Mar 9, 2022 •

edited

tdipisa commented Mar 9, 2022 •

edited

DavidQuartz commented Mar 9, 2022 •

edited

randomorder commented Mar 9, 2022

allyoucanmap commented Mar 15, 2022 •

edited

tdipisa commented Mar 15, 2022

allyoucanmap commented Mar 15, 2022

tdipisa commented Mar 15, 2022

aaime commented Mar 25, 2022

randomorder commented Mar 25, 2022

aaime commented Mar 25, 2022

tdipisa commented Mar 25, 2022

aaime commented Mar 25, 2022 •

edited

nmco commented Apr 7, 2022

tdipisa commented Apr 28, 2022

giohappy commented May 26, 2022 •

edited

tdipisa commented May 26, 2022 •

edited

WPS exports fail for all file formats except CSV - GeoServer #7932

WPS exports fail for all file formats except CSV - GeoServer #7932

Comments

DavidQuartz commented Mar 9, 2022 • edited

Description

How to reproduce

Other useful information

tdipisa commented Mar 9, 2022 • edited

DavidQuartz commented Mar 9, 2022 • edited

cURL request for a shapefile -- unsuccessful

cURL for Geojson --unsuccesful

cURL for CSV -- successful

randomorder commented Mar 9, 2022

allyoucanmap commented Mar 15, 2022 • edited

tdipisa commented Mar 15, 2022

allyoucanmap commented Mar 15, 2022

tdipisa commented Mar 15, 2022

aaime commented Mar 25, 2022

randomorder commented Mar 25, 2022

aaime commented Mar 25, 2022

tdipisa commented Mar 25, 2022

aaime commented Mar 25, 2022 • edited

nmco commented Apr 7, 2022

tdipisa commented Apr 28, 2022

giohappy commented May 26, 2022 • edited

tdipisa commented May 26, 2022 • edited

DavidQuartz commented Mar 9, 2022 •

edited

tdipisa commented Mar 9, 2022 •

edited

DavidQuartz commented Mar 9, 2022 •

edited

allyoucanmap commented Mar 15, 2022 •

edited

aaime commented Mar 25, 2022 •

edited

giohappy commented May 26, 2022 •

edited

tdipisa commented May 26, 2022 •

edited