data loss on getmap requests when vector data is reprojected EPSG:2056 to EPSG:4326

[ltclm]

Expected behavior and actual behavior.

When zooming into a reprojected polygon layer we expect that all the features are displayed in all scales. The original data and the mapfile configuration is in EPSG:2056. The strange behaviour and the data loss has been observed when requesting the map in EPSG:4326.

Steps to reproduce the problem.

Using the mapfile config [1] and the data index.shp from the attachment [2], start zooming into the center of the 4 polygons in EPSG:4326. After a certain zoom/scale some data gets lost. The same effect occurs with tiled wms queries (256x256), but much earlier.

Attach simple test case (drag/drop it here)

here are some screenshots and getmap requests of the observed data loss with EPSG:4326 GetMap Requests:
http://localhost:9178/local/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=47.70122100713624036,8.631010777621343166,47.73459661396240961,8.673423389810198003&CRS=EPSG:4326&WIDTH=1591&HEIGHT=1252&LAYERS=test&STYLES=&FORMAT=image/jpeg&DPI=96&MAP_RESOLUTION=96&FORMAT_OPTIONS=dpi:96

service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: CGI Request 4 on process 10071
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap(): rendering using outputformat named jpeg (AGG/JPEG).
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap(): WMS/WFS set-up and query, 0.000s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap(): Layer 0 (test), 0.016s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap(): Drawing Label Cache, 0.000s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap() total time: 0.017s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msSaveImage(stdout) total time: 0.014s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: mapserv request processing time (msLoadMap not incl.): 0.031s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msFreeMap(): freeing map at 0x562ec0ee5e50.
service-wms_1  | 172.24.0.1 - - [26/Jan/2024:07:23:39 +0000] "GET /local/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=47.70122100713624036,8.631010777621343166,47.73459661396240961,8.673423389810198003&CRS=EPSG:4326&WIDTH=1591&HEIGHT=1252&LAYERS=test&STYLES=&FORMAT=image/jpeg&DPI=96&MAP_RESOLUTION=96&FORMAT_OPTIONS=dpi:96 HTTP/1.1" 200 25612

http://localhost:9178/local/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=47.70991146067085253,8.640854182430102171,47.72659926408394426,8.662060488524529589&CRS=EPSG:4326&WIDTH=1591&HEIGHT=1253&LAYERS=test&STYLES=&FORMAT=image/jpeg&DPI=96&MAP_RESOLUTION=96&FORMAT_OPTIONS=dpi:96

service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: CGI Request 5 on process 10071
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap(): rendering using outputformat named jpeg (AGG/JPEG).
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap(): WMS/WFS set-up and query, 0.000s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap(): Layer 0 (test), 0.038s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap(): Drawing Label Cache, 0.000s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msDrawMap() total time: 0.039s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msSaveImage(stdout) total time: 0.021s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: mapserv request processing time (msLoadMap not incl.): 0.061s
service-wms_1  |  [warn] [pid 10016] mod_fcgid: stderr: msFreeMap(): freeing map at 0x562ec0ee5e50.
service-wms_1  | 172.24.0.1 - - [26/Jan/2024:07:25:10 +0000] "GET /local/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=47.70991146067085253,8.640854182430102171,47.72659926408394426,8.662060488524529589&CRS=EPSG:4326&WIDTH=1591&HEIGHT=1253&LAYERS=test&STYLES=&FORMAT=image/jpeg&DPI=96&MAP_RESOLUTION=96&FORMAT_OPTIONS=dpi:96 HTTP/1.1" 200 19836

http://localhost:9178/local/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=47.7172290358096447,8.65006363064953554,47.71931501123628294,8.652714418911338967&CRS=EPSG:4326&WIDTH=1591&HEIGHT=1253&LAYERS=test&STYLES=&FORMAT=image/jpeg&DPI=96&MAP_RESOLUTION=96&FORMAT_OPTIONS=dpi:96

service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: CGI Request 6 on process 10071
service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: msDrawMap(): rendering using outputformat named jpeg (AGG/JPEG).
service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: msDrawMap(): WMS/WFS set-up and query, 0.000s
service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: msDrawMap(): Layer 0 (test), 0.020s
service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: msDrawMap(): Drawing Label Cache, 0.000s
service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: msDrawMap() total time: 0.021s
service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: msSaveImage(stdout) total time: 0.018s
service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: mapserv request processing time (msLoadMap not incl.): 0.039s
service-wms_1  |  [warn] [pid 10015] mod_fcgid: stderr: msFreeMap(): freeing map at 0x562ec0ee5e50.
service-wms_1  | 172.24.0.1 - - [26/Jan/2024:07:25:52 +0000] "GET /local/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&BBOX=47.7172290358096447,8.65006363064953554,47.71931501123628294,8.652714418911338967&CRS=EPSG:4326&WIDTH=1591&HEIGHT=1253&LAYERS=test&STYLES=&FORMAT=image/jpeg&DPI=96&MAP_RESOLUTION=96&FORMAT_OPTIONS=dpi:96 HTTP/1.1" 200 18001

Operating system

We are operating mapserver on aws/kubernetes using this docker image from dockerhub: camptocamp/mapserver:7.6-gdal3.3. The docker image is based on Ubuntu 20.04.6 LTS

MapServer version and installation method

mapserv -v
MapServer version 7.6.5 OUTPUT=PNG OUTPUT=JPEG OUTPUT=KML SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=CAIRO SUPPORTS=SVG_SYMBOLS SUPPORTS=RSVG SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=GEOS SUPPORTS=POINT_Z_M SUPPORTS=PBF INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE
root@1b9c4dc20d26:/# proj --version
Rel. 8.2.0, November 1st, 2021
root@1b9c4dc20d26:/# gdal_translate --version
GDAL 3.3.3, released 2021/10/25

the same behaviour has been observed with mapserver 8.0 / gdal 3.6 and mapserver 8.0 / gdal 3.8
[1] mapfile

MAP
  NAME "WMS-BGDI"

  EXTENT 2100000 1030000 2900000 1400000
  SHAPEPATH "/var/local/geodata"

  PROJECTION
    "init=epsg:2056"
  END

  WEB
    METADATA
      "wms_title"   "WMS-BGDI"
      "wms_srs"     "EPSG:2056 EPSG:21781 EPSG:4326 EPSG:31466 EPSG:31467 EPSG:31468 EPSG:25832 EPSG:27582 EPSG:26591 EPSG:26592 EPSG:900913"
      "wms_extent"  "2100000 1030000 2900000 1400000"
    END
  END

  CONFIG "MS_ERRORFILE" stderr
  CONFIG "CPL_DEBUG" "ON"
  CONFIG "CPL_CURL_VERBOSE" "ON"
  CONFIG "CPL_TIMESTAMP" "ON"
  CONFIG "PROJ_DEBUG" "ON"
  DEBUG 5

  LAYER
    NAME "test"
    TYPE POLYGON
    STATUS ON
    DATA bund/swisstopo/swissimage/test/index.shp
    COMPOSITE
        OPACITY 40
    END
    METADATA
        "wms_enable_request" "*"
        "wms_extent"  "2100000 1030000 2900000 1400000"
    END
    CLASS
      STYLE
        OUTLINECOLOR 255 0 0
        COLOR 255 255 0
        WIDTH 5
      END
    END
    PROJECTION
        "init=epsg:2056"
    END
  END
END

[2]
data.zip

[geographika]

@ltclm - thanks for providing a fully working test case. I can confirm the issue still exists in the MapServer main branch.
The issue is that the Map extent is modified at various points in the WMS code, and then the map drawing code. At some point the search rectangle used for selecting features is moved from the extent passed in the original request.

This can be seen in the image below. The orange area is the original request extent, and the pink is the extent used to select the underlying features. As you can see in the image only the lower two features intersect, and so are the only features drawn.

It is harder to pin down which of the various modifications to the extent is the cause of the shift, and why this may be necessary.

[geographika]

This appears to be a projection issue, and possibly related to #6479 and #6478.

Setting int apply_over = MS_FALSE; in the following code brings back the correct results:

MapServer/src/mapproject.c

Line 1902 in da769fc

int apply_over = MS_TRUE;

[ltclm]

thank you @geographika for your response, the insights and the link to the related issues 👍
i can confirm that changing the projection of the layer from

"init=epsg:2056"

to

"+proj=somerc +lat_0=46.9524055555556 +lon_0=7.43958333333333 +k_0=1 +x_0=2600000 +y_0=1200000 +ellps=bessel +towgs84=674.374,15.056,405.346,0,0,0,0 +units=m +no_defs +type=crs"

as proposed here #6478 (comment) is fixing the projection issue for the reprojected getmap requests.

But with the new projection settings at layer level some getmap requests in the native data projection are not working anymore and an empty map is returned:

service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: CGI Request 4 on process 10107
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap(): rendering using outputformat named jpeg (AGG/JPEG).
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap(): WMS/WFS set-up and query, 0.000s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap(): Layer 0 (test), 0.013s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap(): Drawing Label Cache, 0.000s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msDrawMap() total time: 0.013s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msSaveImage(stdout) total time: 0.000s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: mapserv request processing time (msLoadMap not incl.): 0.014s
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: msFreeMap(): freeing map at 0x560b568291d0.
service-wms_1  |  [warn] [pid 10052] mod_fcgid: stderr: freeLayer(): freeing layer at 0x560b56831e30.
service-wms_1  | 172.24.0.1 - - [12/Feb/2024:16:26:55 +0000] "GET /local/?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image/jpeg&TRANSPARENT=false&LAYERS=test&WIDTH=256&HEIGHT=256&CRS=EPSG:2056&STYLES=&TIME=current&BBOX=2690771.2,1285872.0,2690796.8,1285897.6 HTTP/1.1" 200 993

[mizzPpY]

Hi @ltclm ,
this comment is not related to the original issue, but I would like to know how does your mapserver dockerfile and deployment file looks like?

I was using camptocamp/mapserver:7.4 image deploy on kubernetes cluster, it was working fine but encounter issue updating to 8.0. When curl http://localhost it is returning status 500 instead of the usual response "No query information to decode. QUERY_STRING is set, but empty.".

Had been looking everywhere what is causing the upgrade failure but couldn't find any answer, since I saw you using the same image and able to at least get something, I hope you can provide some insight for me.
Thank you.