Missing tiles are not correctly generated from other zoom level

[RicoHoff]

I am trying to use mapproxy to display a map with WMTS at zoom level 16. The tile source is only available from zoom level 12-15.

Mapproxy should calculate the missing tiles of zoom level 16 from zoom level 15. However, this does not work as expected. The problem exists independently of the tiles source.

I can't tell if it's a bug or I'm doing something wrong in my mapproxy.xaml.

The tiles for zoom level 16 are generated and also delivered, but the content of the tiles is probably not calculated correctly during generation and the tiles then no longer fit together. It looks like the tile/image from zoom level 15 is not scaled and this leads to wrong tiles in zoom level 16 and subsequently also in possible higher zoom levels.

Here is the example to follow. In zoom level 16 it goes wrong.

services:
  demo:
  wmts:
    restful: true
    restful_template: '/{Layer}/{TileMatrixSet}/{TileMatrix}/{TileCol}/{TileRow}.{Format}'
    kvp: true

layers:
  - name: kdr
    title: KDR
    sources: [kdr_cache]
    
caches:
  kdr_cache:
    disable_storage: true
    sources: [kdr_tiles]
    format: image/png
    upscale_tiles: 1
    concurrent_tile_creators: 2
  
sources:
  kdr_tiles:
     type: tile
     url: https://tiles.historic.place/kdr1893/%(z)s/%(x)s/%(y)s.png
     grid: kdr_grid
     coverage:
        clip: true
        intersection:
             - bbox: [1097335.1233980813995004,6479772.5293630501255393,1410253.6714844407979399,6739642.3423987086862326]
               srs: 'EPSG:900913'
     on_error:
        500:
         response: '#ede9e3'
         cache: False
        other:
         response: '#ede9e3'
         cache: False

grids:
  kdr_grid:
    base: GLOBAL_MERCATOR
    srs: EPSG:900913
    origin: nw
    num_levels: 16
 
globals:
  cache:
    meta_size: [4, 4]
    bulk_meta_tiles: true
    meta_buffer: 20
    lock_dir: './cache_data/locks'
    tile_lock_dir: './cache_data/tile_locks'
  http:
    ssl_no_cert_checks: true

[weskamm]

Try to remove the coverage section in the source. It may interfere with correct tile index calculation.
Any maybe defining resolutions ind the grid matching the source helps.

[JacobJeppesen]

I had the same issue, and it was fixed by removing the coverage section.

@weskamm, do you have an idea of where in the code the error is happening? Then I could take a look and see if I can find a way to solve it.

[weskamm]

This may be related to #687 and possibly fixed by #688
I am currently investigating this

[weskamm]

Ok its not related. But i found the problem.
You are missing the grid reference in your cache.
Add
grids: [kdr_grid]
to your kdr_cache and it should be working.

Please reopen if you feel there is still something wrong

[JacobJeppesen]

I couldn't get it to work with grids in the cache, so ended up spending some time putting together a test configuration (pasted below) to get an overview of what was going on. It seems like it's the clip in the coverage that results in blank tiles, which can be seen in the osm_coverage_clip_upscale layer. The osm_coverage_upscale layer is almost the same, but without any clipping, and works without any issues. As an example, when running the configuration, these two tiles should be the same:

http://localhost:8080/wmts/osm_coverage_upscale/osm_grid_upscale/20/218386/397713.png
http://localhost:8080/wmts/osm_coverage_clip_upscale/osm_grid_upscale/20/218386/397713.png

The first one works, but the second one is blank.

Let me know if you would like me to investigate something further. The configuration below contains some extra code I thought might be relevant, so I didn't remove it, although it made it a bit bloated 🙂

services:
  demo:
  wmts:
    restful: true
    restful_template: '/{Layer}/{TileMatrixSet}/{TileMatrix}/{TileCol}/{TileRow}.{Format}'
    kvp: true

layers:
  # Layers that work as expected
  - name: osm
    title: OpenStreetMap
    sources: [osm_cache]
  - name: osm_upscale  # Test: http://localhost:8080/wmts/osm_upscale/osm_grid_upscale/20/218386/397713.png
    title: OpenStreetMap
    sources: [osm_cache_upscale]
  - name: osm_coverage_upscale  # Test: http://localhost:8080/wmts/osm_coverage_upscale/osm_grid_upscale/20/218386/397713.png
    title: OpenStreetMap
    sources: [osm_cache_coverage_upscale]

  # Layer where we use the 'clip' coverage option. This breaks the upscaling (we just get blank tiles).
  - name: osm_coverage_clip_upscale  # Test: http://localhost:8080/wmts/osm_coverage_clip_upscale/osm_grid_upscale/20/218386/397713.png
    title: OpenStreetMap
    sources: [osm_cache_coverage_clip_upscale]

  # Layer where we use the 'osm_grid' grid in the source instead of max_res. This results in the tiles from zoom level
  # 18 being placed next to each other instead of being upscaled.
  - name: osm_upscale_from_custom_grid  # Test: http://localhost:8080/wmts/osm_upscale_from_custom_grid/osm_grid_upscale/20/218386/397713.png
    title: OpenStreetMap
    sources: [osm_cache_upscale_from_custom_grid]
    
  # Layer where we use the 'clip' coverage option, but then upscale from another cache. This makes upscaling work, but
  # breaks the coverage, as the second cache has no coverage provided. This would probably be fixed by
  # https://github.com/mapproxy/mapproxy/pull/691
  - name: osm_coverage_clip_upscale_from_osm_cache # Test: http://localhost:8080/wmts/osm_coverage_clip_upscale_from_osm_cache/osm_grid_upscale/20/218386/397713.png
    title: OpenStreetMap
    sources: [osm_cache_coverage_clip_upscale_from_cache]

caches:
  osm_cache:
    disable_storage: true
    sources: [osm_tiles]
    format: image/png
    grids: [osm_grid]
  osm_cache_upscale:
    disable_storage: true
    sources: [osm_tiles]
    format: image/png
    upscale_tiles: 4
    grids: [osm_grid_upscale]
  osm_cache_coverage_upscale:
    disable_storage: true
    sources: [osm_tiles_coverage]
    format: image/png
    upscale_tiles: 4
    grids: [osm_grid_upscale]
  osm_cache_coverage_clip_upscale:  # Doesn't work (tiles are blank at zoom levels > 18)
    disable_storage: true
    sources: [osm_tiles_coverage_clip]
    format: image/png
    upscale_tiles: 4
    grids: [osm_grid_upscale]
  osm_cache_upscale_from_custom_grid:  # Doesn't work (tiles are placed next to each other instead of being upscaled)
    disable_storage: true
    sources: [osm_tiles_custom_grid]
    format: image/png
    upscale_tiles: 4
    grids: [osm_grid_upscale]
  osm_cache_coverage_clip:  # Used as source for osm_cache_coverage_clip_upscale_from_cache
    disable_storage: true
    sources: [osm_tiles_coverage_clip]
    format: image/png
    grids: [osm_grid]
  osm_cache_coverage_clip_upscale_from_cache:  # Upscaling now works but coverage is broken (should be fixed by https://github.com/mapproxy/mapproxy/pull/691)
    disable_storage: true
    sources: [osm_cache_coverage_clip]
    format: image/png
    upscale_tiles: 4
    grids: [osm_grid_upscale]

sources:
  osm_tiles:
    type: tile
    url: https://tile.openstreetmap.org/%(z)s/%(x)s/%(y)s.png
    grid: GLOBAL_WEBMERCATOR
    max_res: 0.29858214173896974
  osm_tiles_custom_grid:
    type: tile
    url: https://tile.openstreetmap.org/%(z)s/%(x)s/%(y)s.png
    grid: osm_grid  # Used instead of max_res, but results in tiles being placed next to each other instead of being upscaled
  osm_tiles_coverage:
    type: tile
    url: https://tile.openstreetmap.org/%(z)s/%(x)s/%(y)s.png
    grid: GLOBAL_WEBMERCATOR
    max_res: 0.29858214173896974
    transparent: true
    coverage:
      bbox: [ -110, 35, -90, 45 ]
      srs: 'EPSG:4326'
  osm_tiles_coverage_clip:
    type: tile
    url: https://tile.openstreetmap.org/%(z)s/%(x)s/%(y)s.png
    grid: GLOBAL_WEBMERCATOR
    max_res: 0.29858214173896974
    transparent: true
    coverage:
      clip: true
      intersection:
        - bbox: [ -110, 35, -90, 45 ]
          srs: 'EPSG:4326'

grids:
  osm_grid:
    base: GLOBAL_WEBMERCATOR
    num_levels: 19  # I.e., max. zoom level is 18
  osm_grid_upscale:
    base: GLOBAL_WEBMERCATOR
    num_levels: 23  # I.e., max. zoom level is 22

globals:
  cache:
    meta_size: [1, 1]
    meta_buffer: 0

[weskamm]

Why don't you just define a bbox on the grid? Shouldn't that be enough?

[JacobJeppesen]

I normally use a polygon file for the coverage, but used a bbox in the code above to make it easier to test (got the blank tiles both with bbox and polygon files).

Is there a way to use a polygon file in the grid? Or do you have an idea of where the issue with blank tiles is coming from when using clip coverage?

[weskamm]

Is there a way to use a polygon file in the grid?

I dont't think so.

Or do you have an idea of where the issue with blank tiles is coming from when using clip coverage?

It was just an idea that you have a conflicting situation, where the grid defines its own extent while the source uses another one. I guess there is a bug or implementation missing in the part of source clipping, not respecting or wrongly calculating tile indexes for the underlying base grid

[JacobJeppesen]

Is there a way to use a polygon file in the grid?

I dont't think so.

Or do you have an idea of where the issue with blank tiles is coming from when using clip coverage?

It was just an idea that you have a conflicting situation, where the grid defines its own extent while the source uses another one. I guess there is a bug or implementation missing in the part of source clipping, not respecting or wrongly calculating tile indexes for the underlying base grid

Got it. Is it possible for you to point me in the direction of the relevant file(s) in the repo where the bug might be? Then I could take a look and see if I can fix it.

@shimoncohen, did you by any chance see this issue when implementing #691? I.e., enabling upscaling results in blank tiles at the zoom levels where the upscaling should be "active" when using coverage with clip to intersection (both for bbox and polygon file).

[shimoncohen]

@JacobJeppesen I haven't seen this issue while implementing #691.
I had a look, and it seems that in your situation the max_res defined in the osm_tiles_coverage_clip source is the issue. When removing this, the original issue occurres where tiles are incorrectly generated from last zoom_level.

In addition, I don't think supporting polygon file for a grid is wanted. Defining a coverage should be enough in my opinion.

[JacobJeppesen]

Got it. Thanks for taking a look.

[JacobJeppesen]

Just to give a short update on this issue, the problem disappears when the coverage is in the cache instead of the source. There is an example configuration here to try it out: #789 (comment)

However, I encountered another error. The upscaling sometimes return an entirely white tile, instead of the upscaled one. I haven't tested it extensively, but it happens for roughly 10% of the tiles when using a layer that natively goes to zoom level 18 and is upscaled to zoom level 22. I've also only tested it when using S3 for the cache. So far, I've narrowed it down to the line below returning a white tile in source.source, instead of the actual upscaled one:

mapproxy/mapproxy/cache/tile.py

Line 398 in 4315eca

source = self._query_sources(query)

I made quick hack (JacobJeppesen@9241634) to see if running the query again fixes it, and it does. So now all the tiles are correctly upscaled. However, this hack doesn't fix the actual problem, and isn't really a solution.

It's quite difficult to debug, as there are no error messages, and it only happens periodically. If anyone has any idea of where the issue might come from, please let me know. Then I can try to see if I can fix it 🙂