Convert Mapbox Streets V6 example style to flat format #15413

jahow · 2023-12-08T12:24:47Z

This is a draft PR that aims at converting the existing Mapbox Streets v6 style used in some examples to the flat style format.

See the working WebGL vector tiles example

Achieving this requires implementing the following:

Reading from style variables in canvas renderer (discussion ongoing in Add consistent support for variables in styling #15146)
Improving CPU expression compiler to support geometry-type and dynamic var names
Support for concat and dynamic var names in GPU expression compiler
WebGL Renderer: apply the style filter before processing features in order to make buffers much (much) smaller
Allow evaluating expressions on the CPU with a partially-unknown context
- Evaluation contexts are now initialized with every property set to UNKNOWN_VALUE; if an operator needs to read from one of these, it will also return UNKNOWN_VALUE; eventually, this means expressions can be evaluated on various contexts (feature available or not, resolution available or not etc.)
- This is essential for WebGL Rendering as some expressions can then be processed on the CPU (e.g. string operations and style filter)

I will break this into several separate PRs in order to progress towards this goal.

Note: icons are not loaded as separate images any more, so the maki icons were compiled into a sprite sheet using https://github.com/flother/spreet.

Evaluation contexts are now initialized with every properties set to UNKNOWN_VALUE. If an expression needs an input which is unknown, it will return UNKNOWN_VALUE.

This will be used to filter out features that will eventually be rendered

…e types Previously the style parser was having this responsibility. Now reading from feature attributes and style variables is done in the gpu compiler logic

This avoids a crash if a prop or var is not defined at runtime

This required compiling all the maki icons to a spritesheet Done using https://github.com/flother/spreet The MapboxStreetsV6 style was converted with partial automation

This was also taken into account for the cpu and gpu compilers. The GPU compiler uses the CPU compiler to evaluate var/get names at compile time

This operator is evaluated on the CPU at compile time, which means that it won't have access to runtime values (feature props, variables etc.)

github-actions · 2023-12-08T15:46:38Z

📦 Preview the website for this branch here: https://deploy-preview-15413--ol-site.netlify.app/.

ahocevar

I added a few remarks about geometry-type. The rest looks good.

Just a note on the streets-v6 example style: Use caution, I think it is not the best reference for what needs to be supported - it was a very early attempt a long time ago to get Mapbox data rendered with a minimal sensible style, and its origin was a loose manual translation of some layers of the old streets-v6 Mapbox style.

ahocevar · 2023-12-09T17:51:36Z

examples/resources/mapbox-streets-v6-style.js

+        'all',
+        ['==', ['get', 'layer'], 'aeroway'],
+        ['==', ['geometry-type'], 'Polygon'],


Shouldn't this work implicitly, without the geometry-type filter? I'd expect when I specify a style with a fill-color that it's applied to a polygon. Other geometry types cannot be styled with a fill-color anyway.

I had been thinking we would add things like hasFill, hasStroke, and hasPoint to the parsing context. Then we could skip points and lines, for example, in cases where a style only had fill related properties. I added a similar comment to #15416.

That sounds good but the parsing context exists for an expression, not a whole style, so this would have to be done somewhere else. Maybe there could be hasFill() etc. utility functions for flat styles?

This filtering would be a great performance improvement, I definitely agree.

ahocevar · 2023-12-09T17:52:24Z

examples/resources/mapbox-streets-v6-style.js

+      ],
+      style: {
+        'fill-color': '#f0ede9',


This looks suspicious. A fill-color cannot be applied to LineString geometry, right?

Yes absolutely! I'll fix it

ahocevar · 2023-12-09T17:53:28Z

examples/resources/mapbox-streets-v6-style.js

+          ['==', ['get', 'class'], 'street_limited'],
+        ],
+        ['==', ['geometry-type'], 'LineString'],


Same as above with Polygon: stroke-color and stroke-width can only be applied to a LineString, so there should not be a need to filter by geometry-type.

In this case, a polygon or a linestring could have a stroke applied, so I think the geometry type filter is useful here (though I'm not sure if this data has street polygons in the "road" layer).

tschaub · 2023-12-09T22:27:47Z

@jahow - Thanks for taking this on. Having more complex / realistic style examples will really help identify what else needs work.

I'm curious about the dynamic variable name part. If we require that var and get args must be string literals, then we know at parsing time what subset of feature properties and variables need to be made available in the execution context.

This would be a more significant optimization when rendering many features with many properties in cases where only a small subset of those properties are used. And perhaps it wouldn't be a very significant optimization in most cases with user-provided variables (if instead of pruning these down to just the ones that are used we always include all variables as uniforms).

But I'm curious about the use case for dynamic variable names.

tschaub · 2023-12-09T22:52:56Z

I see now that the dynamic var is about the icon offset variables.

As an alternative to this

{
  'icon-offset': ['var', ['concat', 'offset-', ['get', 'maki']]],
}

we could do this

{
  // ...
  'icon-offset': ['match', ['get', 'maki'], 'bbq-15', [225, 0], 'bicycle-15', [0, 15] // etc.],
}

But I see how this would make for a large match expression.

I still feel like we may regret allowing both var and get to be dynamic.

jahow · 2023-12-10T09:43:02Z

Thanks for the first look!

Just a note on the streets-v6 example style: Use caution, I think it is not the best reference for what needs to be supported - it was a very early attempt a long time ago to get Mapbox data rendered with a minimal sensible style, and its origin was a loose manual translation of some layers of the old streets-v6 Mapbox style.

Yes I realize that, it does not represent a "real-life" mapbox basemap style. I figured it was still a good exercise to try and convert it, and have it be rendered by both Canvas and WebGL renderers.

I'm curious about the dynamic variable name part. If we require that var and get args must be string literals, then we know at parsing time what subset of feature properties and variables need to be made available in the execution context.

Dynamic var name doesn't sound too risky for me; making all variables available all the time isn't too costly, at least with the current system. On the other hand I now see how having dynamic get name will prevent many optimizations when parsing the data, so I will probably drop that. It wasn't even useful for this use-case, I made it as an effort to get closer to the Mapbox Style spec.

I still think having it possible for variables would be good, even if it's just for the sake of flexibility.

jahow mentioned this pull request Dec 8, 2023

Implement geometry-type operator in CPU #15414

Merged

Implement geometry-type operator in cpu

adeb884

jahow mentioned this pull request Dec 8, 2023

Allow partial expression evaluation on the CPU #15415

Open

jahow force-pushed the webgl-flat-mapbox-streets-style branch from a53d034 to 4a95e95 Compare December 8, 2023 14:19

Allow evaluating expressions on CPU with missing inputs

895e1b9

Evaluation contexts are now initialized with every properties set to UNKNOWN_VALUE. If an expression needs an input which is unknown, it will return UNKNOWN_VALUE.

jahow force-pushed the webgl-flat-mapbox-streets-style branch from 4a95e95 to 2382682 Compare December 8, 2023 14:23

jahow added 3 commits December 8, 2023 15:52

Cleanup incorrect comment

b2e40c5

Add a filter() method to MixedGeometryBatch

630002d

This will be used to filter out features that will eventually be rendered

WebGL / Use style filter when generating buffers

3bef70a

jahow mentioned this pull request Dec 8, 2023

WebGL / Filter geometries before processing them for rendering #15416

Open

jahow added 6 commits December 8, 2023 16:29

WebGL / the gpu compiler will now take into account props and variabl…

0d7ec7e

…e types Previously the style parser was having this responsibility. Now reading from feature attributes and style variables is done in the gpu compiler logic

WebGL / use a specific constant when a prop or var is not defined

c810592

This avoids a crash if a prop or var is not defined at runtime

Convert MapboxStreetsV6 format to expression-based

4d4abbb

This required compiling all the maki icons to a spritesheet Done using https://github.com/flother/spreet The MapboxStreetsV6 style was converted with partial automation

Allow dynamic name for var and get operators

948e4d0

This was also taken into account for the cpu and gpu compilers. The GPU compiler uses the CPU compiler to evaluate var/get names at compile time

WebGL / add support for concat operator on gpu

89fb58d

This operator is evaluated on the CPU at compile time, which means that it won't have access to runtime values (feature props, variables etc.)

wip better support for dynamic name expressions in get/var

e6bee20

jahow force-pushed the webgl-flat-mapbox-streets-style branch from 2382682 to e6bee20 Compare December 8, 2023 15:29

wip make concat more lenient on cpu

0cff7be

ahocevar reviewed Dec 9, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Mapbox Streets V6 example style to flat format #15413

Convert Mapbox Streets V6 example style to flat format #15413

jahow commented Dec 8, 2023 •

edited

github-actions bot commented Dec 8, 2023

ahocevar left a comment

ahocevar Dec 9, 2023

tschaub Dec 9, 2023

jahow Dec 10, 2023

ahocevar Dec 9, 2023

jahow Dec 10, 2023

ahocevar Dec 9, 2023

tschaub Dec 9, 2023

tschaub commented Dec 9, 2023

tschaub commented Dec 9, 2023

jahow commented Dec 10, 2023

Convert Mapbox Streets V6 example style to flat format #15413

Are you sure you want to change the base?

Convert Mapbox Streets V6 example style to flat format #15413

Conversation

jahow commented Dec 8, 2023 • edited

github-actions bot commented Dec 8, 2023

ahocevar left a comment

Choose a reason for hiding this comment

ahocevar Dec 9, 2023

Choose a reason for hiding this comment

tschaub Dec 9, 2023

Choose a reason for hiding this comment

jahow Dec 10, 2023

Choose a reason for hiding this comment

ahocevar Dec 9, 2023

Choose a reason for hiding this comment

jahow Dec 10, 2023

Choose a reason for hiding this comment

ahocevar Dec 9, 2023

Choose a reason for hiding this comment

tschaub Dec 9, 2023

Choose a reason for hiding this comment

tschaub commented Dec 9, 2023

tschaub commented Dec 9, 2023

jahow commented Dec 10, 2023

jahow commented Dec 8, 2023 •

edited