[WIP] Tangram 3D Hack

[cleeus]

Hello everyone, I hacked Tangram-ES to render 3D terrain using the terrarium tiles. Enjoy the screenshot from SanFrancisco below:

wireframe view:

So ... this is not in a shape that can be merged into master, of course. There are a bunch of issues.
The most problematic one is that I cannot figure out how to fetch the terrain data in the background before constructing the tile geometry. As you can see in tileTask.cpp the load_terrain_tile_data function downloads the data and blocks the task/thread until the download is finished. I think it would be much nicer to specify the terrain source as a raster in the scene.yaml, something like this:

sources:
    elevation-tiles:
        type: Terrarium
        url: https://elevation-tiles-prod.s3.amazonaws.com/terrarium/{z}/{x}/{y}.png
    satellite:
        type: Raster
        url: https://your-preferred-tile-service/{z}/{x}/{y}.jpg
        terrain: [elevation-tiles]
    osm:
        type: MVT
        url: https://tile.nextzen.org/tilezen/vector/v1/256/all/{z}/{x}/{y}.mvt
        max_zoom: 16
        url_params:
            api_key: global.sdk_api_key
        terrain: [elevation-tiles]

and then have the task scheduling mechanism solve the problem of first fetching the elevation-tiles.

I really would enjoy your feedback.

There also are a bunch of visual problems, e.g.:

• points markers are not projected properly
• labels are not projected properly
• tile bounds are not seamless (solvable by looking at neighboring tiles during mesh generation, or using server-side generated meshes)
• lines are not draped perfectly (solvable by intersecting lines with terrain triangles and adding artificial new segments)
• polygons are not draped perfectly (solvable by applying polygon intersection to terrain mesh)

[tallytalwar]

This is superb @cleeus. Definitely all the steps are in the right direction.

I can look at the task scheduling to add terrain as a dependent source, which will block geometry construction until "dependent" terrain tiles are fetched. We do the same for our raster-vector rendering and use terrain normals from the dependent terrain tiles. We should also be able to define the construction of the 3d terrain from the style sheet.

With respect to other issues you have mentioned some of them are related, example both point markers and labels are part of the same internal projection pipeline and applying apt projections for these (most likely including normal information from the underlying terrain data) should be able to fix the projections.

Seamless tile bounds can be resolved by blending the buffer tile. @matteblair did have some work in unity as a proof-of-concept to get this working (https://github.com/tangrams/unity-terrain-example). We include some buffer pixel data outside the bounds of the tile which can be used to blend the boundaries to be seamless.

Your solutions for line and polygon draping sounds good, but might not be performant. That being said first thing is to get it in a working state and we can look at other performant ways.

Again, congrats on this and happy to see some major contributions from you to the tangram source. 🤘 .

[nvkelso]

This is super rad!

[matteblair]

Great stuff! A while ago I worked on a different implementation of rendering elevated terrain in tangram-es, maybe some of it is useful to refining this feature: https://github.com/tangrams/tangram-es/tree/grid-style

Not sure when we're gonna have the time to start working on a proper version of this feature. There's a lot of questions around how to displace or deform the 2D map features that we would have to sort out. But this is exciting to see nonetheless :)

[cleeus]

I can look at the task scheduling to add terrain as a dependent source

This would be super nice. I see there is stuff like sub-tasks but I couldn't figure out how to use it and where to put the terrain transformation code.

Seamless tile bounds can be resolved by blending the buffer tile. @matteblair did have some work in unity as a proof-of-concept to get this working (https://github.com/tangrams/unity-terrain-example). We include some buffer pixel data outside the bounds of the tile which can be used to blend the boundaries to be seamless.

Ah, now I understand why there are 260/516px terrain tiles ... :) that will work.
There also is another way: generate seamless tiled meshes on the server (which also will generate fewer triangles). Cesium is going into that direction. They have a neat format for storing those meshes called "quantizes mesh" (https://cesiumjs.org/data-and-assets/terrain/formats/quantized-mesh-1.0/) .

Your solutions for line and polygon draping sounds good, but might not be performant. That being said first thing is to get it in a working state and we can look at other performant ways.

Yes, draping stuff is not trivial and needs experiments and iterations, but I think it's solvable. Draping big landuse polygons can be done by "cutting out" the exact piece of terrain with the shape of the poly. There are a bunch of papers about the topic and a few implementations, so it's basically a solved problem and "just" needs a nice implementation.
Draping lines can also be done with same approach (after they are converted to polygons for drawing). I think as long as the polygon intersection stuff is only done once per tile, it's going to be O.K. - it will add more polygons to the scene, of course.
Draping building shoeboxes is a different story - since you want to preserve the flat rooftops, there is no obvious "perfect" solution. One method that I think could work would be to select one center point for the base elevation and then extend the walls into the ground.

[ccxxbb]

When add height map， screenPositionToLngLat shouble be consider， this function seem have problem，hava anly idea to resolve this problem?

[ETE-Design]

@matteblair Is this feature developed yet?

[Mis012]

it seems to me that all that is really needed is a way to subdivide the tiles so that there are enough vertices...
positioning said vertices accordingly seems to already be possible with the "position" shader block:

        base: polygons
        raster: custom
        shaders:
            blocks:
                global: |
                    float getGrayscale(vec3 p) { return (p.r + p.g + p.b) / 3.0; }
                    #ifndef TANGRAM_FRAGMENT_SHADER
                    uniform sampler2D u_rasters[TANGRAM_NUM_RASTER_SOURCES];
                    uniform vec2 u_raster_sizes[TANGRAM_NUM_RASTER_SOURCES];
                    uniform vec3 u_raster_offsets[TANGRAM_NUM_RASTER_SOURCES];
                    #endif
                color: |
                    color = sampleRaster(0); // color from first raster (basemap)
                position: |
                    #define adjustRasterUV(raster_index, uv) ((uv) * u_raster_offsets[raster_index].z + u_raster_offsets[raster_index].xy)
                    #define currentRasterUV(raster_index) (adjustRasterUV(raster_index, v_modelpos_base_zoom.xy))
                    #define currentRasterPixel(raster_index) (currentRasterUV(raster_index) * rasterPixelSize(raster_index))
                    #define sampleRasterAtPixel(raster_index, pixel) (texture2D(u_rasters[raster_index], (pixel) / rasterPixelSize(raster_index)))
                    #define sampleRaster(raster_index) (texture2D(u_rasters[raster_index], currentRasterUV(raster_index)))
                    #define rasterPixelSize(raster_index) (u_raster_sizes[raster_index])

                    // the 100.0 seems to be needed to get a visible effect, should presumably *not* be needed if using a non-8bit-compressed raster
                    position.z = 100.0 * u_meters_per_pixel * getGrayscale(vec3(sampleRaster(1)));

further, support for F32 single band TIFFs and other proper >8bit raster formats would be nice (currently it seems that r/g/b rasters are assumed, not sure if 8bit precision is assumed or not)

FWIW, the shader code mentioned above works with no modifications to the source code (but the low amount of vertices makes it useless), and seems to work better with more vertices (using the vertex subdivision part of this merge request), though I seem to be getting random crashes if I use a number anywhere near 64 with that code

EDIT: just confirmed that at least the crashes that happen after it loads up and lets me look around seem to be caused by me running out of RAM, which suggests there is definitely some optimization needed... for example a JS-based 3D terrain viewer which works with the same dataset (except not uselessly clamped to 256 possible heights) doesn't even consume enough RAM to make firefox kill it