[core, ios, macos, android] Support TinySDF for local glyph generation

[ChrisLoer]

This is a core/darwin/android port of mapbox/mapbox-gl-js#4895, which uses https://github.com/mapbox/tiny-sdf in combination with locally available fonts to generate SDF for ideographic characters. This makes a dramatic improvement in loading times for maps that contain lots of CJK characters.

Qt is left out for now, but this PR establishes the pattern to follow if we find demand for a Qt implementation.

The branch currently contains:

• A port of tiny-sdf from @mourner's Javascript implementation to C++.
• Changes to GlyphManager to allow it to load local versions of glyphs (and transform them into SDFs) if they're available
• A new class LocalGlyphRasterizer that needs platform specific font loading and rasterization implementations
• A test implementation of LocalGlyphRasterizer that just returns a pre-rasterized '中' for any glyph that allows ideographic breaking.

- A darwin implementation that renders CJK glyphs using CoreText - An android implementation that renders CJK glyphs using `android.graphics.Canvas`

I've lightly tested these builds locally with macosapp, mbgl-glfw, the ios app running on my iPhone, and the Android test app running on an emulated Pixel. I've only tested installing fonts on OS X -- on the phones I just rely on whatever happens to be installed.

TODO:

• Darwin automated testing (or manual test strategy?)
• Android testing via LocalGlyphActivity
• Darwin configuration interface
• Android configuration interface
• Darwin font weight support (current implementation doesn't seem to work)

Out of scope:

• Extracting glyph metrics. If we did extract glyph metrics, we could generate all local glyphs instead of just CJK -- which could be helpful for making offline bundles smaller. However, this isn't necessary for the core "make CJK maps load faster" case (also, GL JS can't do it).
• Per-style-layer font configuration. This would allow any style to be completely reproduced using local fonts. Again, this would be important for the offline bundling case, but is less important for the "CJK only" case (I'm extrapolating from limited information, but it seems common that a single font family, perhaps with a couple font weights, is used to display all CJK characters).

/cc @boundsj @kkaefer @chriswu42 @lilykaiser @ivovandongen @1ec5 @tobrun @akitchen @zugaldia

[jfirebaugh]

how does the interface into GlyphManager look to you?

LGTM!

[ChrisLoer]

All right, I think this is ready for iOS and Android work to start. Since core behavior won't actually change without a platform-specific implementation, I might hold off trying to merge it until the first of the platform implementations is close to ready. @boundsj, you mentioned that you might consider trying to do this work on the release-agua branch -- after I squash the commits, I think it should be easy to cherry-pick.

On the core side, the only testing is the LoadLocalCJKGlyph unit test that tests that (1) network requests are skipped for local glyphs, and (2) the SDF generation works. Once we have platform-specific implementations, I think we'd ideally have some kind of render test that exercises the actual font loading (although making tests depend on locally installed fonts could get problematic quickly).

@mourner Can you review the tiny_sdf implementation?

/cc @ivovandongen b/c I remember you were interested in the Android side of this when I first started looking at doing it back in June.

/cc @nickidlugash and @lilykaiser for 🇨🇳 product design expertise

[1ec5]

Really excited to see progress toward #7862, even if it ends up being limited to CJK fonts for the time being.

How should local fonts be chosen? Will the local font properties be affected by the FontStack (on JS, they are mildly affected by the font stack, in that we pick up keywords like "bold" and use them to change font-weight when we rasterize the glyphs)?

On Apple platforms, font selection is a bit more nuanced than searching for keywords inside PostScript font identifiers. In part, this is because non-Western fonts (especially East Asian fonts) don’t necessarily have the same variants as Western fonts.

In C, the entry point for font selection is Core Text’s CTFontDescriptor. (There are iOS- and macOS-specific wrappers around this type in Objective-C.) For example, we might pass into CTFontDescriptorCreateWithAttributes() an attribute dictionary specifying a variation, a style name, or a traits dictionary that specifies a numeric font weight. Variations and style names are strings (like Bold) that could vary by font designer, but the numeric font weight is normalized.

CTFontCreateWithFontDescriptorAndOptions() chooses the best matching font based on a font descriptor. Ideally, we’d expose a UIFontDescriptor- or NSFontDescriptor-typed configuration option directly on MGLSymbolStyleLayer. This way, we could elegantly support countless font options and also custom fonts bundled with the application.

What character ranges should we handle? The JS side is conservative and sticks to just CJK because it doesn't have a way of extracting glyph metrics. However, on iOS/Android it may be easy to return the glyph metrics, in which case there's nothing preventing us from using TinySDF for all glyph generation. It's not clear to me if that would be an improvement over just doing it for CJK.

Yes, on Apple platforms, it’s straightforward to get the glyph metrics using CTFontGetBoundingRectsForGlyphs(), CTFontGetAdvancesForGlyphs(), or CTFontGetVerticalTranslationsForGlyphs(). It’s also possible to get the glyph’s path with an optional transform matrix using CTFontCreatePathForGlyph(), but that’s probably out of scope for this PR – it would be an alternative to TinySDF altogether.

Outside of CJK, the main benefit would be the ability to use local or system fonts (#7862). That would result in large savings to offline download sizes.

[mourner]

@ChrisLoer the TinySDF part of the PR looks great to me.

[ChrisLoer]

Ideally, we’d expose a UIFontDescriptor- or NSFontDescriptor-typed configuration option directly on MGLSymbolStyleLayer. This way, we could elegantly support countless font options and also custom fonts bundled with the application.

Interesting. That's a lot of granularity, although I can see how it would be useful. I worry about how much we end up forcing people to implement their style in multiple places (i.e. one part in the style itself via Studio, the other part in their app using the SDK). Are there other situations where we have this kind of split, with platform-specific style options being pushed down to SDK settings? Would it ever make sense to push platform-specific style options into the style spec (so instead of just text-font, you might have something like ios-font-descriptor)?

To keep the scope of an initial implementation limited to the CJK case (and provide an interface similar to what we have for GL JS), it sounds to me like it would work to introduce a localIdeographFontDescriptor to MGLMapView. It'd be straightforward for the SDK user to provide a UIFontDescriptor, and straightforward for us to use it. One part I'm unclear on is how best to pass a platform-specific UIFontDescriptor from MGLMapView down through all the portable/core code to end up available in platform/darwin/mbgl/local_glyph_rasterizer.cpp

[1ec5]

I suggested NSFontDescriptor/UIFontDescriptor based on seeing the parameters offered by TinySDF. However, it looks like GL JS only exposes the localIdeographFontFamily option directly as part of its API (and as a global option, not a per-layer option). If that’s the only option we ever foresee exposing to developers, then we could simply add a string-typed localIdeographFontFamily property to MGLStyle or MGLMapView (as well as MGLMapSnapshotOptions) that corresponds to kCTFontFamilyNameAttribute (e.g., Arial Unicode MS) or kCTFontStyleNameAttribute (sans-serif), then build a CTFontDescriptor around it internally. However, if there are no technical limitations against exposing an NSFontDescriptor/UIFontDescriptor option at the moment, then I’d recommend that approach for forwards compatibility reasons.

I worry about how much we end up forcing people to implement their style in multiple places (i.e. one part in the style itself via Studio, the other part in their app using the SDK).

If we expose a per-layer option (as opposed to a global option), then I don’t see a problem with forcing cross-platform developers to implement things slightly differently on each platform. After all, the runtime styling API already differs across platforms in terms of syntax and terminology, and local fonts behave differently on every platform anyways.

Would it ever make sense to push platform-specific style options into the style spec?

In principle, the style JSON format should be platform-agnostic. (For this reason, I’d like for us to discourage developers from using the canvas source type in style JSON files: mapbox/mapbox-gl-js#5545.)

[ChrisLoer]

I merged in the macOS/iOS implementation from ios-tinysdf-prototype because it adds a "local font family" argument to the Renderer constructor which the platform implementations can use to pass in their configuration.

There are loose ends that need addressing (and I'll need help from someone experienced with iOS):

• An actual entry point in the SDK (right now MGLMapView commonInit takes a localIdeographFontFamily argument, but it's not publicly exposed)
• Automated testing? This will be tricky to do reliably, since rendering depends on locally installed fonts. GL JS doesn't have any automated testing for the local font drawing itself, but the "surface area" to be tested is smaller there since we're just having the browser draw it for us.
• The "font weight" implementation doesn't seem to be having any effect on the font that gets loaded. I've tried a couple different approaches and failed -- I think I may be missing something more fundamental about CoreText font loading rules. This was a "nice to have" feature on the JS side, so it may be OK to just put it aside for now.

I'll start exploring an Android implementation now.

[ChrisLoer]

I've started work on an Android implementation based off this PR -- it lives at PR #10608.

[1ec5]

This guide to common Core Text font operations may contain some leads.

[1ec5]

It needs to be possible to set the font family at an arbitrary time after creating the mbgl::Renderer. The reason this method doesn’t take any parameters – not even the view’s frame – is that we have no control over the parameters at initialization when the view is initialized from a XIB or storyboard.

[1ec5]

#10612 would address the immediate issue here pretty elegantly, although there would remain room for future improvement.

[ChrisLoer]

👍

[1ec5]

Any changes to this file should also be reflected in MGLMapView.mm in the iOS project.

[ChrisLoer]

@akitchen has my back. 😉

[1ec5]

Per #10522, I think NSFontDescriptor/UIFontDescriptor is the more forward-compatible datatype for this option. It’s important to consider forward compatibility for iOS especially, because changing the type of a property or parameter will require a semver-major release.

If we aren’t ready to support the many options in NSFontDescriptor/UIFontDescriptor, we can specify in the documentation that only the font family option is supported for now.

[ChrisLoer]

Does @akitchen's change from #10612 dodge the forward-compatibility issue well enough in your mind?

[1ec5]

Yes, it’s good enough for now. In the future, if we open up more options, then we can introduce formal APIs for them without worrying about backwards compatibility.

[1ec5]

In terms of plumbing the value through mbgl, consider adding an overridable method to mbgl::RendererFrontend that returns a CTFontDescriptor. The Darwin implementation of MGLRenderFrontend can convert to and from NSFontDescriptor or UIFontDescriptor, using typedefs and conditional compilation to paper over differences between iOS and macOS.

Since CTFontDescriptor is itself platform-specific, consider wrapping it in a new mbgl::FontDescriptor class along the lines of mbgl::RendererFrontend before returning it to mbgl::RendererFrontend.

[1ec5]

Would it be possible to include CFFontTraits.h for CTFontSymbolicTraits?

[ChrisLoer]

In terms of weight, the only thing CTFontSymbolTraits exposes is kCTFontTraitBold. Or did you have something else in mind? I did try using that to load a bold font without luck...

[1ec5]

Are you seeing this even if you make the boundingRects and advances arrays the same length as glyphCount?

[ChrisLoer]

The assert(glyphCount == 1); never fires, so yes, the arrays should be the same length.

[ChrisLoer]

OK, I think this branch may be ready to merge. I've modestly squashed the commit history to get rid of obvious fixups, but I left the iOS attempts at getting glyph metrics and setting font weights in the history, I also didn't squash into @akitchen 's or @asheemmamoowala 's commits so that they get blamed appropriately. 😉

Here's the current state of all the compromises involved:

• iOS configuration is string-based and global to the app
• macOS/iOS don't get support for the "font-weight" heuristics that GL JS and Android have. It'd be great to get this working in a follow-up PR, but this was a "nice-to-have" and I don't think it should block merging this PR.
• macOS font loading gets automated testing via local_glyph_rasterizer.test.cpp
• The iosapp target defaults to using "PingFang" locally, for manual testing
• Android configuration is string-based and belongs to the MapView via MapboxMapOptions (this is pretty closely analogous to GL JS behavior)
• The Android test app has a "Local CJK Glyphs" activity which loads a map of Suzhou using "Droid Sans"

Both platforms have API-level documentation. After this PR merges, we should consider platform-specific "Examples" analogs to the GL JS example. @lilykaiser can you help keep track of this loose end? I'm happy to work on the examples but I don't know where they live for the various platforms, when/how we generate them, etc.

This PR would be a good candidate for some device-specific testing as part of the release process -- I've relied on using emulators for testing the iOS and Android code. @akitchen / @zugaldia is there some way we can put "exercise this functionality on our local devices" on part of a todo-list for testing the versions of the SDKs that incorporate this functionality?

@ivovandongen and @1ec5, I'll wait for 🍏 from both of you before merging -- thanks for all your help in getting this ready, I know both of you have been busy working against other deadlines.

[1ec5]

iOS and macOS changes look fine, other than the naming issue. Thanks @akitchen for sorting that out.

Would you do the honors and add a blurb in the iOS and macOS changelogs?

Map snapshots won’t benefit from this feature yet, correct?

[1ec5]

“Ideograph font” is ungrammatical and sounds funny. The previous terminology, “ideographic font” was more correct. If inconsistency between the iOS/macOS map SDK’s public API and the underlying types is a problem – it rarely is – can we rename the latter?

[ChrisLoer]

Reverted to "ideographic"

[1ec5]

iOS and macOS both come with PingFangHK, PingFangSC, and PingFangTC. I wonder which font PingFang ends up loading.

[ChrisLoer]

I'm not sure which iOS and Android releases this will be part of, but here's a blurb that we can add to the changelog once we know:

* Added optional support for using local fonts to generate Chinese, Japanese, and Korean glyphs in order to significantly decrease data usage
 [#10522](https://github.com/mapbox/mapbox-gl-native/pull/10522)

Good point regarding map snapshots -- they're not hooked up now, but it should be easy to do. I split that out into #10665.

[ivovandongen]

Looks good.

[ivovandongen]

Could use a short JavaDoc explanation what we're testing

[ivovandongen]

This is now inconsistently named with the implementation file.