can you attach a small file that illustrates the issue for us ?

4 byte UTF-8 characters issue

Below you can see glyphs of Arial Unicode font.

And here you see, how GD draws consecutive glyphs starting from 0xFFD0 and ending at 0x1001F.

You can clearly see, that characters after 0xFFFF are incorrectly parsed.

Check UTF-8 byte representation of 0xFFFF and 0x10000 Unicode characters here:

• http://unicode-table.com/en/FFFF/
• http://unicode-table.com/en/10000/

i'm not saying you're wrong ... i'm asking for a small program that we can build/run to check the behavior on our side so that we can develop a fix & testcase. that way we don't spend time recreating something you/someone else has already written, and we don't end up making changes for a different codepath w/out actually fixing the ones you're reporting.

Adam Harvey already supplied a respective reproduce script including a respective font file and actual and expected images in the respective PHP bug report, and the first halve of the necessary bugfixes. I have now analyzed the issue there further. TL;DR: we have to also select the appropriate charmap (a font can have several of them).

Specifically for emoji, we need to implement the ft flag support. As far as I remember we have another issue for that.

@pierrejoye Indeed there is issue #184, which would be a nice enhancement.

Wrt. this issue, it seems that in the long run we can simplify finding the desired charmap by just using FT_Select_Charmap() instead. For GD-2.2.x (and maybe GD-2.x.y) I would suggest to go with a minimal fix, namely to use FT_SelectCharmap() only if encoding is detected as gdFTEX_Unicode.

The biggest hurdle I'm currently facing is finding a font which we can use for the test case. Adam's test uses code2001.ttf which is apparently freeware, but it's lacking a license file.

you could always take an existing free font and hack it up with something like FontForge or Birdfont. we don't need the glyphs to be "correct", just unique so we can compare them against a reference image.

Thanks, Mike! I'll give FontForge a try.

Hello, this seems to be still unfixed? Any ETA? Thanks

Any ETA?

No, sorry. A PR including a regression test would be very welcome!

A PR including a regression test would be very welcome!

I'd love to help! Is there a way we can discuss what needs to be done, without polluting the thread? :)
A basic patch, as proposed here and mentioned above, would be to just extend the number of chars that the function is parsing.. Is that not enough? If there's more to be done, I can try helping, I just need some direction :)

any update of this in 2019?

I've modified libgd and got it printing the entire 4 byte range now.
I'm still testing and am unsure about the html5 entities.
Also, I noticed xshow don't seem to account for the horiBearingX, is this intentional ie: meant to return the glyph origin ?

Here is a patch to test.
Issue-185-01.diff

I refined the patch and extended it to include all html5 entities.
http://sterlingdesktops.com/pub/test/Issue-185-02.diff
I just need to do a complete test to verify that everything works.
Also, we need to possibly use a different charmap when there is no glyph present. This patch it hardcoding to let freetype decide with: FT_Select_Charmap(face, FT_ENCODING_UNICODE);

FT_Set_Charmap(face, charmap); stops working when higher codepages are used here.

I'll submit a pull request after further testing and would appreciate some comments.

http://sterlingdesktops.com/pub/test/Issue-185-05.diff
I verified that this one works with all the different html/utf input formats.

Edit latest:
http://sterlingdesktops.com/pub/test/Issue-185-06.diff

the HTML spec links to a JSON database:
https://html.spec.whatwg.org/entities.json

seems like that'd be a lot easier to parse than an ad-hoc HTML parser ? could even do it in Python pretty easily.

Who actually runs the generation script/program ?
I also worry that /usr/include/entities.h is not a good idea since libgd is the only thing using it(I could be wrong).
If this is the case, then we should compile a entities.h/.c only and not install it as a header.
The best solution might be to include a header that is already on most people's systems from a different project.
Also, that entities.json will more than double the array members because of case sensitivity and ";".

If we are going to be completely thorough then we should check that there are no conflicts with unicode as well. IE: & as the first byte matching some of these entities.

i agree we should stop installing entities.h. i'll do that for the next major release. this should only exist for our own gdft.c usage.

the generation script is for us devs. we're committing the header file to the tree so we can ship it in releases.

gd_Entity_To_Unicode should be moved to entities.c as well, but then that'd complicate the gen script we end up using.

wrt entities.json, we can easily dedupe based on the trailing ; being omitted. similarly, i think we can normalize the entity names to lowercase and call it a day.

"Entity names (the things that follow ampersands) are case-senstive, but many browsers will accept many of them entirely in uppercase or entirely in lowercase; a few must be cased in particular ways."

I think you're right, we have to go with entities.json.

4 different ways of specifying the same thing in this one example:
Dot die DoubleDot uml

My original solution was just taking the first, I hadn't considered this.
I'll give it some thought and try to put something together tomorrow.

Also there can be two unicode codepoints per entity ie: NotLessLess ≪̸

gdImageStringFTEx will have to be modified to handle two codepoints returned.
struct entities_s must have two ints to hold all these.

This will take some time. I will try not to flood this thread anymore untill I have something substantial.
I'm playing with json-c right now.

I would imagine python would be faster to do this in, so feel free to use that

I dunno about that, i'm already printing all the key/value pairs as strings in C.
It seems most the functions are just macros. The fastest might just be a bash script tbh.

I know perl better than I know python. perl regex would make quick work of it too.

we're in the process of removing perl from the codebase ;)

Ok, got a python generator working.
I need to modify gdft.c some more now

Everything is in place at this point. It just needs testing.

Things i've tested:
All single codepage unicode given as utf8 (one per function call same image).
All single entity string/hex/dec (one per function call same image).
A string containing all utf8 encoded unicode for various fonts (single function call).
A string containing all entity strings (single function call).

I still need to test mixing of these as the function supports that and should autodetect.
Also I need to test that the JISX0208 or any non-unicode has not been broken.

#695

I think we are good with the PR. We need tests, but I can give @sterlingpickens a hand for this. @sterlingpickens did an amazing job here and libgd will be better with this addition.

@vapier @cmb69 merge and I can help then, thought?

Yeah, I suggest to merge PR #695, and to go from there.