letters and spacing in Arabic — bug in OpenType layout of LuaLaTeX

[ClintEastwood]

I perceive the following unwanted outcome:

Two issues:

1. In the first line, the last word on the right has a small gap between the second and the third letter from the right that is not supposed to be there.
2. In the second line, the word is entirely jumbled up.

I had issue #1 before in June 2013. I wrote to Khaled Hosny about it and he told me that it is a

bug in the OpenType layout engine currently used by LuaLaTeX.

The solution back in 2013 was to update luaotfload through issuing luaotfload-tool --update.

Today, that does not do the trick.

So what am I going to do? I am very surpised to suddenly see the old error again after 2,5 years. What happened?

I also reported the bug on tex.sx: http://tex.stackexchange.com/q/285674/12277 .

The MWE, in case somebody is interested:

% !TEX TS-program = lualatexmk
\documentclass{article}
\usepackage{fontspec}
    \newfontfamily\arabicfont[  Script=Arabic,%
                            Numbers=Arabic,%
                            Scale=MatchLowercase%
                                ]{Amiri}%
    \newfontfamily\arabicfontz[ Script=Arabic,%
                            Numbers=Arabic,%
                            Scale=1.6%
                                ]{Arabic Typesetting}%
\newcommand{\arabtext}[1]       % Arabic inside LTR
    {\bgroup\textdir TRT\arabicfont #1\egroup}
\newcommand{\arabtextz}[1]      % Arabic inside LTR
    {\bgroup\textdir TRT\arabicfontz #1\egroup}
\begin{document}
\noindent Font: Amiri => wrong: \arabtext{تعريف}\\
Font: Arabic Typesetting => very wrong: \arabtextz{تعريف}
\end{document}

[ClintEastwood]

Let me please add that I am certain that this is only a minor bug (because (a) I experienced it before (b) it occurred only very recently, and (c) it was already fixed until a few days ago). In addition, I really need this to be fixed soon and would like to offer my help on this in any way I can. Please do contact me, if you need help, e.g., with the Arabic.

[phi-gamma]

···<date: 2016-01-03, Sunday>···<from: ClintEastwood>···

I perceive the following unwanted outcome:

Two issues:

1 In the first line, the last word on the right has a small gap
between the second and the third letter from the right that is
not supposed to be there

Reproducible with Amiri and Luaotfload. Interestingly, it doesn’t
happen in Context even if we imitate Luaotfload’s arabic
features.

Test files:

https://bitbucket.org/phg/lua-la-tex-tests/src/tip/plain/pln-arabic-1.tex
https://bitbucket.org/phg/lua-la-tex-tests/src/tip/latex/ltx-arabic-1.tex
https://bitbucket.org/phg/lua-la-tex-tests/src/tip/context/cnt-arabic-1.tex

2 In the second line, the word is entirely jumbled up

Can you reproduce this with a free font, if possible one that
comes with Texlive? Otherwise it’ll be hard to draw any
conclusions.

I had issue #1 before in June 2013 I wrote to Khaled Hosny about it and he told me that it is a

bug in the OpenType layout engine currently used by LuaLaTeX

The solution back in 2013 was to update luaotfload through issuing luaotfload-tool --update

That shouldn’t affect text rendering at all.

[ClintEastwood]

Thanks for your reply.
Actually I don't know if TeXLive has any fonts for Arabic. Arial is fine, Scheherazade and Lateef are fine, Noto Naskh Arabic and Noto Kufi Arabic seem to be fine. But Reem Kufi and Aref Ruqaa, both from Khaled Hosny -- just like Amiri --, produce the error, too.

Actually I am guessing that the errors produced by Amiri and by Arabic Typesetting are the same i.e. have the same cause, they just manifest themselves differently. Only the letters for R (ر) and Z (ز) seem to be affected and, as it seems, only in some combinations (eg. after ح/خ/ج or ع/غ).

I have proof that it was everything alright by 11th of December 2015. The error was introduced probably around Christmas (thanks for the present ... =) ). As I said, I had the same issue before in 2013, with the same erroneous results.

[ClintEastwood]

I just found a similar error appearing in the word وكلّما.

[phi-gamma]

···<date: 2016-01-08, Friday>···<from: ClintEastwood>···

Let me please add that I am certain that this is only a minor
bug (because (a) I experienced it before (b) it occurred only
very recently, and (c) it was already fixed until a few days
ago).

Probably not the same bug. This one is related to upstream
changes in the feature mechanism. Bisection reveals it was
introduced in November:

adc43c3c243ed95215af6fe02d71e393c178da23 is the first bad commit
commit adc43c3c243ed95215af6fe02d71e393c178da23
Author: Context Git Mirror Bot <phg42.2a@gmail.com>
Date:   Thu Nov 19 20:15:06 2015 +0100

    2015-11-19 19:15:00

Which corresponds to this changeset:

https://bitbucket.org/phg/context-mirror/commits/adc43c3c243ed95215af6fe02d71e393c178da23

For the time being, reverting to the TL 2014 fontloader seems to
do the trick:

$ > luaotfload.conf cat <<THATSALL
> [run]
>   fontloader = tl2014
> 
> THATSALL

NB this can introduce other issues but it builds the minimal
examples.

[ClintEastwood]

Thanks, I will try this. Can you please tell me what I have to do with your "trick" on a mac? (Sorry for my ignorance.)

[phi-gamma]

···<date: 2016-01-11, Monday>···<from: ClintEastwood>···

Thanks, I will try this. Can you please tell me what I have to
do with your "trick" on a mac? (Sorry for my ignorance.)

Create a config file (luaotfload.conf) somewhere where
Luaotfload can find it, preferably the root directory of your TeX
project where Luatex is invoked. The contents should be as in the
command I gave earlier:

[run]
  fontloader = tl2014

That makes Luaotfload use the fontloader code from the 2014
edition of texlive.

Fwiw the documentation for config files can be accessed as a man
page:

man 5 luaotfload.conf

[ClintEastwood]

I am now using your suggested way to bypass the error and it works.

Another user complained about a similar issue on tex.sx: http://tex.stackexchange.com/q/287150/12277 . I referred him to your suggestion and it works for him too.

Now, that I am able to compare an old erroneous pdf (with the recent fontloader) and a new correct pdf (with the old fontloader), I see many many erroneous differences. The issue concerns not only the letters for R and Z (as in my original post) or to the diacritical strokes and lines (as in the other user's complaint). If you are interested, I can show you examples of a number of errors.

Now that we know that it does not only affect a limit number of characters but that there must be something fundamentally wrong, do you have an idea what it could be and how it could be fixed, despite the fact that your suggestion to use "fontloader = tl2014" works for the time being?

[ralessi]

I have another issue which is in my opinion related to this one. This mwe:

\documentclass[12pt]{article}

\usepackage{fontspec}
\usepackage{etoolbox}
\newfontfamily\libertine{Linux Libertine O}
\newfontfamily\arabicfont[Script=Arabic]{Amiri}

\newcommand{\arabtext}[1]       % Arabic inside LTR
    {\bgroup\textdir TRT\arabicfont #1\egroup}

\begin{document}

As you can see, this is rightly typeset LTR, but the vowels,
which are seemingly RTL: \arabtext{دَخَلَ مُبتَسِمًا}.

\end{document}

gives me this:

To be more precise, each vowel seems to be moved one letter back (i.e. to the right).

fontloader = tl2014 solves this issue.

[phi-gamma]

···<date: 2016-01-19, Tuesday>···<from: Robert Alessi>···

To be more precise, each vowel seems to be moved one letter back (i.e. to the right).

fontloader = tl2014 solves this issue

If you have Context installed (I’m assuming TeX Live) you could
try fontloader = context too; this gets you a mostly new
fontloader without that specific bug.

[phi-gamma]

adc43c3c243ed95215af6fe02d71e393c178da23 is the first bad commit

Ok, narrowed it down to font-otn.lua: If that file is reverted
to the “good” revision, the test file renders as expected.

[phi-gamma]

Solved it. The Luatex in TL is too old. The culprit appears to be
revision 5502
in which the whatsit kind local_par and dir nodes were
promoted to toplevel node types. When Hans changed the
fontloader to the new model (which was in experimental for
some time already at this point, I gather), he did away with the
old code paths.

We’re going to have to keep those hunks reverted until TL2016.

[phi-gamma]

New release
is on its way to CTAN. Please test!

[ClintEastwood]

You do an amazing work here! I am infinitely grateful!
Should you need help in the future with Arabic/Persian/Syriac and RTL in general, don't hesitate to contact me.

I'll test as soon as it is up.

[ralessi]

I'd like to express my thanks as well for your excellent work! I'll test it too as soon as it is up.

[ralessi]

I just tested the latest release: everything seems ok to me. Great!

[phi-gamma]

Excellent, closing this.