xparse: eats space token while looking for not provided, trailing optional argument

[frougon]

Hi,

With more or less recent xparse, a function definition such as
\NewDocumentCommand \foobar { m o } { ... }
results in strange behavior when \foobar is called without the optional argument. The strange behavior is that the following space token (at least) gets eaten. This did not happen in TL 2016 (Debian stretch), but does happen in TL 2018 (Debian unstable, xparse 2018-04-30; I also manually added xparse.sty 2018-05-12 for testing, same problem). Here is a MWE:

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{xparse}
\usepackage{expl3}

\ExplSyntaxOn

\NewDocumentCommand \foobar { m o }
  { #1 -#2- }

\ExplSyntaxOff

\begin{document}
Hell\foobar{o} .
\end{document}

Note that there is a space token between the \foobar{o} and the following period. In TL 2018 (Debian unstable, xparse 2018-04-30, and also with manually added xparse.sty 2018-05-12), the resulting PDF file after running pdflatex contains Hello—NoValue—. (there is no space before the period). In TL 2016 however (Debian stretch, aka "stable"), the output is what I expected, that is Hello—NoValue— . (one space before the period).

Thanks!

[josephwright]

We know about this, and there is work ongoing.

There was a deliberate change made to do with how we deal with trailing optional arguments: this comes down to the need to allow space skipping to be turned off in some cases. However, where commands with trailing optional arguments are used in the document body, this reveals an issue with the expl3 peek functions: they do no reinsert 'skipped' spaces if no match is found. We are currently sorting an appropriate syntax to extend peek functions to deal with this.

(In LaTeX2e, the only commands with trailing optional arguments are intended for the preamble, so the question of how to handle this scenario doesn't arise.)

I'd expect to get something concrete sorted during the TUG2018 meeting at the end of next week.

[frougon]

Thank you very much for your answer, great to see the problem is under such tight scrutiny. :)

[josephwright]

@blefloch is reviewing and thinking about the low-level support issues here.

[blefloch]

I haven't worked out the low-level peek issues but I'm committing a fix in xparse.

[agrahn]

Thank you for fixing this!

[frougon]

Hi,

I confirm that after installing the latest l3kernel and l3packages from CTAN, the problem disappeared. Thank you so much!