Declaring local variables

[EvanAad]

The document The expl3 package and LaTeX3 programming (v. 2017/09/18) states on p. 7:

The LaTeX3 coding convention is that all variables must be declared before use.

Similarly, the document The LaTeX3 Interfaces (v. 2017/09/18) states on p. 10:

... in contrast to variables, which must always be declared

expl3 provides no facilities to declare a variable locally; only globally. However, it is still possible to create a local variable implicitly using the various \..._set:N... functions.

It is a principle of structured programming that dumping variables into the global scope unnecessarily should be avoided. The TeX language, like all high-level programming languages that I know (including C, C#, Fortran, Java, Lisp, Pascal, and Python), provides a scoping construct for creating local variables, namely groups. Moreover, the LaTeX3 programming language itself, thankfully, supports the creation of local variables, despite what the manuals suggest.

In my opinion, the admonitions quoted above should be deleted from the manuals, and it should be explained that variables can, and should, be created locally whenever possible.

Moreover, if it is indeed desired to encourage a programming style in which all variables are declared before they are used (I personally think this is a matter of programming style that should be left to the individual taste of the programmer), functions need to be provided for declaring local variables just as there are functions for declaring global variables.

[josephwright]

The current position follows a number of experiments by the team to establish a pattern which works with the TeX fundamentals supporting expl3. In particular, it's important to note that variables may be implemented using macros or using registers, and to bear in mind how TeX grouping works.

When using registers (e.g. for the int type) we need an allocator to link the register number and cs we are using for access. In contrast, for macro-based storage that is not needed. Thus whilst \cs_set_eq:NN can be used to generate new tl names (for example), it will fail with anything using registers:

\int_set_eq:NN \l_undeclared_int \l_tmpa_in

will raise an error.

It is possible to create a local register allocator, and we have tried that out in the past for e.g. \int_local_new:N. (See the etex package for one such implementation, or look back through the expl3 history.) The team eventually decided against this as it did not seem to 'fit' well with TeX scoping. Key here is that TeX grouping is not based on declarations but rather on explicit groups, and that a local variable exists within any nested groups:

\begingroup
  \def\foo{}
  \begingroup
  \show\foo

When we tried local creation of variables, it seemed that there was a danger of obscuring how TeX grouping worked, and that programmers could be misled.

We therefore decided to go with 'all global' declaration of variables, but with both local and global setting of such variables (the \l_... _versus \g_... convention). This localises the important thing: the value. We therefore recommend that all declarations are at the top level

\tl_new:N \l_my_tl
\int_new:N \l_my_int

...
\cs_new_protected:Npn \my_func:nn #1#2
  {
    \group_begin:
      \tl_set:Nx \l_my_tl { \tl_lower_case:n {#1} }

To-date, this pattern appears to work well for the tasks expl3 has been used for.

[u-fischer]

In tex even locally defined variables are not completly destroyed after the end of the group. They still use some string space: https://tex.stackexchange.com/questions/316999/release-space-in-the-string-pool. So imho it is better to view variables as global objects.

[josephwright]

@u-fischer Good point: not one we actually considered at the time but worth bearing in mind.

[EvanAad]

In my opinion this approach is mistaken. It goes against the principles of structured programming, and it goes against TeX, which enables the creation of local variables. It also makes LaTeX3 inconsistent, since some data types accommodate local variables and some don't.

I would like to request that you re-institute local creation of variables of all types. This gives the programmers a choice: those who prefer to define all variables globally can do so, and those who wish to define some globally and some locally can do so too.

If there are caveats, memory-wise or otherwise, they can be mentioned in the documentation.

[davidcarlisle]

On 18 October 2017 at 21:17, EvanAad ***@***.***> wrote: In my opinion this is approach mistaken, and I would like to request that you re-institute local creation of variables. This gives the programmers a choice: those who prefer to define all variables globally can do so, and those who wish to defines some globally and some locally can do so too. for variables based on a register type you are allocating resources from a

fixed global pool so global allocation of the name is in fact far more natural. Given that we assume etex so there are more that 256 registers of each type, the allocation behaviour can perhaps be hidden more than it could with classic tex but it is still a fundamental feature of the underlying TeX system. expl3 can never be a completely general programming language: it has to work with the underlying TeX system. You should not be comparing with C# but with other TeX based languages and in particular with \newcount and friends. It's also not really true to say all other languages allow variables to be declared in local scopes, fortran for example only allows variables to be declared at the start of a function/subroutine. None of the above means that we definitely wouldn't add a local declaration system, but appealing to general purpose languages isn't a good use case. There would need to be reasonable use cases within tex typesetting where using a global declaration system was a problem.

[EvanAad]

The fact that Fortran requires variables to be declared at the start of a function/subroutine does not mean that those variables are global. If you insist on encouraging a programming style where all variables must be declared, so be it, but then provide functions for declaring local variables corresponding to the ones for declaring global ones.

The rationale for scoping and local variables is fore and foremost conceptual. The fact that the underlying TeX engine does not release some of the resources associated with local variables is not to be dismissed, but, in my opinion, this cannot be the reason for obliterating the notion of scoping and local variables. Except for power programmers, the underlying implementation is of no concern; and for power programmers, tips and caveats can be offered in the documentation.

appealing to general purpose languages isn't a good use case.

I would say that diverging from the principles of structured programming which have been implemented in virtually every programming language from the 1960s till today, including TeX, should be something that the LaTeX3 team should have very compelling reasons for doing. The burden of proof should be on those who wish to abolish what is ubiquitously considered good programming practice, and what already exists in TeX; not on those who wish to preserve it.

[eg9]

@EvanAad Token list, comma list sequence and property list variables are actually TeX macros, but variables of different kind aren't. Let's assume declaring an integer variable locally is allowed. This consumes a register and it's necessary to globally take care of this so as the allocation doesn't touch an already assigned register.

Say your local integer variable \x is assigned register 100 which is already taken by variable \y at an upper level; using \y at the same level where \x is locally defined would be simply disastrous. Thus global bookkeeping of assigned registers is needed, making it very hard to release locally assigned registers. Reserving a block of registers for local assignments is not a solution.

I'm not saying it can't be done, but I don't think it's worth the pain.

[davidcarlisle]

you are missing the fact that \newcount is not simply locally (or globally) declaring a name, it is associating a name with an externally defined fixed resource. there is only one count register 42 and all names that are declared to mean count42 are referring to the same register, whether the name allocation is local or global. As I mentioned comparisons to other languages are not that useful but if you want a comparison you should compare allocating file streams or some such: you can not always isolate local declarations when they are interfacing to an externally defined resource.

…

On 18 October 2017 at 21:43, EvanAad ***@***.***> wrote: The fact that Fortran requires variables to be declared at the start of a function/subroutine does not mean that those variables are global. As I wrote, if you insist to encourage a programming style where all variables must be declared, so be it, but then provide functions for declaring local variables corresponding to the ones for declaring global ones. The rationale for scoping and local variables is fore and foremost conceptual. The fact that the underlying TeX engine does not release some of the resources associated with local variables is not to be dismissed, but, in my opinion, this cannot be the reason for obliterating the notion of scoping and local variables. Except for power programmers, the underlying implementation is of no concern; and for power programmers, tips and caveats can be offered in the documentation. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#410 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNcAimMfBDqA-e96Q7tkS-ERr5fv_2Mks5stmLqgaJpZM4P-Mpq> .

[EvanAad]

The crucial point is that, as Joseph and David mentioned, allowing ints to be defined locally can be (and has been) done. I think it is worth the pain in order to create a language that presents a consistent abstraction layer that is in agreement with the principles of structured programming.

However, saying it is worth the pain is easy for me to say, because I am not the one subjected to the pain of implementing the LaTeX3 language. So let's take a merciful approach, and say that it is in fact not worth the pain. Fine. In this case divide the data types into two categories: those that allow for the creation of local variables, and those that don't. And describe these categories in the manuals. Don't say: all variables must be declared before use. Say: "There are two categories of data types. The first, consisting of cs, tl, clist, ... accommodates local variables, and the second, consisting of int, ... does not." And explain why the second category exists. This way the documentation is true to the facts, and the programmers are well informed and have a choice. Some programs do not need to use data types of the second category at all, and in these cases why should the programmer have to declare global variables?

[josephwright]

@EvanAad I suspect this is because I'm used to TeX programming (including the 'traditional' restrictions of TeX90), but I'm not sure at present what the issue is with the current approach. We do support and indeed encourage locally-assigned variables: they are very common and indeed part of the syntax we have for variable naming (\l_.../\g_...). The fact they are allocated/'reserved' globally doesn't interfere with that.

BTW, we don't want to tie interfaces to implementation: for example, the prop data type has had at least a couple different implementations I know of, one of which used registers and the current one using macros.

[wspr]

To add one more note to this, the straw that broke the camel's back (in my memory) that the local register/variable allocation system fell down was because of inconsistency. Because macros and registers behaved differently, you ended up with different behaviour when writing

\group_begin:
  \int_new_local:N \l_tmpa_int
  \int_gset:Nn \l_tmpa_int {7}
\group_end:
% `\l_tmpa_int` undefined

vs

\group_begin:
  \tl_new_local:N \l_tmpa_tl
  \tl_gset:Nn \l_tmpa_tl {7}
\group_end:
% `\l_tmpa_tl` defined as `7`

And the only way to resolve this problem would be to write an allocation system for macros, which was never seriously considered due to overhead. (TeX slows down the more variables are defined, and so doubling the number of token lists could have affected performance noticeably.)

I still think that expl3 did the right thing by abstracting macros and registers into functions that look and feel the same — and if the downside is that the syntax doesn't naturally support truly local variables and registers, which have seen extremely limited use in TeX due to the reasons outlined above, then from my perspective that's an acceptable trade-off.

[EvanAad]

@wspr But the examples you cited as the ones the broke the camel's back are examples where the programmer abused the language by assigning globally to a local variable. This is on the programmer. The current system doesn't solve the problem of abuse of language, as a programmer can still use an \l_... variable as though it was global.

[FrankMittelbach]

... that is in agreement with the principles of structured programming.

that's quite a bold statement given that there are more than one set of principles and given that attempts to unite different principles in supersets have usually created big but useless languages in the past. The crucial point is not necessarily that something can be done (as within Turing complete bases all things are equal on that level) but whether or not something can be done efficiently and consistently, etc.

[EvanAad]

@wspr : Why did you close this issue like a bully? It is anything but settled.

[wspr]

@EvanAad — just trying to keep things tidy. Discussion can certainly continue and if warranted we will re-open.

[wspr]

I'm locking these issues temporarily due to some heated comments.

[blefloch]

I think this is a relevant question. One outcome of this discussion should at least be that we describe more carefully in the doc why we chose global declarations only. @EvanAad I don't think you quite appreciate the limitations of the TeX system, but let us step back and ignore that issue for a moment. Could you give an example (of 10-20 lines, say) using a hypothetical \int_local_new:N or a different system of your choice with the semantics of your choice? It would help frame the discussion in a more concrete setting, and we can hash out benefits/drawbacks.

Presumably we have to wait for the lock to lift, I don't know how to do that. In any case most people in this conversation are going to be sleeping for a few hours. (By the way, I don't think copying #410 into #411 as a jumbled discussion was very reasonable.)

[josephwright]

Looking over the fact this is quite a detailed discussion, I think it's worth summarising the technical and social history that got us to where we are.

At a technical level, TeX provides registers and macros for storage. Macros can just be 'created' using \def, but to use registers by name we need some allocator to link the global register number, e.g. \count40, to some name, e.g. \mycount. That has been provided from 'day one', with the plain TeX \newcount, etc. providing the model. In plain, \newcount allocates globally and, despite the name, doesn't do any checking. Other formats, most notably LaTeX2e and ConTeXt, have both adopted this approach in general terms. They have also generally adopted the idea that individual registers are then assigned either locally or globally, as this prevents save-stack buildup.

TeX90 provided only 256 registers of the common types, so register re-use in local contexts was vital. One sees that in e.g. graphics where to keep the code 'sane' a large number of registers are given new names inside a group, and used purely locally. With e-TeX we have a lot more registers, so this approach is less necessary: we can freely allocate more registers (and macros) for dedicated purposes. In particular, it means that we are nothing like as pressurised to 'recycle' registers under multiple names.

The code base of expl3 has been in development for a long time, and originally did not require e-TeX. Development of expl3 is also done primarily 'on top' of LaTeX2e, with a general principal that expl3 doesn't pollute the document name space nor change core LaTeX2e behaviours.

In particular, we have to bear in mind that LaTeX2e's \newcount is similar to plain's: it allocates globally and does not checks. Thus in particular

\def\foo{%
  \begingroup
    \newcount\localfoocnt

is 'bad style' as if \foo is called multiple times then we will use up registers which are never released.

For expl3, the team have tried to avoid tying the documented behaviour of variables to implementation of macros or registers. (I do note that we allow use of tl without an accessor, which ultimately does rely on them being macros.) We also still make use of registers for integers, dimens, etc. as they are available and offer better performance and 'self termination' than doing everything in macros. (That would be doable using e-TeX and the various \<thing>expr primitives.) As such, we need to have an allocator system, and to be consistent we allocate all variable types, not just those which are based on registers. As @EvanAad has observed, when checking is not active one can do e.g. \tl_set_eq:NN \l_new_tl \l_existing_tl with no error, but this is not the supported behaviour (checking will flag it up, for example).

As part of the expl3 allocator, the decision was made that new would check for existence, in contrast to \newcount, so

\cs_new_protected:Npn \my_foo:
  {
    \group_begin:
      \tl_new:N \l_my_tl

will raise an error if \my_foo: is used repeatedly. This pushes one in the direction which has been standard TeX practice since the first days of plain: allocations go outside of macro. (Note that in plain, \newcount is \outer so is 'forbidden' inside macros.)

\tl_new:N \l_my_tl
\cs_new_protected:Npn \my_foo:
  {
    \group_begin:
      \tl_clear:N \l_my_tl

The etex package introduced many years ago a register allocator which can do local as well as global allocation of register, \loccount versus \globcount, _etc. For quite some time, expl3 loaded etex and it was used to provide functionality for \int_local_new:N and similar. Given the fact that most TeX programmers are used to doing global allocation, it is perhaps unsurprising that this was not taken up so widely. However, the team were also concerned about possible misunderstandings. With something like

\cs_new_protected:Npn \my_foo:
  {
    \group_begin:
      \tl_local_new:N \l_my_tl

we have the question of what happens on nested calls to \my_foo:, or more likely places where some 'simple' name such as \l_my_tmp_tl is to be used. Based on the general idea that we want new to do checks, that should be an error. Programmers coming from a lot of other languages might find that somewhat odd. Of course, it does fit with TeX's scoping rules.

Notably, changes to the LaTe2e kernel in recent years mean we'd no longer want to load etex (indeed, we've worked hard to make expl3 'self-contained'). So any new local allocator would have to be written to work with LaTeX2e without 'disturbing' behaviours for those not using expl3: doable but not entirely trivial.

For something like \l_my_tmp_tl one might of course still do global allocation, but then one has to know which \l_... variables are locally allocated in the current group and which are globally allocated but locally assigned.

I've not got test data to hand, but it is probably worth noting that allocating a variable is more work than simply setting it (certainly when checking is not active), so using a local allocator would be slightly slower than a global allocator plus local assignments.

Thus for a mix of technical reasons and 'fitting with history' we decided to stick to strictly global allocation. This doesn't in any way prevent local _assignment of variables, which are encouraged and very widely used.

That takes us to the 'social' element. Take up of expl3 in recent years has been strongly aided by making parts 'broadly' stable. At some stage the team have to make a decision on stuff, partly so people can use it and partly so we move on to other tasks. That doesn't prevent us revisiting things, but more established conventions need a good reason to be altered.

Here, at present I guess I'm not seeing what is wrong with the 'usual' approach of

\tl_new:N \l_my_tl
\cs_new_protected:Npn \my_foo:
  {
    \group_begin:
      \tl_clear:N \l_my_tl

at least to the point where it's required that we go back and change the current set up.

[EvanAad]

@blefloch Yes, I agree that my reaction to @wspr 's closing the thread was unreasonable and I apologize to him and to the rest of you. I felt I was being dismissed and silenced, and I reacted with a tantrum. This is unacceptable. Sorry about that.

@josephwright thank you for the detailed reply. After giving the matter some thought, I've realized that if I only use \<module>_clear_new:N, then in effect all my variables will behave as though they were local to the surrounding group, as long as I assign using \<module>_set... rather than \<module>_gset.... Moreover, I think this practice agrees with the LaTeX3 convention of declaring a variable before use, so I won't be flagged if the checks are on.

Following this practice, I would rewrite @josephwright last example thus:

\cs_new_protected:Npn \my_foo:
{
    \group_begin:
        \tl_clear_new:N \l_my_tl

The only data type that doesn't fit this schema is cs, but as far as I know it is legal to create variables of this type with \cs_set:Npn et al. without first declaring them, because LaTeX3 doesn't really treat cs as a normal data type on par with the others (which is a separate issue I'd like to take up with you guys, but I'll leave it to another thread). Using this approach, cs variables too can be created locally. If I want to keep syntax consistent, I can always write my own \cs_clear_new:N wrapper.

So, as far as I'm concerned, now you can close this issue if you want to, @wspr .

[josephwright]

@EvanAad On \<thing>_clear_new:N, notice that the declaration is global where the variable doesn't already exist. So

\cs_new_protected:Npn \my_foo:
{
    \group_begin:
        \tl_clear_new:N \l_my_tl
    \group_end:
}
\my_foo:
\tl_show:N \l_my_tl

will show that \l_my_tl is defined and empty, assuming you had not previously set it to something else.

[EvanAad]

I'd like to join @blefloch in hoping this discussion will lead to better documentation. In particular, I think that when you write that the programmer "must" follow a certain pattern of coding, as in the sentences I quoted in my original post, the documentation should elucidate what this "must" mean. What will follow if the pattern is not adhered to?

1. Will the language's behavior be undefined?
2. Is non-adherence currently supported, but may not be so in future versions?
3. Will the engine report an error?
4. Will the engine report an error, but only when the checks are on?
5. Will some minor inconveniences result?
6. Will the LaTeX3 team grumble in discontent, but no error or loss of functionality result?

As an example for 5, take the convention of appending argument specifications to the end of a function's name. As far as I can tell, the only function that won't work properly if this convention is not adhered to is \cs_new:Nn, but the rest, including `\cs_new:Npn' will work perfectly fine.

As an example for 6, take the convention of marking local and global variables with \g_... and \l_.... As far as I know there will be absolutely no repercussions to not following this convention. Everything will work properly.

[wspr]

@EvanAad — apology accepted, and I'm sorry in my part for closing the issue before the discussion was complete.

@josephwright — as one teeny tiny advantage of benefits to local allocation, if I write

\cs_new_protected:Npn \my_foo:
  {
    \group_begin:
      \tl_new_local:N \l_my_tl
      ...

then I know that \l_my_tl is not only free from interference from the outside, but unlike using a _clear function I know that all trace of the variable is gone outside of the use of the function. In other words, just by looking at the code I know that it cannot be used as a semi-global variable down the line. (And I can check that nothing odd is going on by using \tl_if_exist_p:N.)

(Must run but can continue later if there's any point discussing further.)

[josephwright]

@wspr Yes but that's back with the fact it is based on TeX group scope. So it can be used 'semi-globally' in a construct of the form

\cs_new_protected:Npn \my_foo:
  {
    \group_begin:
      \tl_new_local:N \l_my_tl
      \tl_set:Nn \l_my_tl { foo }
      \__my_foo:
     ..
  }
\cs_new_protected:Npn \__my_foo:
  {
    \group_begin:
        \tl_use:N \l_my_foo % Definition from \my_foo: => "foo"
...
  }

which was at least part of our thinking.

[josephwright]

I think before we leave this I would like to emphasise that at a technical level there are a number of ways to set up a local register allocator.

[wspr]

@josephwright — I’ve never seen that as a confusing example though; in Matlab for example they distinguish between sub functions which don’t share scope, and nested sub functions which do. So to my thinking that’s always been, well, why wouldn’t \__my_foo: inherit the scope of the “outer” function? It’s perfectly consistent with TeX’s grouping behaviour.

[wspr]

(Sorry to keep the discussion going. It's been a long day. Is anyone interested in actually continuing?)

[FrankMittelbach]

Am 19.10.2017 um 09:23 schrieb Joseph Wright:

I think before we leave this I would like to emphasise that at a technical level there are a number of ways to set up a local register allocator.

yes but fairly close to the Turing argument, ie any such implementation is going to be very inefficient at runtime because the underlying engine is managing the register storage globally. And expl3 (on that level) should stay "reasonably efficient. that is like, say, global and local variables and their mutator. Rather than testing in each function if it is doing a global operation on a local variable, the concept is by default only present in the names, e.g. \l_... is supposed to be a local variable and should not be used with a global function like \..._gset:Nn but we aren't testing that at runtime However we do offer a check module (which runs how many times slower) that makes sure that all these conventions are actually obeyed. So to come back to global/local variables the standard concept of expl3 is - declare the name of a variable once (globally) -- many because at least some of the types have only global storage bins - use the name convention \l_ \g_ to denote local and global variables that gives you both globals and locals but has the restrictions that - you don't declare a local variable at the beginning of its scope, instead you either use _set or _clear_new at that point and the latter may mean that the name of the variable may come at that point globally in existance - outside of the scope the variable still exists with the default value of the type (eg _int being 0 etc) so you don't get a compiler error if you refer to such a variable "by mistake" outside its intended scope So basically the only thing you don't get is being able to declare a local variable so that its name vanishes outside the scope (and produces an undefined error of some kind at runtime if referred to outside the declared scope). To do that (which yes is possible) expl3 would need to maintain its own resource pool far beyond a simple association between a name an a global pool offered by the engine and that would mean *all* access would be very notably slowed done - which goes against the design criteria of expl3.

[wspr]

@FrankMittelbach — now you have me curious; is the etex.sty approach for registers really that inefficient? Or maybe you mean for tl variables, in which case I agree!

[EvanAad]

@FrankMittelbach In my opinion it is important to distinguish, and make it clear in the documentation, what is a best practice according to the LaTeX3 team vs. what is a formal requirement, either grammatical or semantic, of the rules of the LaTeX3 language.

The two rules that you cited, namely:

• declare the name of a variable once (globally)
• use the name convention \l_ \g_ to denote local and global variables

fall under this rubric of coding conventions that the LaTeX3 team deems advisable, but neither of them is mandated by the grammatical rules of the language, and not adhering to these conventions causes no error or loss of functionality.

In other words, following these rules is a matter of personal taste and coding style, and this should be made clear in the documentation, in my opinion.

[FrankMittelbach]

that is a bit like saying that the sign 50mh is not a rule but a driving convention and it is a matter of taste if a driver adheres to it (just because it is not immediately checked most of the time) if you use a \l_ variable with _gset then you still program in TeX but you have stopped obeying to the grammatical rules of the expl3 language. Does it immediately break your code? probably not, but you generate savestack build-up (see TeXbook index) are you saying it is only a grammatical rule if we are checking it at run-time, say, each Tuesday?

[EvanAad]

@FrankMittelbach Sure, if you define this to be part of the rules of the language than it is so by definition, but my point is that this should not be defined as part of the rules of the language, because it is never checked, and because not adhering to this convention does not in itself cause an error or loss of functionality.

It's the difference between saying "When you write English it is best to hold the pen in your right hand because this avoids smearing the ink." and passing a law that says that English must be written with the pen held in the right hand. Sure, you can pass such a law, and there is sound rationale to justify it, but ultimately the choice in which hand to hold the pen should be left to the individual writer. And there will be people who will choose not adhere to this rule, and still end up writing just as tidily as those who adhere to it.

[blefloch]

I think @FrankMittelbach overstates the overhead of local declarations (\loccount manages counts locally, nothing it does is global, so that keeps the overhead fine).

I see no major issue with providing an \int_local:N and \tl_local:N (=\tl_set_eq:NN #1 \c_empty_tl) etc that would be very analogous to \int_zero_new:N and \tl_clear_new:N but would only do "new" locally. This requires a bit of work for registers but not excessively so.

@EvanAad you should know that not declaring a variable may bite you at some point (warning: this code produces infinitely many pages).

\documentclass{article}
\usepackage{expl3}
\begin{document}
\ExplSyntaxOn
\tl_put_left:cn { l_my_tl } { foobar \par }
\l_my_tl
\end{document}

[EvanAad]

@blefloch What I'm saying is, it's perfectly fine to declare a variable inside of a macro (with \<module>_clear_new), and you don't have to name it \l_amount_paid_int, you can simply call it \amount_paid or \amountPaid, like you would in any other programming language. The \l_..._int is a nice mnemonic to remind you that it's integer and that it's supposed to be used locally, but you should not be forced to use this, or any other mnemonic.

[FrankMittelbach]

because it is never checked, and because not adhering to this convention does not in itself cause an error or loss of functionality.

but that's the point a) it does cause harm depending on circumstances -- namely when you mix global and local assignments to the same variable b) we do check upon request (and right now that code may not be functional but it was and probably will be again eventually because of a)) ps yes I get your point about pen holding (being left-handed) and yes I have been above speed-limit (on my bike) but still I consider this a traffic rule not a traffic convention and yes one can disobey that without harm, but one can also die from it or at least end up with a fine

[blefloch]

(What my previous code example was pointing out is that even for tl it is essential to declare them before use. There is a checking option that tests declarations but in normal use we don't want such overheads.)

I agree with @EvanAad that names are just a convention. We can check that local and global assignments are not mixed even without the l_/g_ name convention: since the checking code is allowed to be somewhat slow it is quite reasonable to store the information about whether a given variable was used in a local/global assignment. For checking types the situation is similar, except for two pairs of types that cannot be distinguished (I'm not going to say which to avoid derailing the conversation).

[josephwright]

On 'conventions', \amountPaid is a document command whilst \l_amount_paid_int isn't, and the latter is part of the amount namespace (by convention but a pretty important one in TeX). That doesn't apply to \l_/\g_, though I think many languages have a strong 'steer' on naming without it being enforced at a technical level.

[eg9]

On 19 Oct 2017, at 16:58, Joseph Wright ***@***.***> wrote: On 'conventions', \amountPaid is a document command whilst \l_amount_paid_int isn't, and the latter is part of the amount namespace (by convention but a pretty important one in TeX). That doesn't apply to \l_/\g_, though I think many languages have a strong 'steer' on naming without it being enforced at a technical level.

The worst nightmares in LaTeX2e programming is defining a command in the “user level space”, say \foo, to discover that it clashes with a command another package defined for _internal_ use (so not documented in the manual). Did it really happen? Yes, and not just once. Sticking to the \<prefix>@commandname convention helped very much in avoiding such problems. Of course clashes at the “user level space” can happen as well, but they’re much easier to spot and solve. When it comes to the internals, it’s often necessary to chase expansions at a very deep level. Differently from legislators, we cannot impose fines on or jail who doesn’t adhere to the laws of LaTeX3 programming. But we’re a community and everybody should. Our guidelines on naming conventions should help in never find a package's internal commands to clash with others’. For personal code one is entitled to do whatever they want: there’s no law disallowing one to speed in their private property, but there is one about doing that on a public road. If you want to locally define the variable \f of whatever type, you should be worried about the command being defined by some package and, by Murphy’s law, ending exactly in the argument to a function using the variable \f. Can you imagine some worse scenario? Ciao Enrico

[u-fischer]

and you don't have to name it \l_amount_paid_int, you can simply call it \amount_paid like you would do in any other programming language. The \l_... is a nice mnemonic to remind you that it's supposed to be used locally, but don't have to use this, or any other mnemonic.

Sure. You could also use \amount:paid or \amount&paid (that's not a joke, I know a package which use &) or whatever your prefer. But even if the names are just a convention: it makes communication easier if people stick to such conventions. You have been asking quite a lot questions on tex.sx. What will you do if you have a problem with your code written in "personal style"? Translate it to standard style, ask a question and translate it back? Or expect everyone to be able to handle your personal style?

[EvanAad]

@eg9

For personal code one is entitled to do whatever they want

You wouldn't know this by reading the documentation, is all I'm saying.

If you want to locally define the variable \f of whatever type, you should be worried about the command being defined by some package

I disagree. If your code is inside a \group_begin: ... \group_end:, and if you define all local variables with \<module>_clear_new:N, and if you only assign to local variables with \<module>_set:N..., you needn't worry about your local variables name-clashing with other packages, unless your code uses another package.

@u-fischer

But even if the names are just a convention: it makes communication easier if people stick to such conventions.

I'm not saying there are no good reasons to stick to the conventions. All I'm saying is, these conventions should not be made language rules, and the documentation should clearly differentiate between conventions, for which the rationale should be stated, and rules. And the choice whether to follow the conventions should rest ultimately with the programmer.

[blefloch]

@EvanAad You do need to worry about name-clashes even in the case you describe. Say you write

\cs_new_protected:Npn \evanaad_halve:n #1
  {
    \group_begin:
      \int_zero_new:N \f
      \int_set:Nn \f { (#1) / 2 }
      \iow_term:x { \int_use:N \f }
    \group_end:
  }

then a user of your package does

 \int_const:Nn \f {123}
 \evenaad_halve:n { \f }

They will be surprised to see 0 and not 62.

[blefloch]

On the other hand if you stick with names such as \evanaad_f etc (or shorter \@@_f etc using l3docstrip magic) you should be safe.

[u-fischer]

You wouldn't know this by reading the documentation, is all I'm saying.

Sorry but expl3.pdf uses the word "convention" around 20 times. In the case of public versus private command there is even the sentence "There is (nearly) no way to enforce this without severe computing overhead, so we implement it only through a naming convention,".

[EvanAad]

@blefloch Good point.

@u-fischer OK, fair enough. What about the convention that a function's name should end with an argument specifier? This is not entirely a convention, because the functions of section 3.3 inspect the argument specifier, but these functions are merely "syntactic sugar", and if you don't use them, there's no hindrance using function names like \mymodule_myfunc, but you wouldn't know it from the manual.

[eg9]

On 19 Oct 2017, at 17:52, EvanAad ***@***.***> wrote: @u-fischer OK, fair enough. What about the convention that functions' names should end with an argument specifier? This is not entirely a convention, because the functions of section 3.3 inspect the argument specifier, but these functions are merely "syntactic sugar", and if you don't use them, there's no hindrance using function names like \mymodule_myfunc.

That’s necessary for \cs_generate_variant:Nn, of course. Ciao Enrico

[blefloch]

With `\mymodule_myfunc` you would not be able to use any function from `l3expan`. These expansion functions, and the notion of variants, are a core part of expl3. While I agree that variable names could be made shorter (removing "l_"/"g_" and "_int"/...), function signature are really not "just a convention".

[EvanAad]

@blefloch I see your point, and it is a good one, but still, in my opinion if all you want is to write a document command, you should know that you can do so by defining a function like

\ExplSyntaxOn
\cs_new:Npn \MyDocumentCommand {Hello,~world!}
\ExplSyntaxOff

and you don't have to first define a "shadow" function \mymodule_my_document_command:, and then copy it with

\cs_new_eq:NN \MyDocumentCommand \mymodule_my_document_command:

[EvanAad]

@blefloch And, by the way, beside \cs_generate_variant:Nn, which @eg9 mentioned, do any other functions of the l3expan module make use of the argument specifier part of the function name?

[FrankMittelbach]

by now it sounds to me that you are largely arguing for the sake of arguing yes you can do all this and by the end of the day the only hard language rules are what the TeX engine hard-wires in its primitives. And given that TeX is a self-modifying language you can go basically anywhere from there eg \endlinechar-1\def~#1{\catcode`#113}~Q~S~U~_~V~W~J~K~L~M~N~O~@~X~Y~[~]~( ~|~&~Z~'~"~*~h~z~:~q~j~k~;~/~)~!~,~$~+\let_\let_~\newcount~$$-1~Q~J~V~S ~K~W~U~L~,~''1~""2~**1_&\count&144'&155'&145"&154"_[\ifnum_(\ifcase_O\or _|\else_]\fi_N\number_@\advance_X\expandafter_Z\global_Y\typeout_~\newif ~\ifG~\if_~\def~j{[0<Q[9>Q[0<J[9>Jk|$]|$]|$]|$]}~k{&1NQNJ}~\2#1#2{}~:#1{ #11#12#13#14#15#16#17#18}~h#1#2{#2:{~\q#1}~#2^^J}~\q#1#2{(&1#1#2~~O.O$]} ~/{Y{Row and column? e.g. E6}\read$toM\ifcat~X\2M~$$X\jM|!input!]}~!#1!{ Y{Invalid #1.}/}~\j#1#2{Q`#1@Q-`@J`#2@J-`0;(V!move!]}~;{V0 (jS1z1z0z{$}S 0z1z{$}S$z1z0z{$}]}~_{@,\ifodd'-]}~z#1{{\trueK#1{\falseq}}}~q{@qs@JK[j=" \ifZk'Z_2]@V1q|[j='ZVV\ifG\if|\aftergroupq]]]]}~\,#1{Q#1:\.}~\.#1{J#1;[0 <V@V5@V+Q@V+J~[V>WWVUQLJ]]}~+#1{(#1O2O-2O0O0O0O0O-2O2]}~){'X"X"N'Y{^^J : ~^^Jh1Ah2Bh3Ch4Dh5Eh6Fh7Gh8H :~^^J}\GfalseW(W$|0]~:\,\Gtrue[0<W[*='QUJL| /];k'_1][$=WY{(,Tie| Player [0>,.|$]~wins by N[0>,-],].}X\dump])}~~{ }) which is a beautiful TeX document (in fact a beautiful LaTeX document) but as far as its code is concerned it is not at all very useful. And the fact that Bruno is able to write such a document doesn't mean that the expl3 manual (or in this case a LaTeX manual) should describe any of it. LaTeX (2.09 and also 2e) code had the big problem that in the early days too many people programming it did understand about low-level shortcuts in TeX and used and misused them because they did think it is not harmful. As a result a huge amount of existing 2e packages are incompatible with each other or have subtle issues in places when used together etc or break once in a while because they bypassed one or the user interface (because it seemed worked without it). You are basically asking us over and over again to document just this, i.e., possible shortcuts that that violate the design principles just because they sometimes work (or even at the moment always). But expl3 and its conventions/rules are largely derived from experiencing that coders disobeyed such rules in the past and the mess that is the result from this. So no, the rules are deliberate and not just mere whims (most of the time) and even though "if you know what you are doing then you can bypass most of them in a specific situation" that doesn't mean that if you move your code then from one place to the next that will still be the case or does it need to be the case over time. As said by others, there is a big difference between code you write on the fly for yourself and code you write as "officially" distributed package. A least for the latter, we ask for accepting the rules and consider them being part of the language. For yourself, you may want to learn how to code a document like the above but the knowledge how to do that will not come out of the expl3 manual

…

--------------------------------- having said that, I don't want to discourage you challenging concepts, commands, interfaces, what have you. Many of of points you raised on other occasions have been well taken (or at least got us rethinking one or the other point). But as far as the expl3 manual is concerned I think what you heard from several people is that there is no interest in documenting the "non-conventions" and the "non-rules". Moreover if code diverts too much from what we call rules/conventions then it will still be TeX code but no longer expl3 code.

[josephwright]

I think we've talked this one through: I'll close but of course will reopen if requested.