Feedback for documentation inconsistencies and unnoticed bugs

[stefanos82]

In Global Symbol of documentation, we have the following example:

global function global_f()
end

This throws an error when called:

useglobals.nelua:3:6: error: cannot handle type "void" in print
print(global_f())   -- outputs: f
     ^~~~~~~~~~~~


shell returned 1

If I add return 'f' inside global_f(), I get the desired output.

What is the behavior we initially expected in the first place?

[edubart]

Because global_f returns nothing, so you can't print nothing, you must return at least one value to be able to print. Maybe you expected return nil instead but the Nelua type system at the moment does not convert empty returns to nil, but this may change in the future so some lua style codes are more compatible

[stefanos82]

I guess the documentation needs a bit of clarification, because in the screenshot below this is what we currently see:

I just follow the documentation to understand the language in detail.

[edubart]

Oh, it's a mistake in the documentation.

[stefanos82]

I see. Well, just add a return 'f' inside the function and it will print the desired output.

[stefanos82]

By the way @edubart can we transfer this to Discussion? I will be testing the whole docs and in case I find other buggy code samples, I will be reporting it there instead of opening multiple tickets.

Is that OK with you?

[edubart]

Yes, I am ok with this, continue posting here any mistake you find in the documentation.

[stefanos82]

Thank you @edubart

[stefanos82]

In reality it prints false test 1 3.14

[stefanos82]

In condots.nelua we have the following code:

local function poll_events()
  local event: SDL_Event
  while SDL_PollEvent(event) ~= 0 do
    switch event.type
    case SDL_QUIT then
      return false
    end
  end
  return true
end

but in docs we have

Is do optional in this case or is it a bug? The code compiles and works as expected 🤷‍♂️

[edubart]

Is do optional in this case or is it a bug? The code compiles and works as expected

The do keyword following the switch expression is currently optional at the moment. At first it wasn't possible to use do, but a user suggested to add the do keyword, so it was added just as an optional syntax sugar.

[edubart]

I've fixed the following things you mentioned in the documentation:

• Missing the return 'f' in global symbol section
• 3.14 output instead of 3.1400000 in variables section
• Omitted the do keyword in switch statement (using the do is optional and will remain as optional hidden feature, I won't document this possibility as this may be removed in the future to not have extra syntax).

Thanks for reporting these inconsistencies, once you feel there is nothing more wrong in the documentation you can close this issue.

[stefanos82]

Sure thing @edubart, I currently work on implementing syntax highlighting for Vim editor which I use and it's a nightmare 😄 that's why I report all these inconsistencies; I'm trying to make it work as expected.

As soon as I feel it's ready I will create a repo around it for people to use it and provide feedback accordingly.

[edubart]

Sure thing @edubart, I currently work on implementing syntax highlighting for Vim editor which I use and it's a nightmare that's why I report all these inconsistencies; I'm trying to make it work as expected.

As soon as I feel it's ready I will create a repo around it for people to use it and provide feedback accordingly.

If you are working on a syntax highlighter plugin I highly recommend using the compiler PEG grammar as reference that just shipped the master branch a few hours ago. This file:

https://github.com/edubart/nelua-lang/blob/master/nelua/syntaxdefs.lua#L14

You can read the full syntax definition in a compact form there as reference (assuming you understand LPEG's re syntax).

[stefanos82]

Very nice! I will double-check everything right now; thank you for the feedback.

Vimscript's syntax is pure...magic? Yeah, I think you understand what I mean 😑

[stefanos82]

I have found two types that are undocumented: union and variant.

[stefanos82]

I'm almost done with nelua.vim syntax highlighting, but I'm facing issues with a few mechanisms:

• the - operator; it affects how single line comment mechanism works and I need to figure out a cheat method to fool it.
• the Do expression is killing me; don't ask!
• anonymous functions confuse not only the syntax highlighting mechanism, but also yours truly.

About the last one, can you please explain to me how the following code actually returns its result as argument?

local function g(x: integer, f: function(x: integer): integer)
                              -- ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  return f(x)
end

local y = g(1, function(x: integer): integer
  return 2*x
end)

print(y) -- outputs: 2

Is there an implicit return mechanism that takes place, like ECMAScript's Lambda Expression () => ...?

Update: To resolve the issue with lambda functions, all I need to add is an end at the end of lambda function and makes the syntax highlighting stop complaining.

[edubart]

• the - operator; it affects how single line comment mechanism works and I need to figure out a cheat method to fool it.

Usually a parser can differentiate -- from - with some simple rules in the tokenizer. You should check how vim does for Lua, should be the same.

anonymous functions confuse not only the syntax highlighting mechanism, but also yours truly.

Nelua has 4 contexts when parsing with different rules:

• statements (inside blocks)
• expressions (inside some statements)
• type expressions (after expression preceded by @, or after : in declarations)
• preprocessor (after ##, between #| |#, or between #[ ]#) (use Lua syntax highlighting in this context)

The syntax highlighter should follow different rules according to the context, for example, when in the type expression context, an end is not needed after a function.

[stefanos82]

Usually a parser can differentiate -- from - with some simple rules in the tokenizer. You should check how vim does for Lua, should be the same.

It's based on lua.vim that I extended it to match Nelua's behavior.

Nelua has 4 contexts when parsing with different rules:

* statements (inside blocks)
* expressions (inside some statements)
* type expressions (after expression preceded by `@`, or after `:` in declarations)
* preprocessor (after `##`, between `#| |#`, or between `#[ ]#`) (use Lua syntax highlighting in this context)

I have accomplished this, but not 100% thanks to (do end); I really need to get feedback from an experienced Vim user.

The syntax highlighter should follow different rules according to the context, for example, when in the type expression context, an end is not needed after a function.

I know, but I don't have that kind of knowledge, at least with Vim scripting.

[stefanos82]

Work so far looks something like this:

[edubart]

Looking good!

[stefanos82]

When we find an expert around Vim syntax highlighting scripting, will be very handy because I'm sure it needs a major refactoring on lots of things.

Basically I'm hacking around to make it produce the desire output, nothing more; thus the need for refactoring.

[stefanos82]

When you find time, can you please provide some new examples based on primitive numeric types' suffixes?

Preferably for

• _is, _isize
• _us, _usize

[edubart]

I've added one example with _isize in the overview, I don't see the need to add more numeric literals examples there because the suffixes list already shows all of them.

[stefanos82]

These were the two I did not find an actual example; for byte I have found from Nelua itself how it uses it in the form of 'A'_byte.

[stefanos82]

@edubart this is to announce the release of nelua.vim.

It's not ready yet, but it's usable at 95% of it.

If you know any Vim expert, please let him know about the syntax highlighting project as I need advice and feedback.

[edubart]

Nice! Although I don't use vim nowadays, I have used it before for a while, I will checkout and give a try!

[stefanos82]

No worries, take your time.

[stefanos82]

@edubart was there any change in Importing C functions declared in headers ?

This is what the example shows it generates, this is what I get as C code:

This is of course a lot better than the showed example in overview.

[edubart]

I've updated the examples in line with recent C code generator changes.

[stefanos82]

I have just pulled the latest changes.

Why do I get a different generated C code?

[stefanos82]

Seems like you pasted the same screenshot twice.

The generated C code output was for Importing C functions, but pasted it for Importing C functions declared in headers as well.

[edubart]

Seems like you pasted the same screenshot twice.

The generated C code output was for Importing C functions, but pasted it for Importing C functions declared in headers as well.

Yep, fixed.

[stefanos82]

What is "foo"'s meaning supposed to be?

[edubart]

The typo was fixed.

[stefanos82]

I get 16 as an output.

[edubart]

The example was fixed.

[stefanos82]

What should be the actual output? In overview displays the same as the screenshot above and I still output 16.

[edubart]

The example was changed to:

local Vec2 = @record{x: int32, y: int32}
print(#Vec2) -- outputs: 8

You may need to refresh your page and github may take some minutes to invalidate page caches.

[stefanos82]

Ah you changed it to int32; that was the problem.

[stefanos82]

For output I get

[edubart]

The typo was fixed.

[stefanos82]

So, an interesting behavior is happening with Nelua.

In the directory I work I have named a module string.nelua; this becomes the immediate source of loading and breaks the whole logic of any program that depends on string module.

Isn't there a namespace approach to look for the system modules first and then for a user's modules, much like we have in C with <foo.h> vs "foo.h"?

[edubart]

This is expected, just rename conflicting module names with any standard library module to something else, for example you could move to a folder. This allow the user to override any Nelua standard library if he wants.

But I've put some thought on this, having a way to include system modules first can be useful when overriding, so I've added an extension to require to give priority to system modules first when prefixed with ~. Example, let's say you want to extend the string methods to add a string.split, you can now do the following:

In string.nelua inside your project:

require '~string' -- will require system's 'string.nelua' instead of this file

function string.split(s: string, sep: string): sequence(string)
  local rets: sequence(string)
  for s in s:gmatch('[^'..sep..']') do
    rets[#rets+1] = s
  end
  return rets
end

Then in example.nelua:

require 'string' -- will require current `string.nelua`

local strs = string.split('a;b,c', ';,')
for i,v in ipairs(strs) do
  print(i,v) -- outputs "1 a", "1 b" and "3 c"
end

[stefanos82]

So all examples now that use global modules need to be prefixed with tilde?

[edubart]

So all examples now that use global modules need to be prefixed with tilde?

No, this tilde is just for forcing to look up modules in reverse order in case your project has a conflict module named string.lua (which you should avoid). You can mostly ignore this tilde extension existence, I won't document it soon either, will remain a hidden feature for a while. Old and current code should continue to use require 'string'.

[stefanos82]

Seems like we have an issue; with the latest changes, I cannot compile overview.nelua.

[stefanos82]

Ah I see; I had create_value, but you refactored the code to value.

[stefanos82]

In FAQ Why does Nelua compile to C first? we have the following bullet list

The generated C code can be reused if the user someday decides to move away from Nelua. The user can just take the generated C code, since it is very human readable, and can continue using C/C++.

Can we add a star that clarifies the use of metaprogramming? As far as I can tell, the preprocessing magic is not included in generated C code, therefore it could give the false impression of "write whatever you like in Nelua, metaprogramming included, and will be generated as C code".

If I got it wrong please enlighten me @edubart.

[edubart]

Can we add a star that clarifies the use of metaprogramming? As far as I can tell, the preprocessing magic is not included in generated C code, therefore it could give the false impression of "write whatever you like in Nelua, metaprogramming included, and will be generated as C code".

Well it's kinda obvious that meta programming logic is not included in the generated C code. It's common to use meta programming to generate code with different specializations, but the generated code is still readable and usable C code. I could argue that even more readable than the meta programmed code itself, because metaprogramming sometimes can make things not that easy to read.

For example a vector(int) is implemented with some meta programming logic in Nelua, yet the C generated code is very readable and usable, when you do vector(string) you have another generated vector code for specialized strings also usable. Thus that meta programming code is included and still readable in the generated C code, but it's just in its "rendered" form.

Meta programming in Nelua is just a template engine like in the web development world, actually it's inspired by how HTML pages are rendered in many web frameworks, but instead of HTML elements, here we have AST nodes of statements. In many web frameworks people can use a library to generate HTML, and later if the user decides to move to another web framework he can get the rendered HTML and continue his thing.

[stefanos82]

Hmm...maybe the original Nelua code could become part of comment series during preprocessing procedure, to indicate which part run first and produced which C code exactly?

That of course could be handy to enable it with a flag and not have it active by default; less noise and code pollution, the better!

[stefanos82]

My output is 1.0 2.0; same goes for Record Methods

[edubart]

Fixed.

[stefanos82]

print_info_bycopy(p) should be print_info_bycopy(pref)

[edubart]

Fixed.

[stefanos82]

In Preprocessor macros emitting AST nodes, can you add the following code at the end of it to make it obvious of what it actually generates?

local a = #[create_sequence(integer, 10)]#
for i=0,<#a do
  print(i,a[i])
end

[stefanos82]

Congrats on breaking a new compile-time record!

That's the overview.nelua's total time to compile and execute...one word: DUDE!!! 😮

[stefanos82]

@edubart if I wanted to add inline assembly in my Nelua code, how would I do that?

Is it something I can do via metaprogramming?

[stefanos82]

After recompiling the code, it worked...don't ask why, no idea 🤷‍♂️

[edubart]

I made a minor fix in a recent commit to make the above example work (ccinfo.is_x86_64 world always return false), I think you updated Nelua and fixed it.

[stefanos82]

Yeah, probably that was the remedy after initial output error message.

[stefanos82]

Yes, you could emit C code that uses the __asm__ extension to add inline assembly, but this is not supported by all C compilers and its syntax may vary between C compilers. Here is an example of an assembly for rdtsc instruction, a common x86_64 function used for counting CPU clock cycles:

require 'math'

local function rdtsc(): int64 <inline>
  local lo: cuint <noinit>, hi: cuint <noinit>
  ## if ccinfo.is_gcc and ccinfo.is_x86_64 then -- supported only in GCC x86_64
    ## cemit[[__asm__ __volatile__ ("rdtsc" : "=a" (lo), "=d" (hi));]]
  ## else
    error 'unsupported'
  ## end
  return ((@int64)(hi) << 32) | lo
end

local ITERATIONS = 10000
local start = rdtsc()
for i=1,ITERATIONS do
  local a <volatile> = math.sqrt(10.0)
end
local elapsed = rdtsc() - start
print('math.sqrt', elapsed / ITERATIONS, 'cycles')

Output:

math.sqrt	1.6255	cycles

Update (as of 2023-02-20)

Seems like tcc version 0.9.27 mob:bdec3c5 2023-02-03T18:04:56+01:00 (x86_64 Linux) can compile the generated C code without a problem.

I have even replaced

  ## if ccinfo.is_gcc and ccinfo.is_x86_64 then -- supported only in GCC x86_64

with

  ## if ccinfo.is_gcc or ccinfo.is_tcc and ccinfo.is_x86_64 then -- supported only in GCC x86_64

and works as expected.

Have I done it correctly @edubart ?

[edubart]

Yes, looks correct for me.

[stefanos82]

Output in https://github.com/edubart/nelua-lang/blob/master/docs/pages/overview.md#preprocessing-polymorphic-functions shows different result than mine.

I get 4 4.0, but in comment it shows 4 4.000000; which is the correct result?

[edubart]

This typo was fixed.