Remarks

[sveyret]

Hi,

I was hired a few years ago to make some Lua for embedded systems. Since I’m coming from the TypeScript world, I was missing the ability to type things, and so was looking for a project like Teal. Teal looks very promising and I’d like to push my company to use it. But in a production environment, I cannot take it before the syntax is stabilised and complete enough to meet our needs. Anyway, I hope I’ll convince them to give me some time to work on the project.

Here are my first remarks regarding syntax. Maybe some of my remarks are against the policy of the project, or already discussed. If so, please tell me…

Nil checking

​In Lua, every variable can be nil. But a nil variable may not be expected. It would be good to indicate that an attribute, function or method parameter can be optional (i.e. is nil-able). In TypeScript, we use question marks and exclamation points.

local optional?: string # This is a string, but allowed to be nil

​local function myfunc(mandatory: integer, optional?: string) # Optional may be nil (or not given at all)

end

local myvar?: integer # This variable is optional

myvar = 2

myfunc(myvar!) # This variable was indicated as optional, but I tell the checker that I know it is not

As the language is not stabilised yet, I think this change can be made directly, but if not, it could be protected with a “tlconfig.lua” option. TypeScript has such an option called StrictNullCheck.

Types and objects

We need 2 different ways to define types. “record” is currently emitting some code, making the type really exist in Lua. We need another kind of definition (if “record” were called “class”, I’d call it “interface”, but there, I think “type” or even “shape” would be a better candidate) which would only define the shape of what is expected and would totally be erased at compilation time. This can be useful for, for example, an “options” parameter to give to a function. We don’t need to emit anything in Lua, only to document the different options that may be given as entries in a table.

It would also be a good thing to be able to define anonymous shapes. We wouldn’t always need to name the expected shape of the “options” table. For example:

local function myfunc(options: shape {option1?: string, option2?: boolean, parameter3: integer})

This would only be used to make sure the function is called with appropriate options, but the emitted Lua file would simply be:

local function myfunc(options)

Inheritage

Regarding records, we should also have a syntax allowing inheritage like this:

local record Circle extends Object, Shape
…
end

By default, this would emit, as it is doing now:

local Circle = {}

One of the things that I think is important about Teal, is that it can interact with existing code. It is then rather easy to use it in an existing Lua project, migrating files step by step. Defining an entry such as “createObject” in “tlconfig.lua” would allow every existing code to use the record the way they want. The value of this option parameter would be a callable (either a function or an object with a metatable defining “__call”), called with the name of the type and the list of inherited types and returning the created type. A “shape” could define the default (inherited) properties of the created object. For example:

# tlconfig.lua
return {
  createObject: "src/object/create.lua"
}

# src/object/create.lua

local shape Object
  typeName: string
  metamethod __call: function(...): Object
end

local function createObject<T>(name: string, parents: {Object}): T
  …
end

return createObject

Consequences of inheritage

What would be interesting also would be to have the ability to indicate that a field is private, protected or public. Nothing would change in the emitted code, but the type checker would ensure proper usage.

It would also be interesting to have the ability to indicate that a method returns self. Indeed, if you have a class “A” with a method returning self, you can simply declare it as returning an object of type “A”. But then, if a class “B” inherits from “A”, then this declaration will not be totally right. If it is possible to directly indicate that the method returns self, the problem is solved.

Unions and type guards

In order to keep Teal as a type checker, I think it would be a good idea not to emit unexpected code. I’d rather see the “is” keyword used as in TypeScript. So, the type checker would detect code as:

if type(var) == "string" then

and would know that “var” is a string after the “if” condition.

If you want to create a union of any kind, it should be possible. And if you want to narrow the type, you either need to use the type(var) construction or to write a custom type guard:

local var: Fish | Bird

local function isBird(var: Fish | Bird): var is Bird
  return not var.swim
end

if isBird(var) then
  # The type checker here knows var is of type Bird
end

Enumerations

I saw that you were discussing about emitting code for enumerations. I think it is a good idea, but we need to keep a pure typing enum, not emitting any code. TypeScript defines the “const enum” for that. It would also be very useful to be able to define more restrictive types, maybe using unions which, as a first approach, could replace this “const enum”:

local type MySpecificType = "open" | "closed" | 0 | 100

Note in this example the usage of “type”, which could be a way of defining aliases for complex types used in multiple places.

Requires and unknown

Again, what I like with Teal is that it can interact with Lua which allows us to introduce it in a Lua project step by step. But for that, it is needed that we can require Lua files without having errors nor warnings. For that reason, type checker could make a best effort to guess the type of imported object, but by default, it should use any, and not unknown. So, if I require any Lua file, I don’t have any error, any warning.

On the contrary, “unknown” should be a type which can be given by hand to a variable. The difference between “any” and “unknown” is that the former can be assigned to any type while the latter cannot.

Metamethods

A simple note about metamethods : I’m not sure it is a better idea than using metatable (inside record) and defining the content of the metatable there. That’s because a metatable can have more than the defined metamethods, and a metatable can even have a metatable, and this does not seem to be definable in Teal.

Tools

This is more a question than a remark: are there already tools compatible with Teal ? Prettier, Luacheck, source map for debugging, etc. ?

I’d be glad to hear your thoughts about all this. If you’d prefer me to write multiple issues, please tell me also. Anyway, I also think it may be good to have a single issue ticket, which would be a reference for everything needed for version 1.0?

[lenscas]

Remind me to do a bigger response later but something that might be a nice to know:

source map for debugging

You don't need sourcemaps when using teal. Teal does a great job at keeping the line numbers at the same place (and maybe more?) So, if lua complains about an error being at line X you can just open the corresponding teal file and go to the same line.

For other tools, see the "awesome-teal" repo.

Also, about the any vs unknown thing, in Teal any is not like Typescripts any but much more similar to its unknown. I can see reasons why you want a more flexible "no type information available" type but I'm not sure if it should be as easy as just saying it is any. Maybe teal's unknown type can be this and be restricted in how you can get it to things that are useful for lua interop? (So, requiring a .lua file without a .d.tl file available could give it and that being pretty much the only way?)

[sveyret]

something that might be a nice to know:

source map for debugging

You don't need sourcemaps when using teal. Teal does a great job at keeping the line numbers at the same place (and maybe more?)

That’s good to know, indeed. But in a long term, not enough, imho. I mean, if you have tens of lines of definitions which are erased after transpilation, shall we really keep those empty lines in the result file? For a big embedded project, caring for the room taken by files, it may be a problem.

[lenscas]

As far as I know, lua itself doesn't really have support for source maps so that would probably need to get hacked in if sourcemaps are desired. Which i fear might end up just being worse than the current system as a result.

I am happy to get proven wrong on that front though.

[lenscas]

May have found a bit of extra time, so lets see how far I get :)

Nil checking

Yep, almost everyone wants this but a lot of the time people can't agree on how it would actually look/work. The problem lays within the fact that nil is used a lot more in Lua than null in JS as while JS can throw exceptions, Lua often may just return nil and maybe some second value. In addition, the general consensous seems to be that people want to avoid TS's problem of indexing returning a T rather than T?, but then how to make indexing not a total pain?

Types and objects

Yes, something closer to interfaces would be nice. I would even go as far as being able to make them implemented implicitly but that is opening the "structural typing" can of worms which is probably best kept closed, at least for now.

Some strict examples on when one versus the other should be used would need to be made though. ESPECIALLY when it comes to writing .d.tl files for lua or even worse, C/Rust libraries.

Inheritage

I am not quite sure why it needs code in the tlconfig.lua file. Do you want to be able to extend types that aren't your own? (It almost reads like that but not quite).

Unions and type guards

Yes, removing the error on table | otherTable and similar and instead make is smart enough to not be able to disquintish between those 2 would already come a long way as from there people are able to work around the lack of type guards.

Of course, type guards and similar are also welcome if you ask me. But one step at a time, shall we? :)

Enumerations

I am not sure about introducing the type keyword. Perhaps it can be merged with shape in some way as the two sound very similar? (Both not emitting code at all). Might also just be that the current use of record and enum should be revised. I do like shortcuts when writing more complex types. Right now it isn't needed much because teal doesn't allow for much but sooner or later that would need to change.

Requires and unknown

Already went over this in my previous message
TL;DR: Teal's any is like TS's unknown. A type that acts more like TS's any might be nice, but should be hard to get and ideally only appears when talking about doing FFI with lua.

Metamethods

Giving them their own section in records does make sense if you ask me. This might then also lead to a reasonably way to write the kind of inheritance that works this way, especially if the metamethod table can be its own proper type that you can reuse.

Tools

look at https://github.com/teal-language/awesome-teal

Note: This is just my look at things as someone who maintains a Rust <-> Teal FFI library and as I am not a maintainer and as it has been a while since I used teal (or lua for that matter) take my take on things with a decent grain of salt ... :)

[sveyret]

Inheritage

I am not quite sure why it needs code in the tlconfig.lua file. Do you want to be able to extend types that aren't your own? (It almost reads like that but not quite).

Not really code in tlconfig.lua, but more a description of what to do when creating a type. A little bit like what you can do for React in Typescript with tsconfig option jsxFactory. I think this would be important in order for Teal to be used in already existing code base.

Unions and type guards

Yes, removing the error on table | otherTable and similar and instead make is smart enough to not be able to disquintish between those 2 would already come a long way as from there people are able to work around the lack of type guards.

Of course, type guards and similar are also welcome if you ask me. But one step at a time, shall we? :)

Sure! I just meant that we should keep the keyword is for type guards, and auto-detect type(xxx) == "string".

[lenscas]

Not really code in tlconfig.lua, but more a description of what to do when creating a type. A little bit like what you can do for React in Typescript with tsconfig option jsxFactory. I think this would be important in order for Teal to be used in already existing code base.

I'm still not quite getting why that code would need to live in the tlconfig file I fear.

[euclidianAce]

Nil checking, inheritance, and interfaces (and probably intersection types) I think are the big elephants in the room for Teal right now. The main issue I would say is implementation. Not to be rude, but everybody and their mother has an opinion on how these should be done, but very few people are actually able/willing to put the time in to implement them (which is totally fair).

See the following discussions and issues for more information:

• Nil: Much ado about nil #339
• Optional Arguments: Optional / required arguments for functions #366
• Classes and Inheritance:
  • Standardized class model? #97
  • Functions that create and return new classes/records (OOP wrappers) #200
  • Tagged records #124

Unions and Type Guards

Teal has a limited form of this via the is operator:

local var: string | number = getSomeValue()
if var is string then
   local str = "+" .. var
else
   local num = 1 + var
end

but due to the limitations of discriminating table types at runtime, there has been discussion of user defined is operators.

Types and objects

In Teal, records are nominally typed, and I do think there is a strong argument for having a way to structurally type things as a lot of Lua code only cares about the structure of its tables rather than the names.

I think most of this comes down to the fact that this is an open source project that people are graciously donating their free time and effort to. I for example have gotten extremely busy over the past year or so and haven't been able to make any meaningful strides on any of this.

[sveyret]

Well, as everyone, I am busy, but I am willing to put time for Teal, and hope to convince my firm to also give me time for that. But then, I need Teal to really be helpful in our work, so my comments… 😉

[13k]

Not to be rude, but everybody and their mother has an opinion on how these should be done, but very few people are actually able/willing to put the time in to implement them (which is totally fair).

disclaimer: I re-read before posting and I sound too confrontational and/or agressive, that's not my intention

To be brutally honest, tl's project as it is now is extremely difficult to contribute to, in terms of actually writing code.

The main tl.tl file having around 10 kLOC is a big issue (that was already raised before: #549). It raises a list of concerns, ranging from the simple stuff - like editor performance, navigating it or even tooling around it, like language server performance, - to more subjective/harder issues like cognitive load, difficulty to actually read it (because it's so big, it encourages jam-packing everything in fewer lines, which hinders readability) and "fear of breaking everything".

In recent weeks, I've been creating declaration files for existing Lua libraries (mainly neovim plugins) and hitting a few bugs along the way and I'd love to create PRs to fix these instead of simply dumping issues on contributors. But for the reasons I mentioned, I've been largely discouraged to do so.

I understand there are technical issues keeping tl.tl as a single gigantic file, but to be honest again, I believe we, as developers, should be over the idea of distributing software as a single file that you can copy and paste into other projects.

Even if that's a desirable goal, there is free and professional tooling and infrastructure (like GitHub actions and release artifacts) that can be used to compile, bundle and distribute complex software in a format that could be downloaded as a single file to be used in other projects.

[lenscas]

but to be honest again, I believe we, as developers, should be over the idea of distributing software as a single file that you can copy and paste into other projects.

Keep in mind that lua is pretty often embedded in other applications and those may have restrictions on what can be loaded. So, for those purposes a single file can make things considerably easier.

there is free and professional tooling and infrastructure (like GitHub actions and release artifacts) that can be used to compile, bundle and distribute complex software in a format that could be downloaded as a single file to be used in other projects.

There are indeed ways to bundle multiple lua files so I do agree that it should be possible to bundle everything together after the teal compiler is build.

[13k]

Dear fellow internet stranger (and I mean no offense by that),

I got my reasoning wrong. Narrow-mindedly focused on the wrong subject and I feel I must apologize and make a better attempt at explaining my concerns. (Okay, it's mostly let-me-make-myself-clear saving face and one can never, ever, under no circumstances at all, be caught being wrong on the internet.)

It's a pet peeve of mine that I keep seeing two opposites on development tooling: either overengineering the living shit out of it like the Javascript ecosystem; or no tooling at all, like the Lua ecosystem. Praised be our champions carrying out the ungodly task of targeting heterogeneous runtime environments.

So a little too much emotional investment later and I, steam coming out of my ears no less!, wrongfully attacked the thing that what I perceived to be my source of frustration: single-file distribution format, which, by all means, makes no sense, as it's a perfectly valid use case. The actual issue being single-file development, which doesn't need to be the case for the aforementioned reasons. Reasons that you gracefully and professionally agreed on, despite having to circumvent my horrid display, for which I am most grateful.

In my minor personal frustration-induced torrential and chaotic stream of consciousness, I forgot to present what I find to be compeling arguments that would decrease the value of the specific use case that seems to be hindering the evolution of this prestigious project, even though all are moot points since we already agreed on a solution, but please let me indulge myself, so here they are in all their very questionable glory:

Argument One: Teal being a type-checking pre-processor that generates a final viable product and pre-processors being... Okay, does it make sense to run Teal itself inside an embedded environment? That's right, it does not! Case = closed.

Argument Number Two: Do we want to cater to nonsense scenarios? Like brushing my teeth before I eat so I don't infect the food? (ha ha who does that?) That's fine. What about considering it an optional feature - and a bad, ugly, nasty feature that requires a bad, ugly, nasty strict limitation (did I mention a SINGLE FILE?!) - and optional features being, you know, optional, they shouldn't extend their little toxic, vile, wet tendrils over the whole body of the project, constricting it, suffocating it, making it cry to sleep every night.

Argument Numero What's Uno + Dos: did I mention a SINGLE FILE? Come on. Is. This. For. Real. Now. People. Please.

Yes, I'm fine, thanks for the concern.

Argument 4, bringing in the rear of the Four Horsemen of Socrates' Maieutic Method: The world is a big, wonderful, place, you guys. Let's make it better, together! The sky is the limit. Believe in yourselves! With love, everything is possible. (I remembered the course I took from an online debating coach and they said that for it be more effective, my argument should end on a positive note. To think it cost only $99.99/month. You should see me 2 years ago when I started it, I was a mess of a dumpster fire. Good thing I kept going, still subscribed and practicing it here today. Humble brag, I know, but I can't avoid it!)

Thank you for your time and I hope this letter finds you well.

[13k]

I tried a serious post before that one, but I realized I was only digging my hole deeper. So I search-engine'd "how to get out of a hole" and they said "go over the top", so I went for that. I don't know what I'm doing.