Language Specification: An explosion of nulls - the eleventy billion dollar mistake

[dkearns]

Can anyone explain the need for these values? This seems rather horrifying at first blush, I'm not sure why I've never noticed before. Is there really no sane way to integrate null into the type system?

null_blob
null_channel
null_class
null_dict
null_function
null_job
null_list
null_object
null_partial
null_string

[yegappan]

As explained in the help for :help null, these predefined null types have a specific type and are used to clear a variable of the corresponding type. It will be harder to use the null special value for all the null types. For example, the type of a list is derived from the type of the list items. If the generic null is used to initialize the list, then it will be harder to determine the list type.

[qeatzy]

Isn't it possible to limit null_xxx to implementation side, and only expose null to language user?
Since vim9script is a compiled language, I don't see any inherent difficulty for that.
Other languages had already faced/solved similar issues.

eg, in golang, there is one keyword null, but many type has null of specific type, eg, null channel, null function, etc (these are implementation details). And it also do not introduce null when not necessary, there is no null for string, an uninitialized string equal to empty string.

Also I don't see any merit for introduce unknown type, since type any can already holding value of any type.

var lst: list<any>
echo typename(lst)

I do understand, there is somewhere lingering legacy vimscript impact of initialize to number 0 if type is any (probably also list related) (correct me if I remember it wrong).
But a lvalue whose type is any should never be affected by the fact it's rvalue is number 0 (or any other type).
Introducing unknown type is a failed attempt which IMHO worsen the situation.

Back to null, rust chose another approach, I'm not too vested to it, but for some type, it's value cannot be null/nill, instead it use Option<YouType>. https://news.ycombinator.com/item?id=22479365 and https://doc.rust-lang.org/std/option/.

As for distinction of uninitialized value nil and empty value, for many types, you don't need both, in golang a nil slice is an empty slice, you don't need another default/empty value for that. I think this is true for vim type such as job, channel, etc.

As already mentioned in many other issues, transitivity matters, which is a touchstone for good design. I hope vim do not follow javascript's evil route of == being insufficient to compare equality.

As for list and item type, I think Java faced similar type covariance problem in it's generics. Currently I cannot recall the details in it, but it basically pin down to type covariance's choice. If Type B is subclass of type A, should a variable of List<A> accept a List<B> value as rhs for assignment.
I don't known how much can we learn from Java here (list and type covariance), since Java's type hierarchy is pretty unified (single inheritance + everything beside basic type is object), whereas vim is not.

[errael]

Can anyone explain the need for these values?

I've wondered about this myself.

For example, the type of a list is derived from the type of the list items.
If the generic null is used to initialize the list, then it will be harder to determine the list type.

I don't see that, consider

vim9script
var x: string = null

The above fails; it doesn't seem like it should; it's not ambiguous; the type of x is obviously string; it doesn't leave x without a type. And i think the following should give an error

var x = null

like E####: no-type specified or I suppose it could be typed as any and give a warning (unless the strict-type flag is set) but neither warnings nor compilation flags exists, so I'd go with the error.

There may be cases where these type specific nulls are needed, I haven't looked deeply enough, but the language could be null friendlier.

I definitely think null in the language could be improved, but all these varieties of null are part of vim9.0. so I don't think you can just get rid of them.

[lifepillar]

Related to #11770. There I suggested that, if “typed nulls” have to be kept, they should be disallowed at least in comparisons.

But, imo, the semantics of nulls should be carefully revised. The main problem I see with “typed nulls” is that they stand for a missing value and for an empty (but defined) value at the same time. That seems to me like walking on a razor's blade. Some (non-exhaustive) random weirdness:

(1)
echo null == null_string      # true
echo "" .. null == "null"     # true, special is cast to string
echo "" .. null_string == ""  # true, null_string is “the same” as an empty string

(2)
echo null == null_list        # true
echo [] + null                # E745: using List as Number
echo [] + null_list = []      # true, null_list is “the same” as an empty list

(3)
var x: string
echo x == null                # true
echo x == null_string         # true
echo x == ""                  # true

(4)
var x: string = null_string
echo x == null                # true
echo x == null_string         # true
echo x == ""                  # true

(5)
var x: string = ""
echo x == null                # false
echo x == null_string         # true
echo x == ""                  # true

(6)
var x: list<string>
echo x == null                # false
echo x == null_list           # true
echo x == []                  # true

(7)
echo extend([], null_list)    # 0 (error?), but:
echo [] == null_list          # true
echo extend({}, null_dict)    # 0 (error?), but:
echo {} == null_dict          # true

(I think those with extend() are bugs.)

[lifepillar]

And nulls are broken for complex types I think. In the following, there is no way to set x to null

vim9script
var x: list = null

AFAICT, nothing works to set x to null

You can do:

var x: list<string> = null_list
echo x == null  # true

But then, also these are true:

echo x == null_list # true
echo x == []        # true

[errael]

Right, thanks. Hallucinating...

[errael]

Related to #11770. There I suggested that, if “typed nulls” have to be kept, they should be disallowed at least in comparisons.

But, imo, the semantics of nulls should be carefully revised. The main problem I see with “typed nulls” is that they stand for a missing value and for an empty (but defined) value at the same time. That seems to me like walking on a razor's blade. Some (non-exhaustive) random weirdness:

I'm aghast (not to mention dumbfounded) and wondering how I missed this (or maybe I forgot about it). [] == null_list, wow. I guess, as you point out,

“typed nulls” have to be kept, they should be disallowed at least in comparisons

So never, ever use a "typed null" in a comparison, if you want your code to make sense.

I wonder if

var x: string = null

could be made to work, such that it sets a typed variable x to a null value, and not an empty string, and preserves its type. This must be already supported internally in vim9script since

vim9script

var a = []
echo a == null         # false
var b = null_list
echo b == null        # true

works as expected. [Corrected example 2023 Oct 31]

I think this would obviate the need for typed nulls. Then proper null semantics would be available. They could legitimately be deprecated or just left_around if you want to use null_string instead of []. As much as I hate to say it, this could be done after vim9.1 is released; but the sooner the better.

Do you (anyone) think that handling var x: string = null (with some possible "minor" adjustment to semantics of how an x that is so defined behaves) would take care of the null problem, without requiring a change in current behavior. Let the "typed nulls" die a quiet death? If this could be done for vim9.1, then maybe all the "typed nulls" could be removed in the next release or two. A deprecation-warning would be useful.

I wonder if the "typed null" is all about implementation issues, not language coherence.

semantics of nulls should be carefully revised

It would be good to do something.

[lifepillar]

I don't know anything about the implementation, unfortunately. Maybe, I'll find some time to start digging into the code some day.

Just to be clear, when I say that the semantics should be revised, I mean to go through what has been implemented, and resolve inconsistencies and reduce the number of idiosyncracies in the most logical way. I'm not proposing to scrap everything (it's not in the spirit of how Vim is developed).

For instance, some types (job, channel, function, …) have no obvious default value, so if they are declared without being initialized they are left undefined (“null”). Therefore, var j: job and var j: job = null_job mean the same thing (afaict). Besides, it is possible to switch between two states: “defined” (not null and not null_<type>) and “cleared” (both null and null_<type>):

var j: job         # undefined
echo j == null     # true
echo j == null_job # true

j = job_start()    # defined
echo j == null     # false
echo j == null_job # false

j = null_job       # cleared (undefined again)
echo j == null     # true
echo j == null_job # true

So far, so good. On the other hand, types with a “neutral” value, such as numbers, lists, dicts, … are implicitly initialized to a default value if no initializer is provided (see :help E1022). But, var l: list<bool> is not the same as var l: list<bool> = []:

var l1: list<bool>
var l2: list<bool> = []
echo l1 == []        # true
echo l1 == null_list # true
echo l1 == null      # true   <==
echo l2 == []        # true
echo l2 == null_list # true
echo l2 == null      # false  <==

There are now three distinguishable states: “uninitialized or cleared” (null and null_<type>), “defined and empty” (null and not null_<type> not null and null_<type>), “defined and not empty” (not null and not null_<type>). Given that there is empty(), why this difference?

This is the kind of things that, imo, needs some attention.

Edit: fixed typo
Edit 2: fixed definition

[yegappan]

I see the following with the above example:

var l1: list<bool>
var l2: list<bool> = []

echo l1 == []        # true
echo l1 == null_list # true
echo l1 == null      # false <==

echo l2 == []        # true
echo l2 == null_list # true
echo l2 == null      # false  <==

Both produce the same output.

[lifepillar]

You are right! I stand corrected.

Ok, the puzzling thing is always the same: null_list and [] compare equal, yet behave differently:

var l1: list<bool> = null_list
var l2: list<bool> = []
echo l1 == []        # true
echo l1 == null_list # true
echo l1 == null      # true   <==
echo l2 == []        # true
echo l2 == null_list # true
echo l2 == null      # false  <==
echo null_list == [] # true
l1->add(true)  # E1130
l2->add(true)  # OK

IIUC, things are this way because a way is needed to signal that a list is undefined/null, yet there is no value available for the purpose. Is that correct?

What if null_list were set different from an empty list (and similarly for the other types with an empty value)?

var l1: list<bool> = null_list
var l2: list<bool> = []
echo l1 == []        # false  <== HYPOTHETICAL
echo l1 == null_list # true 
echo l1 == null      # true
echo l2 == []        # true
echo l2 == null_list # false <== HYPOTHETICAL
echo l2 == null      # false
echo null_list == [] # false <== HYPOTHETICAL
l1->add(true)  # E1130
l2->add(true)  # OK

This looks more reasonable to me: (1) null and null_<type> never disagree with each other, and (2) a value is non-null (defined) iff it has been assigned a non-null value. The fact that a null_list variable actually stores [] is irrelevant, because that empty list cannot be used. That means that things like [] + null_list should be errors. null_<type> may continue to exist and be mandatory for clearing variables and for stricter type checking (e.g., x == null works for any type, but x == null_list raises an error if x is not a list). Thoughts?

[yegappan]

IIUC, things are this way because a way is needed to signal that a list is undefined/null, yet there is no value available for the purpose. Is that correct?

A variable which is not yet initialized or initialized to some value is not equal to null. A variable is equal to null only when it is explicitly set to the corresponding null_<type>. Example:

vim9script
var l1: list<number>
var l2: list<number> = []
var l3: list<number> = null_list
echo l1 == null        # returns false
echo l2 == null        # returns false
echo l3 == null        # returns true

The first two comparisons will return false and the last will return true.

A variable which is not yet initialized or initialized to an empty value is equal to the null_<type>. Example:

vim9script
var l1: list<number>
var l2: list<number> = []
var l3: list<number> = null_list
echo l1 == null_list     # returns true
echo l2 == null_list     # returns true
echo l3 == null_list     # returns true

A null_<type> is equal to an empty value.

vim9script
echo [] == null_list    # returns true
echo [] == null          # returns false

In general, you should use the specific null_<type> value for initialization and comparisons and not use the generic null value.

[Rhialto]

Equality is supposed to have the transitive property: if a == b and b == c then a == c.
And here we have that (quoting from your first code block) null == null_list:

var l3: list<number> = null_list
echo l3 == null        # returns true

So from the transitive property we can conclude that null == null_list (where a is null_list, l3 is b and null is c)

but in your last block you say that null != null_list:

echo [] == null_list    # returns true
echo [] == null          # returns false

(where a is null_list, [] is b and null is c)

Something seems badly inconsistent here.

[errael]

(I think those with extend() are bugs.)

Yeah, fixing those wouldn't seem to cross the compatibility boundary.

[errael]

Not a good idea, too many implementation issues. IGNORE THIS POST

The idea of a new null model didn't get much traction, but I'll try again (hoping I wasn't clear before). It doesn't need to be done right away; but if it could be in vim9.1, then there would be a sane alternative to the current null_type model (not sure about the implementation effort).

The idea is to deprecate the null_type model, but still insure that a variable always has a type. Some advantages are

• Common model
• Backwards compatible. (Until the null_type are removed after the deprecation period)
• No effort trying to improve/fix something that no one (few?) seems to like/understand.

An uninitialized type has the well known default value, primitives are zero (or close to it) containers are empty. For example the two declarations in the following are equivalent.

var v1: list<string>
var v2: list<string> = []

Any item can be set to null

var v3: list<string> = null

A variable always has a type

var v4 = null       # ERROR

A null variable can only be compared or assigned; any other use is an error. For example

var v3: list<string> = null
if v3 == null | endif
var biggerList = someList + v3  # ERROR
echo [ 1, v3, "foo" ]    # a null variable can be a list item
var v4: list<dict>
v4 = v3     # ERROR, type mismatch

Details have to be specified. Guessing: specialized types, like job or channel default to null; variables of a primitive type cannot be set to null; any comparison of a primitive type to null is false.

After reading this discussion, I've lost my blissful ignorance. I'm still a little confused about the current null_type model, what it provides and what advantages it has. Since there are proposals to fix it, some people do. If I've missed something important about the null_type model, this proposal may not work.

Warnings and/or lint would be useful in transitioning to a new null model.

[errael]

Looking for a fix for #13433, it looks like that problem resonates with this one. I think the proposed fix for that could be a basis for the fix as described in the previous post.

I think the null stuff may be implemented the way it is because it was a late addition to vim9, and not fully integrated.

[errael]

There look to be deeply ingrained implementation reasons why nulls are handled the way they are. In the dev forum, March of 2022, there's vim9script and null which gives insight into how the current model came about.

Without considerable work, the current model remains; not sure how best to simplify understanding of how it works.

Use null or null_<type> in comparison?
In this discussion, there's disagreement on whether null_<type> or null should be used in comparisons; and, of course, the correct answer is: it depends (the nightmare answer). Using list as an example, how to differentiate between an empty list and a non-existent list and how you want to detect that difference.

• If you want an empty list, [], to be nullish, then do
  listvar == null_list
• If you want to find out if listvar came from var listvar = null_list, and not var listvar = [] then do
  listvar is null_list

It does seem that usually you can stick with null_<type>, but saying "you should not use the generic null value" ignores some use cases. [Not to mention using both == and is ;-)]

If a variable is assigned null_<type>, then some_var == null is true. Here are some cases where null must be used in the comparison.

For example,

def F(arg: any)
    if arg == null    # cannot use null_<type>; type could be "list" or "dict" or ...
        ...
    endif
    ...
enddef

Another example,
If you have var some_list: list<any>, and some of the list items may have been assigned null_<type>, then to remove only the null items, and not the empty ones, you can do

some_list->filter((_, val) => val != null)

[yegappan]

I have created the draft PR #13492 to address the issues with null described in this issue. Please try this PR out.

[dkearns]

There look to be deeply ingrained implementation reasons why nulls are handled the way they are. In the dev forum, March of 2022, there's vim9script and null which gives insight into how the current model came about.

From the discussion, it seems like a combination of implementation convenience and a dubious assertion that the difference between null and an empty value is meaningless.

Without considerable work, the current model remains; not sure how best to simplify understanding of how it works.

I think they're best conceptualised as a tagged default value that can be compared against null. It's an unusual language that even initialises local variables to the default value so I guess this can be viewed as being in keeping with that principle.

I find this sort of line in the help a bit disconcerting as a user: "Quite often a null value is handled the same as an empty value, but not always." I thought they were some sort of implicit intersection type but there is no dedicated null type. Perhaps it should be discussed in the help more in the context of being a shade-tree null-safety model?

I'm not sure how much there is to be fixed here at the moment, obvious bugs aside, but it might be wise to keep open the possibility of adding a better null-safety implementation in the future rather than make minor improvements to this model.

And they said legacy script was confusing... 😄

[errael]

I have created the draft PR #13492 to address the issues with null described in this issue. Please try this PR out.

Tried a few things, comments with the PR.

[errael]

Hoping this discussion can restart from a clean slate...
(But if considering vim9script's null handling, the phrase opening a can of worms comes to mind).

Implementation is not considered other that to note there is a current implementation which mostly works. (list is used in examples)

Summarizing this discussion and the spec.

1. Why all the null_<type> values? [1]
2. some_var = null_list clears a variable releasing its resources.
3. var some_var = null_list means some_var is both an empty list and a null list
   depending on usage/context; only meaningful for container type category (notes below).
4. Equality is not transitive (when nulls are involved). [2]
5. There's a PR, [4]; its spec is not clear.

Problems

• A. Bloat given all the null_<type>s.
• B. Semantics of null and null_<type> are not easy to understand.
  null_<type> stands for a missing value and an empty value at the same time,
  but only for type category container; null equality. (3) and (4) above.
• C. Should you use null or null_<type> in comparison. [3]

What to do now

Let's hear from everybody about null spec issues. Especially @dkearns, @yegappan, @lifepillar, @Rhialto

And especially about the downside of "Equality is not transitive (when nulls are involved)"? Example?

---

Notes

• There are 4 basic type categories.
  • primitive: number, float, boolean
  • container: string, blob, list, dict
  • specialized: function, job, channel
  • language: class, object, typealias (this category not clearly defined)
• Each type has it's own initialization semantics; consider var some_var: <type>
  • primitive types: 0 or false; this type is never null.
  • container types: empty, not null.
  • specialized types: null.
• After var some_var = null_<type> then some_var == null is true.

[1] vim9script's null handling evolution
[2] Equality is not transitive (when nulls are involved)
[3] Use null or null_<type> in comparison?
[4] Treat null the same as using null_<type> when used in an assignment

[errael]

Although the current null semantics are messy, they work (maybe some bugs).

IMHO, don't open the can of worms, try to replace it; or as @dkearns said in summary of #13458 (comment)

I'm not sure how much there is to be fixed here at the moment, obvious bugs
aside, but it might be wise to keep open the possibility of adding a better
null-safety implementation in the future rather than make minor
improvements to this model.

Better documentation might help.

[errael]

additional :help null documentation

Thinking about adding some documentation near :help null (probably should have led with this rather than starting with the TL;DR comment a few days ago). Here's a proposal. If there's interest LMK. There are still open issues that need to be resolved before documenting.

This 5 point outline takes the first two points from the Notes in #13458 (comment), followed by a few more points

1. type categories: primitive/container/specialized
2. declared, not initialized, semantics of the type categories
3. assigning null_<type> to a variable; note container semantics
4. "Familiar null comparison" semantics using null and empty() in a compare as described by @lifepillar in [vim9] Equality is not transitive (when nulls are involved) #11770 (comment).
5. "Empty container is null comparison" semantics using null_<type>.

---

(4) and (5) above are presented as two ways the programmer can work with null comparison; they may be controversial. As outlined above, the documentation is based on the current code; and so may change depending on how #13492 is resolved.

---

Here are some results checking compare performance. Mostly as expected.
Testing variable l is null_list, or [], or ["a"]. 50 loops and 100 loops.

I was a initially surprised that empty() was so cheap, I had tested user functions
in the past and they are very slow; but since any vim9script instruction is a call
into the vim9script runtime, it makes sense that a simple builtin function is close to
the cost of any simple instruction. The lists are small so len() is not a problem
in this example. Note that comparing to [], where the empty list has to be constructed,
almost doubles the cost of the compare.

$ vmicro_sum -w 85
=== null-empty micro-prof, 5 run/params ===
 nullx50  l0x50  l1x50 nullx100 l0x100 l1x100 : argXnLoop (usec/op)
  0.069  0.065  0.067  0.064  0.065  0.065 : var x: bool = l->empty()           ###-A
  0.098  0.105  0.104  0.095  0.103  0.100 : var x: bool = l == []              ###-C
  0.066  0.069  0.065  0.062  0.063  0.064 : var x: bool = l->len() == 0        ###-E
  0.061  0.061  0.060  0.059  0.060  0.060 : var x: bool = l == empty_list      ###-G
  0.054  0.053  0.053  0.051  0.052  0.052 : var x: bool = l == null            ###-I
  0.057  0.057  0.056  0.055  0.056  0.056 : var x: bool = l isnot null_list    ###-K

[yegappan]

Definitely it is useful to update the help with the above description about the null usage. Should we wait till the changes in #13492 are merged?

[errael]

Should we wait till the changes in #13492 are merged?

I think we should wait until there's a minimal spec that is unlikely to change.

[errael]

Not sure if there's interest in more help on null behavior...
I've adjusted the proposed outline statement of (4) and (5) above as follows

4. "Familiar null" comparison semantics using null and empty() in a compare as described by @lifepillar in [vim9] Equality is not transitive (when nulls are involved) #11770 (comment).
5. "Empty container is null" comparison semantics using null_<type>.

[dkearns]

Not sure if there's interest in more help on null behavior...

There certainly is, and thanks for offering, but it would be sensible to wait until it's fully specified. The existing documentation is far from Vim's normally excellent standards and I don't think it's possible to fully understand how null is currently handled in the language without experimentation.

Edit: Hadn't noticed YL's recent post.

Let's hear from everybody about null spec issues.

I assume there's at least some tacit agreement that the existing model, and not just implementation, is poor?

I think it's currently serviceable, some null-like value can be assigned for each reference type and reliably checked for with is in comparison tests. However, the behaviour when the code is incorrect, and null checks are forgotten is broken. Some types limp on with hobbled empty-like values and undefined semantics while others behave like literal null and fail hard and fast.

At a minimum I'd like to see Yegappan's great work in #13492 use null assignment to deprecate the empty-like value semantics and actually assign null. null_<type> values are named oddly enough that their peculiar semantics are less of a problem, especially in deprecated form, than having null alias them in a context-dependent manner.

More generally, it's becoming hard to justify even this sort of hand-wavy bottom type model for a language designed with hindsight of the last thirty years. I think nullable types, as implemented in some fashion by Kotlin, C#, Dart, and Java, among others, would be a good fit and could probably be implemented incrementally. Even being able to just specify the types that are nullable would be progress.

In regards to #11770, and similar issues, my view is as at least as strong as LifePillar's, "Fundamental mathematical properties such as transitivity of equality should not be violated." I, uncomfortably, disagree with Bram's assessment there and think the current implementation is causing "obscure problems" and find it unlikely that anyone is relying on that behaviour.

[vim-ml]

On Mon, Nov 20, 2023 at 12:08 PM dkearns ***@***.***> wrote: More generally, it's becoming hard to justify even this sort of hand-wavy bottom type model for a language designed with hindsight of the last thirty years. I think nullable types, as implemented in some fashion by Kotlin, C#, Dart, and Java, among others, would be a good fit and could probably be implemented incrementally. Even being able to just specify the types that are nullable would be progress.

In regards to #11770 <#11770>, and similar

issues, my view is as at least as strong as LifePillar's, "Fundamental mathematical properties such as transitivity of equality should not be violated." I, uncomfortably, disagree with Bram's assessment there and think the current implementation is causing "obscure problems" and find it unlikely that anyone is relying on that behaviour.

I made a similar comment about Option types, but nullables (and perhaps associated occurrence typing so that types under null checks make sense) might be a way to do this without needing pattern matching and other machinery. Perhaps it would be useful to work up some "desired" example programs and their semantics, and figure out how to make the implementation match that? Rather than start from one implementation and duct-tape it?

…

— Reply to this email directly, view it on GitHub <#13458 (comment)>,. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to a topic in the Google Groups "vim_dev" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/vim_dev/JZ_BC4tYotU/unsubscribe. To unsubscribe from this group and all its topics, send an email to ***@***.*** To view this discussion on the web visit https://groups.google.com/d/msgid/vim_dev/vim/vim/repo-discussions/13458/comments/7621864%40github.com <https://groups.google.com/d/msgid/vim_dev/vim/vim/repo-discussions/13458/comments/7621864%40github.com?utm_medium=email&utm_source=footer> .

-- D. Ben Knoble

[dkearns]

Ben Knoble wrote:

I made a similar comment about Option types, but nullables (and perhaps associated occurrence typing so that types under null checks make sense) might be a way to do this without needing pattern matching and other machinery.

I thought I referenced your comment but it must have got the chop, sorry. I read somewhere that the C# team's research suggested that Option types were too complicated for the average C# developer. 🤷

[errael]

At a minimum I'd like to see Yegappan's great work in #13492 use null assignment to deprecate the empty-like value semantics and actually assign null. null_ values are named oddly enough that their peculiar semantics are less of a problem, especially in deprecated form, than having null alias them in a context-dependent manner.

I'm not sure I understand. Are you saying to recommend use null in assignments as part of deprecating null_<type>? In particular, what does "actually assign null" mean?

I think nullable types, as implemented in some fashion by Kotlin, C#, Dart, and Java, among others, would be a good fit and could probably be implemented incrementally. Even being able to just specify the types that are nullable would be progress.

What's the rationale for adding "optional" or "nullable" (hadn't seen C#'s ?, very nice); the added complexity doesn't seem to add all that much; vim9script is a small language. I don't think null is used all that much anyway. What's the problem with simply allowing some_var = null?

It looks like you're saying additional null safety could be added in the future in a backwards compatible way.

[errael]

some tacit agreement that the existing model ... is poor?

If I wasn't vociferous about it then I must have trying to stop pissing people off.

I'd recommend a null model for 9.1 that uses null and empty() as discussed in #11770; print a deprecation error when null_<type> is encountered (some way to disable msg might be nice); #13492 obviates the need of null_<type> (I think). Error if null initialized item is used as value in any context (might be hard to do as long as null_<type> is allowed given it means empty, at least a deprecation/warning for now).

Primitive and container and specialized types have slightly different initialization semantics (has nothing to do with null); not a problem (I see no reason to allow assigning null to a primitive, but who cares?).

I was considering suggesting an error if null_<type>, give up 9.0 compatibility for null handling. But I believe with #13492 there is no longer any need to use null_<type>; so deprecation msgs are sufficient (maybe limit to 1,000,000 per session).

[dkearns]

At a minimum I'd like to see Yegappan's great work in #13492 use null assignment to deprecate the empty-like value semantics and actually assign null. null_ values are named oddly enough that their peculiar semantics are less of a problem, especially in deprecated form, than having null alias them in a context-dependent manner.

I'm not sure I understand. Are you saying to recommend use null in assignments as part of deprecating null_<type>? In particular, what does "actually assign null" mean?

Yes. null is currently unassignable, with a couple of exceptions, so the semantics can be freely redefined, without concern for backward compatibility. If null assignment is going to simply replicate the mistakes of null_<type> it serves little purpose beyond aesthetics.

E.g., in the following len(x) should result in the same error as len(null).

var x: list<any> = null
echomsg len(x)    # => 0
echomsg len(null) # => Error

var y: list<any> = null_list
echomsg len(y)         # => 0
echomsg len(null_list) # => 0

I think nullable types, as implemented in some fashion by Kotlin, C#, Dart, and Java, among others, would be a good fit and could probably be implemented incrementally. Even being able to just specify the types that are nullable would be progress.

What's the rationale for adding "optional" or "nullable" (hadn't seen C#'s ?, very nice); the added complexity doesn't seem to add all that much; vim9script is a small language. I don't think null is used all that much anyway.

That's pretty much the same argument that landed the language in this mess. Only having to reason about null when it's required simplifies the language usage by some measure.

Oberon, Scheme, and dare I say it, Lua, are small languages. Vim9 script is not a small language. I wish it was, but every argument made for simplifying it has failed to gain traction.

What's the problem with simply allowing some_var = null?

The same problem it's been in every language that's ever implemented it in this manner.

It looks like you're saying additional null safety could be added in the future in a backwards compatible way.

I think it could be with a flag to vim9script that changes the default to non-nullable types.

I was considering suggesting an error if null_, give up 9.0 compatibility for null handling.

While Vim9 script hasn't seen the sort of take-up we might have hoped for, a quick search of GitHub suggests that there's enough null_<type> usage out in the wild that it has to be supported. I'm hoping that the project's view on backward compatibility hasn't changed.

[errael]

What's the rationale for adding "optional" or "nullable" (hadn't seen C#'s ?, very nice); the added complexity doesn't seem to add all that much; vim9script is a small language. I don't think null is used all that much anyway.

That's pretty much the same argument that landed the language in this mess.

I thought it was "implementation convenience" as you noted earlier.
(Found this interesting: "research suggested that Option types were too complicated")

What's the problem with simply allowing some_var = null?

The same problem it's been in every language that's ever implemented it in this manner.

Better the devil you know?

It looks like you're saying additional null safety could be added in the future in a backwards compatible way.

I think it could be with a flag to vim9script that changes the default to non-nullable types.

That doesn't seem workable, unless maybe you mean per file. Using syntax like C# uses ? would work, but that needs to be done before release. A reverse meaning could be done post release, something that means "this is not nullable", like: var foo!: list<string>; but that seems like the wrong default.

[dkearns]

What's the rationale for adding "optional" or "nullable" (hadn't seen C#'s ?, very nice); the added complexity doesn't seem to add all that much; vim9script is a small language. I don't think null is used all that much anyway.

That's pretty much the same argument that landed the language in this mess.

I thought it was "implementation convenience" as you noted earlier. (Found this interesting: "research suggested that Option types were too complicated")

I was referring more to Bram's comment "When do you need to know whether a string is really null or empty? That should be rare."

Most of the problems I see in the language, as a user, seem to result from an attempt to "dumb down" features as a reaction against its arguably excessive scope.

What's the problem with simply allowing some_var = null?

The same problem it's been in every language that's ever implemented it in this manner.

Better the devil you know?

The old devil is certainly better than the new but I prefer unicorns, or at least a donkey sporting a party hat.

I think it could be with a flag to vim9script that changes the default to non-nullable types.

That doesn't seem workable, unless maybe you mean per file. Using syntax like C# uses ? would work, but that needs to be done before release. A reverse meaning could be done post release, something that means "this is not nullable", like: var foo!: list<string>; but that seems like the wrong default.

Yes, and presumably a per package configuration would be useful. Some languages, C# included, also offer sub-file region granularity.

Unfortunately, it's not possible to easily evolve the language over time given Vim's strict backward compatibility requirements.

This discussion thread was prompted by me actually suffering at the hands of the current implementation. I noticed I'd missed a null check, because I'm a terrible programmer, and wondered why my plugin hadn't exploded. This was apparently the first time I'd had to deal with null in Vim9 script. I wrote quite a lot of it about eighteen months ago but hadn't had to look into the abyss until recently.

[Rhialto]

Personally, I think there is a big difference between "the spoon is empty" and "there is no spoon". Sure, in many cases you're only interested in "is there a spoon with something in it?" but the difference between the other two cases (empty and null) is also important. If only to ensure that you can fulfill the transitive property of equality.
Now if null_<foo> were renamed to empty_<foo>, it would clear up lots of doubts and make super clear that some object set to empty_<foo> isn't actually equal to null, but to [] or "" or whatever <foo> indicates. If it is named null_<foo> it is all too tempting to "simplify" things, but ultimately make a confusing mess.

(digression, but Option types were mentioned)
Also related is the, imho fact, that Option types aren't fundamentally better than nullable types. For both world views you have to check before using an object if it is there or not. For some reason, languages have grown more readable and succinct support for checking Option types compared to nullable types (or so it seems) but there isn't a fundamental reason why it must be so.

[Shane-XB-Qian]

#13492 (comment)

[errael]

The tricky part of providing new and familiar null semantics is that
some variables must follow the old semantics for backwards compatibility.

It's possible to determine old or new semantics according to usage,
but that's confusing. In particular because you don't necessarily know
which style, old or new, when the variable is defined.

There was mention of C#'s nullable. C# provides simple and intuitive
syntax for specifying that a variable is nullable.
Here's an example of how that might work.
Note that in this example, ? means use the new/familiar null semantics.

var a_nullable_variable?: list<string>      # Notice the "?" after the name
a_nullable_variable = null          # OK
a_nullable_variable = null_list     # ERROR

var a_variable: list<string>
a_variable = null                   # ERROR
a_variable = null_list              # OK

Another possibility: instead of ? use something like
nvar to declare a variable that uses the new semantics.