Rest keyword argument

[sorawee]

This discussion is a reiteration of #66 (comment)

Currently:

• Rhombus supports rest argument via the ... syntax (I don't think this is documented yet?)
• Just like Racket, it does not support rest keyword argument. However, obtaining all keyword arguments can be done by using make-keyword-procedure from Racket.
• It supports unpacking argument objects to arguments via apply and keyword-apply, both from Racket. These "argument objects" must appear in a restricted position. E.g., in apply, the argument list can only appear at the end.

#lang rhombus

import:
  racket/base

fun f_internal(kws, kwargs, xs, ...):
  [kws, kwargs, xs]

def f: base.#{make-keyword-procedure}(f_internal)

f(~x: 1, ~y: 2) //=> [[keyword(~x), keyword(~y)], [1, 2], []]

base.apply(f, [1, 2, 3]) //=> [[], [], [1, 2, 3]]

The proposal is that keyword argument should be as important/easily used as regular argument. More concretely:

1. Since Rhombus has a syntax support for rest argument, it should have a syntax support for rest keyword argument too. This likely suggests that the current ... notation might need to be modified to accommodate this change.

   Here's one possibility:

   fun my_fun(x, ~foo: foo = 1 ::: ~args: rest_args, ~kwargs: rest_kwargs):
     print([x, foo, rest_args, rest_kwargs])
   
   my_fun(10, 11, ~bar: 2) //=> [10, 1, [11], <something like ~bar: 2>]
   

   Here, the rest argument and rest keyword arguments are both independently optional. And when they are both missing, the rest section delimiter ::: is optional as well.

   Note that rest keyword argument must take existing keyword arguments into account. I.e., it extracts the rest, just like how the rest argument extracts the rest.

2. (1) also suggests that there should be a single object representing a map from keyword to the corresponding value. In Racket, kws and kwargs list are separated in make-keyword-procedure, which is unwieldy to work with.

   Note that Racket users have already done this, too. See https://docs.racket-lang.org/arguments/ by @jackfirth and https://docs.racket-lang.org/kw-utils/kw-hash_scrbl.html by @AlexKnauth. This suggests that this feature is highly desirable.

3. apply and keyword-apply should be unified OR even better, entirely eliminated. Function application (with special syntax support) should suffice. For example, we can follow the syntax used in the definition site:

   f(1, 2, ~a: 2 ::: ~args: [10, 11, 12], ~kwargs: <something like ~bar: 2>)
   

   Note that this particular syntax constrains us to have at most one argument object at the end.

Alternatives:

1. Python uses * and ** operators for converting argument objects to arguments, and allow them to be used multiple times. Thus, it doesn't have the "at most one argument object at the end" constraint.

   f(*[1, 2, 3], 4, *[5, 6], **{'a': 1}, b=2, **{'c': 3})
   # equivalent to f(1, 2, 3, 4, 5, 6, a=1, b=2, c=3)
   

2. Alternatively, we can make the argument object contain both regular arguments and keyword arguments, just like https://docs.racket-lang.org/arguments/. Then, the current ... in Rhombus could work out of the box. This would also be more ergonomic for a simple argument object passing. However, it may be more difficult to manipulate in more complex situation, and potentially less efficient.

   fun f(~x: x, args1, ...):
     fun (~y: y, args2, ...):
        g(args1, ..., args2, ...) // provided that we allow multiple ... at callsite
   
   f(~x: 1, 2, ~a: 3)(~y: 4, 5, ~b: 6)
   // calls g(2, ~a: 3, 5, ~b: 6)
   

Concerns

To recap, there are three concerns.

1. What should the argument object representation be? Should it combine arguments with keyword arguments? Should they be separated?
2. What should the syntax for definition site and call site be?
3. Should it be possible to have multiple argument objects at call site?

[Lazerbeak12345]

Here, the rest argument and rest keyword arguments are both independently optional. And when they are both missing, the rest section delimiter ::: is optional as well.

How would an example in this system have only one? How would the parser be able to distinguish between only an args and only a kwargs?

I notice in python, it has a specific marking for each.

[sorawee]

The way I wrote above is:

fun ...(... ::: ~args: rest_args, ~kwargs: rest_kwargs):
  ...

So if you only want rest arguments, you would write:

fun ...(... ::: ~args: rest_args):
  ...

Or if you only want rest keyword arguments, you would write:

fun ...(... ::: ~kwargs: rest_kwargs):
  ...

Or if you want neither, then everything after ::: can be removed.

[Lazerbeak12345]

Cool. So in this suggestion, the name args or kwargs matters.

[AlexKnauth]

Idea: ~...

With ~... put after a rest-keywords argument similar to a ... after a rest-positional argument:

fun my_fun(x, ~foo: foo = 1, rest_args, ..., rest_kwargs, ~...):
  print([x, foo, rest_args, rest_kwargs])

my_fun(10, 11, ~bar: 2) //=> [10, 1, [11], {keyword(~bar): 2}]

Or it might look better without the comma in between:

fun my_fun(x, ~foo: foo = 1, rest_args ..., rest_kwargs ~...):
  print([x, foo, rest_args, rest_kwargs])

my_fun(10, 11, ~bar: 2) //=> [10, 1, [11], {keyword(~bar): 2}]

[AlexKnauth]

Another idea inspired by Clojure: & and ~&

fun my_fun(x, ~foo: foo = 1, & rest_args, ~& rest_kwargs):
  print([x, foo, rest_args, rest_kwargs])

my_fun(10, 11, ~bar: 2) //=> [10, 1, [11], {keyword(~bar): 2}]

[benknoble]

fun my_fun(x, ~foo: foo = 1 ::: ~args: rest_args, ~kwargs: rest_kwargs):
  print([x, foo, rest_args, rest_kwargs])

my_fun(10, 11, ~bar: 2) //=> [10, 1, [11], <something like ~bar: 2>]

This syntax ~args: rest_args seems a little backwards to me, given that it binds rest_args. Doesn't the rest of Rhombus use a style more like id : type or id :: type (interpreting type loosely, here)?

I can see one argument that ~args and ~kwargs are actually special keywords, but having the bound identifier come after the colon is still odd to me. I would prefer the ~... notation or something more like rest_args : Args, rest_kwargs : KwArgs (or whatever the current naming convention/syntax is for the type-like things).

[sorawee]

Summary from the meeting (feel free to add anything that I missed or got wrong)

General

• We generally agree that rest keyword argument should be supported.
• Rest keyword argument and rest argument should be separated, because many functions may want to only accept rest arguments but not rest keyword arguments. Combining rest keyword argument and rest argument into one object would mandate that we accept them all.
• Rest arguments would be a list of values. Rest keyword arguments would be a dictionary mapping keyword to values.
  • There's a discussion whether it should be a keyword or symbol. We settled on keyword, as it's good for interoperability with Racket (see below)

Interoperability & efficiency

• We want keyword arguments in Racket to be keyword arguments in Rhombus for interoperability.
• Racket has a protocol that keyword arguments must be sorted (for performance reason). However, regular dictionary doesn't have ordering. So there an efficiency concern regarding pack/unpack/hash->sorted list/sorted list->hash chain that the keyword argument object needs to go through.
  • We may need to create a new data structure for dictionary that preserves sortedness when arguments to the constructor are already sorted.

Positioning

• At the call site, allowing unpacking at non-last position (or allowing unpacking multiple times) could be useful, but for now, we agree to restrict it to the last position for simplicity.
• If I understand correctly, at the definition site, we decide to restrict with rest (keyword) argument at the last positions for now (Matthew mentioned that supporting them before other arguments would require some considerable adjustment). It's unclear how rest argument and rest keyword argument should be ordered.

On syntax

• We want a syntax at the call site and definition site to mirror each other.
• The syntax should accommodate contract / types.
• ... as a suffix is more attractive than * / & because we already use ... everywhere else.
• At the call site, it should not require rest keyword argument object to be bound to an identifier first.
• We spent time experimenting with various possibilities. ~... by @AlexKnauth appears to be the most attractive one so far.
  • However, it requires that ~... is recognized as an operator, as Rhombus currently doesn't allow ~.... A more general rule is that ~<operator> is an operator.
• Matthew advocates for , between rest argument identifier and .... Several people, including me, do not like it :)
  • The position of contract / type probably depends on whether there is a comma. If there is a comma, the contract / type would be before the comma.

Experiments

fun test(~rest: rest_kwargs ...):
  1

test(~rest: {~a: 1, ~b: 2})

~rest is special here.

---

fun test(~rest ...):
  1

test(~{~a: 1, ~b: 2} ...) // doesn't work well

---

fun test({<something>} ...): // could be confusing with set/dictionary destructuring
  1

Other topics

• Currently, Rhombus already has a keyword pun at the definition site where fun foo(~abc) is equivalent to fun foo(~abc: abc). Sam suggests that the pun could work at the call site too. E.g., test(~abc) is equivalent to test(~abc: abc). Matthew doesn't like it, because it appears to use keyword as expression.

[mflatt]

This is a good summary — thanks, @sorawee!

A couple of clarifications from my perspective:

• Nothing was decided, as such, and discussion should continue here. But as a grpou at the meeting, we seemed to be leaning toward certain directions.

• The possibility of allowing only by-position arguments doesn't inherently imply separate values for by-position arguments and by-keyword arguments, but it does seem more natural with the syntaxes we explored.

[countvajhula]

This was a great discussion - thank you @sorawee for leading it!

Rest keyword argument and rest argument should be separated, because many functions may want to only accept rest arguments but not rest keyword arguments. Combining rest keyword argument and rest argument into one object would mandate that we accept them all.

Did we talk about the reasons for this constraint?

We should probably explore whether single-structure could be more desirable than two-structures from a design perspective (or vice versa). If we think that might be the case, then we could look at the why-nots in more detail and see if they can be resolved. I can think of a few ways in which single-structure could be compelling, including that it avoids the whole issue of conflating the meaning of ~ and operators etc., since we could just do (a b args ...) and that would be consistent with the meaning of ... We could also implement sequence and dictionary generic interfaces on the arguments structure, and that could provide the foundation for designing an intuitive interface. I think it would be ideal if we can develop these competing options to the best versions of themselves before subjecting them to full critical evaluation (which is the approach @mflatt has been encouraging as well).

[sorawee]

The constraint arises from functions like string-append, which would not want to accept any keyword arguments, but it wants to accept variadic arguments.

One possibility, if we decide to have only one argument object which contain both keyword and positional rest arguments, is:

fun string_append(args ...):
  args.restrict_positional()
  <...>
end

I personally dislike this. But there's another concern: efficiency.

I think it would be helpful to be able to statically indicate to Rhombus that we don't want rest keyword arguments. That way, it could just compile the program to a simple (define (f . xs) ...). Otherwise, the system needs to prepare for a possibility that rest keyword arguments may be used. That requires, as I understand, compiling it to a more expensive function definition.

[sorawee]

An alternative to args.restrict_positional() could be using contract:

fun string_append(args ... :: PositionalOnly): // or args :: PositionalOnly ..., or ... args :: Positional, etc.
  args
end

[mflatt]

One of the problems we're running into with ... is whether a bound name refers to an element of the ... sequence or the sequence as a whole.

Looking at syntax patterns for a moment, the current design is that the pattern

 `($f($x), ...)`

binds f to a list of terms and x to a list things inside the parentheses. To reflect that list, I tend to add s to the end of the variable names,

 `($fs($xs), ...)`

but this naming convention grates in a different way, because I'm trying to write a pattern that matches a sequence of individual $f($x) forms.

Scheme macros-by-example resolev this tension by having f and x stand for individual elements, but you can't refer to just one of the elements; you can only refer to them as a group by using ... in a template. That solution is closely tied to the idea that f and x in

 `#'((f x) ...)`

are not variables, but are instead pattern variables. I think we don't want to go back to a "variable" versus "pattern variable" distinction, but we could have a "sequence variable" concept that applies generally.

That is,

 `($f($x), ...)`

would bind f and x as a sequence that you can only access by using the ... operator:

 f ... // => a sequence of syntax objects for the `f` matches

In a template,

 `($f($x), ...)`

would similarly mean that a use of the ... operator is implicitly added after each escaped variable. (There are some issues here; see last paragraph.)

This view would help resolve the question of what kind of contract goes on the argument, because the name x refers to an individual element, and it does not refer to the list of arguments as a whole:

 fun test(x :: Integer, ...):
   x ... // returns a list of integers

 // Not valid:
 fun test(x :: Integer, ...):
   x // syntax error due to missing `...`

Also, note the difference between passing multiple arguments

 fun test(x :: Integer, ...):
   g(x, ...) // propagates multiple arguments to g

and passing one argument that's a list

 fun test(x :: Integer, ...):
   g(x ...) // passes one list argument to g

There are some details to work out here, such as how ... interacts with list values (as opposed to variables bound as sequences), and it's not obvious to me that it all works out. But I wonder what people think of this direction.

[mflatt]

I remain interested in unifying ... for the purposes of syntax pattern/templates and function calls, and I bet this will also show up in some other context. Maybe it's connected to for.

The more I've thought about it, the more I'm happy with the idea that an identifier bound as a repetition with ... must be referenced with .... I'm also happy with the idea that ... is only recognized by forms like like function calls, list constructions, patterns, and templates. That is, if x is bound as a repetition, then [x, ...] gets the list of elements, not x ... as I originally proposed.

The place where things started to break down was trying to allow [produces_a_list(), ...] to work. That idea was based on how syntax templates currently work, but maybe syntax templates should not work that way, either. The right approach might involve being more explicit syntactically when going from a list to a repetition.

One way to be explicit would be to go through a binding like val [y, ...] = produces_a_list(). Of course, that will get verbose in some cases. If you start with a repetition x and want to add one to each element, then

val [y, ...] = [x, ...].map(fun(x): x+1)
f(y, ...)

would do the job, but it would be nicer to write something like

f(x: x+1, ...)

The idea is that the x to the left of : determines the repetition, and then the part after : is an expression that can refer to x. I'm not sure about the specific notation, but that kind of thing is what I mean about being syntactically explicit.

To start with a plain list, maybe it would be something like

f(x in produces_a_list(): x, ...)

In other words, ... would recognize <id>: <expr> and <id> in <expr>: <expr> forms.

[jackfirth]

The more I've thought about it, the more I'm happy with the idea that an identifier bound as a repetition with ... must be referenced with .... I'm also happy with the idea that ... is only recognized by forms like like function calls, list constructions, patterns, and templates. That is, if x is bound as a repetition, then [x, ...] gets the list of elements, not x ... as I originally proposed.

I like this. It's simple, unambiguous, and flexible. It also will make macros easier to learn, since beginners won't have to learn the templating system and the macro system at the same time.

The place where things started to break down was trying to allow [produces_a_list(), ...] to work. That idea was based on how syntax templates currently work, but maybe syntax templates should not work that way, either. The right approach might involve being more explicit syntactically when going from a list to a repetition.

One way to be explicit would be to go through a binding like val [y, ...] = produces_a_list(). Of course, that will get verbose in some cases. If you start with a repetition x and want to add one to each element, then

val [y, ...] = [x, ...].map(fun(x): x+1)
f(y, ...)

would do the job, but it would be nicer to write something like

f(x: x+1, ...)

The idea is that the x to the left of : determines the repetition, and then the part after : is an expression that can refer to x. I'm not sure about the specific notation, but that kind of thing is what I mean about being syntactically explicit.

To start with a plain list, maybe it would be something like

f(x in produces_a_list(): x, ...)

In other words, ... would recognize <id>: <expr> and <id> in <expr>: <expr> forms.

I'm not so sure about this. I think making f(x + 1, ...) work is too valuable a feature to give up. To handle the produces_a_list() case, what if we did something similar to Python and had a sigil for saying "please escape the templating language and unpack the result of this expression" like f(*produces_a_list(), ...)? That's basically what ,@ and #,@ do now in Racket anyway. And I fully agree that var [x, ...] = produces_a_list() is a more appropriate way to handle complex cases.

Also, as an aside, the comma between x and ... is incredibly bizarre looking. Lets please not keep that in the final syntax.

[AlexKnauth]

That's where I was thinking of using the & as the "rest" marker.

f(& produce_a_list())

would produce the same value as

val [y ...]: produce_a_list()
f(y ...)

and f(y ...) would produce the same value as f(& [y ...]), needing y as a pattern-variable with ellipsis-depth 1 in both cases.

I don't see the use in having both & (or *, whichever) and ... required in the same function call. I would rather avoid having ... sometimes take ellipsis-depth 1 and other times take ellipsis-depth 0.

With them separate this way, ... always takes ellipsis-depth 1, and & (or *, whichever) always takes ellipsis-depth 0.

if we did something similar to Python and had a sigil for saying "please escape the templating language and unpack the result of this expression" like f(*produces_a_list(), ...)?
That's basically what ,@ and #,@ do now in Racket anyway

Should this be "splicing" allowed anywhere, or only "rest" allowed at the end?

[mflatt]

Also, as an aside, the comma between x and ... is incredibly bizarre looking. Lets please not keep that in the final syntax.

Setting aside rest arguments for a moment, here's a syntax pattern that matches any number of xs in parentheses:

'(x ...)'

For example, '(x)' would match, and so would '(x x x)' or '()'. The syntax object '(x, x, x)' would not match.

Here's a syntax pattern that matches any number of comma-separated xs in parentheses:

'(x, ...)'

For example, '(x)' would match, and so would '(x, x, x)' or '()'. The syntax object '(x x x)' would not match

This is why it seems to me that a comma is needed before ... in list and function-argument repetitions. If a macro definition for a list of thunks has the form

def 'thunks($body, ...)':
  '[fun(): $body, ...]'

thunks(1+2, 1/0) // => a list of thunks

Then I'd expect a non-macro version to resemble it in the way ... is used:

def procs(thunk, ...):
  [thunk, ...]

procs(fun(): 1+2, fun(): 1/0)

[jackfirth]

@mflatt I think I'm fine with ... working slightly differently for expressions and syntax objects in that sense. Using xs ... in a value expression is unambiguous about whether commas should be inserted or not. Macros have to worry about the differences between terms connected by adjacency and disconnected terms, but I don't think functions need to worry about that difference. The distinction between xs, ... and xs ... seems like a reasonable price to pay for the increased expressivity of syntax templates.

[AlexKnauth]

Related to rest keyword arguments: rest patterns for pattern-matching on maps.

I suggest & as the "rest" marker here too, for example in the pattern { k1: v1, k2: v2, & rst } to bind rst as the map with k1 and k2 removed.

> match { "a": 1, "b": 2, "c": 3 }
  | { "a": a, & rst }: [a, rst]

[1, { "b": 2, "c": 3 }]

[sorawee]

@AlexKnauth has an implementation at #236