Does Rhombus risk missing the trees for enforestation?

[cwebber]

I have a lot of concerns about the direction that Rhombus is heading. In general, it's possible for me to make good sense of something lisp-or-scheme-like, and Racket traditionally may have deviated from Scheme, but it was pretty easy to make sense of. A few years ago at RacketCon I wrote "Racket is an acceptable Python" and I felt pretty good about it, because I was feeling pretty good about Racket: Racket was the lisp you could get things done in. This was its biggest feature. It felt like we were on the verge of something big. That evening @mflatt told me about Rhombus, it was before the big announcement. Apparently aside from Matthias, I was the first one to express hesitation about this direction. Well, I'd like to expand on why.

I think Rhombus could be a great project. I don't consider lisps to be essentially parenthetical; the goal to provide a surface syntax that is non-parenthetical is a good one and probably seems like it should be Rhombus's main goal. (Secondary goals to figure out other useful additions to the language are good; I'd encourage asking how many of them could be libraries though.) I know @AlexKnauth recently presented on Wraith which I had some help with. Wraith seems like a good direction to me: it ultimately expands to something that's still syntactically delimited as trees, and is understandable as trees. The underlying language too is fairly understandable as trees. This is important for reasons I will expand on later.

I have a joke, I've been meaning to make it a blogpost: "LISP Isn't Syntactically Parenthetical". Wisp and Wraith are both good examples of this by using whitespace approaches, but Fructure is maybe the most interesting example of how far this can be taken: no parentheses, but even rethinking how we might edit code. Code shapes!

But Honu-style syntax... well, it's very hard for me to keep up with this project, I've read the Honu paper, and I'm very against that direction. My joking explanation is that I'm afraid we're "missing the trees for enforestation". I think some of my understanding of this congealed when reading an old writeup by Mark Miller: The Power of Irrelevance. Mark explains how in a security review, the E programming language held up well:

In thinking about lessons from the security review you and David put us through ;), I came to realize that much of its success was due to E's support for a similar quick reject property. Given a question and a large body of largely unfamiliar code, the first job of a programming language notation is to help human readers (both with and without IDE help) do a quick reject: to disqualify most of the program as being irrelevant to the current question without needing to look at most of it. (See also Visible Workings.)

Mark points out that Scheme-without-macros is fairly ideal in this regard; scheme-with-macros, well, it can be hard to know where the macros might rearrange or interject things, but still, the tree structure gives some sense of scope and hygiene helps. I have separate thoughts on how to improve this (not important here, but I think Mark's idea of "doubly hygienic macros" are possible), but it's in the polar opposite direction that Rhombus is going, I believe. When I look at Honu's enforestation, I can't make clear sense of how things will expand just by eyeballing it. It's lost the trees for enforestation, and the power of irrelevance too.

I think Rhombus has two big and admirable goals: to expand the frontier of language design, and to increase adoption. Regarding the former, it's a good goal, but the core language should remain simple enough that it can mostly be implemented in a metacircular manner. Otherwise else I think we've departed rapidly from the space that made Racket fairly ideal for this very purpose into something much more tuned to a specific set of language design choices for one specific language representation. Keeping that tree-like is important to this goal: any surface syntax should expand into something. I support things like dot-notation: foo.bar(baz) could be represented as ((%dot foo bar) baz) while not departing too far from the land of core understandability and keeping a tree-like property. I worry about some of the other proposals I see though. Wherever something could be a library, it should be a library. Avoid The Tragedy of the Common Lisp (which ironically, by still allowing a macro foundation, common lisp is less bad at than most languages, but indeed we don't want a core that's as big as common lisp's).

Regarding the latter goal of increasing adoption, the night that @mflatt told me about Rhombus, I expressed concern and skepticism, but also that I didn't want to provide stop energy against useful development. That's still true. But as I said then, it could be that Racket already has one of the core beautiful advantages: the ability to take the world of lisp and actually make it useful for people to get things done. As I said to Matthew then, "I've seen a lot of communities try to make a big energetic push to a new architectural idea, and all its energy gets consumed in that process, and the community ultimately dies." Dare I say, Perl is an example of this. It's amazing that Python wasn't, and it was due to enormous effort, and even then the changes from Python 2 to Python 3 were puny. It's unlikely that Racket is going to win by trying to enter the same space everyone else is already winning. What about running with the strengths of what we have and innovating where everyone isn't? Lisp, as a conceptual idea, is not essentially parenthetical, as Wisp, Wraith, and Fructure indicate. What if we ran with that?

I am not certain where Racket is going to go. I am hedging my bets: I have begun porting some of Spritely's components to Guile, which I wanted to do already, but my assumption was that Racket would remain the primary implementation. I'm not so sure anymore, but it really depends on where Rhombus goes. The Rhombus page gives an expansion pass in terms of "here's all the things we could do!" What's needed, for me, is now a compression pass: "here's the boundaries we will keep ourselves in!"

So let's say, here's the "Christine's wishlist of boundaries/guidelines":

• Trees, as a foundation, are important. Parentheses aren't.
• Lexical scoping is what makes scheme awesome, aim for increasing scope clarity, but certainly don't make it worse.
• Designing cool language features is nice, but them being able to be emergent and composable from a few small primitives is dramatically preferable from increasing the foundational surface.
• Keep a simple and fundamental core which can be simply expanded.
• Libraries over #langs, and separate read approaches from eval approaches, keeping trees as the foundation.
• Functions over macros, but macros over new primitives.
• Some new primitives, such as "dot notation", can be included after careful deliberation, but we should make sure they retain a sensible tree-oriented design structure. foo.bar(baz) as ((%dot foo bar) baz), for instance.
• Stay expression-oriented.
• I'm also okay with infix notation, but we shouldn't have operator precedence: permit infix notation like {3 + {4 * 5} + x}, but not 3 + 4 * 5 - x.
• Metacircular Racket should remain possible, easy, and beautiful (or at least, most of it should be)

So what will Rhombus be? And, invoking the power of irrelevance: what won't Rhombus be?

[cwebber]

Btw, to really see Wisp at its nicest, I recommend looking at the Guix Workflow Language tutorial. It's pretty amazing to see!

[ArneBab]

Yes, it is! And it became reality without me really knowing — I saw a note by Ricardo Wurmus (»Wisp allows people to see code how Lispers perceive it. Its structure becomes apparent.«), but did not realize that he was already providing it for scientists to define reproducible research pipelines in production! I was thrilled to find out the sheer awesomeness of what they build with it.

[mflatt]

The name of this repo recently changed from "rhombus-brainstorming" to "rhombus-prototype". The name change reflects the fact that we're in a phase where you can install a prototype now using a development/snapshot version of Racket, and then you get a concrete example of what Rhombus might be. Use raco pkg install rhombus-prototype in a development/snapshot version of Racket.

Installing gives you documentation on shrubbery notation (the reader level) and the Rhombus prototype (a language built on shrubbery notation), and the Northwestern snapshot's pkg build puts the rendered form here:

https://plt.cs.northwestern.edu/pkg-build/doc/shrubbery@rhombus-prototype/index.html
https://plt.cs.northwestern.edu/pkg-build/doc/rhombus@rhombus-prototype/index.html

Those documents include updated version of #122 and #163, written in (and rendered by) Rhombus-flavored Scribble.

The design rationale for shrubbery notation directly addresses some points in your message. In particular, shrubbery-level grouping limits the reach of a macro transformer, and so you get much the same kind of local reasoning in the Rhombus prototype that you get from Racket. The prototype really does run on Racket, so nothing has changed compared to Racket about the implementation model.

I understand that no amount of prototyping or rationale will convince someone who believes that S-expressions are the one true way. We all agree that it's not about the parentheses, but some of us take the failure of S-expressions to catch on as an indication that something really is wrong with S-expressions — and if it's not about the parentheses, then just removing them or using a different symbol hasn't solved the problem. Reasonable people will disagree.

[cwebber]

In case anyone doesn't scroll further down, it looks like this will be a meeting topic on January 6th. (I appreciate the Rhombus team taking time to listen to my concerns!)

[samth]

Apart from the details of Wraith vs Shrubbery vs something else, I think the point about adoption is not quite right (this is also something that took me a while to understand from Matthew).

The goal for Rhombus is to make something better than we had before, based on what we've learned from 25+ years of Racket. Part of that is the syntax. But the lesson about syntax is not "we have to change the syntax so that people like it", but "what is the unpopularity of s-expressions telling us".

Thus, the current Rhombus design aims to say "how can we adopt good ideas from other syntaxes", rather than trying to pick a new syntax based on its imagined popularity. Personally, I think it does a good job, and doing significantly better would require giving up some of the flexibility that we expect in Racket and in Rhombus. I've written a few non-trivial Rhombus programs, and the syntax works well for me.

[cwebber]

This is the first time I've heard that justification; every time previously has been about adoption. Nonetheless I agree there is space for advancements (though I think they are still achievable via the Wraith approach, or something like it): the main one I'd say is that Python for instance tends to look quite good when looked at on a screen in that it looks like readable pseudocode, and aiming for that kind of thing is a great idea. I'm not sure what the other things to aim for are though... what I feel like is missing is: "what are the structural advantages we wish we had every day that we'd have if we switched our abstract syntax structure"?

[ArneBab]

"what is the unpopularity of s-expressions telling us"

Are s-expressions unpopular? What’s your source? Are there sources supporting this claim that separate between s-expressions and "lots of irritating parentheses"?

The following is current code with s-expressions but fewer parens (taken from my advent of code solutions, day 7). I have yet to see anyone complain about it being s-expressions.

import : only (ice-9 rdelim) read-line
         only (srfi :26) cut
         only (srfi :1) list-index list-ref

define crabs
  let : : port : open-input-file "advent-of-wisp-code-2021-d7p1-example-input.dat"
    define line : read-line port
    close port
    map string->number : string-split line #\,

define min-position : apply min crabs
define max-position : apply max crabs

define possible-positions
  iota (+ 1 {max-position - min-position}) min-position

define : fuel-cost target-position crabs
  define : fuel-cost crab 
    abs {crab - target-position}
  apply + : map fuel-cost crabs

define costs : map (cut fuel-cost <> crabs) possible-positions
define min-cost : apply min costs
define ideal-position
  list-ref possible-positions
    list-index (cut equal? min-cost <>) costs

display min-cost

[rocketnia]

This indentation-based s-expression syntax is also very similar to Tainted Oyster, Wart, Nulan, Sweet-expressions, Animvs, and Cirru/Calcit.

I've never given these a proper try because I do just fine with parentheses, but it's exciting to watch them grow. :)

I've sometimes wondered if people who hadn't gotten the hang of parentheses first would be able to make their way around these syntaxes well and learn to write macros and such. I wonder that about Shrubbery notation a little bit, too.

[ArneBab]

Christine Lemmer-Webber ***@***.***> writes:

This bothered me. I'm sure you didn't intend it, but that's a huge portion of the Racket community, and it felt weirdly dismissive of that group. I'm sure that's not intentional but I think it's worth recognizing that this is a fairly sensitive topic.

On this I want to share that I was surprised at how little direct opposition I got when presenting Wisp. I expected strong opposition, because I knew that I was forgoing the parens and other projects had gotten strong pushback, but the worst I got was along the lines of “I do not see the improvement, but if it helps people, feel free to try”. I think one reason for that is that Wisp was from the start defined for interoperability in a Scheme-world. It is always easy to back out if you realize that using wisp was an error (just call wisp2lisp file.wisp > file.scm) and you can use any Scheme-code because inside parentheses Wisp-parsing is disabled: Once you open a paren, you are out of Wisp and inside regular Scheme. This was criticized as making Wisp itself harder to understand, and that’s true (which is why Wraith did not follow that path) but it also made Wisp much more powerful for an environment where Scheme is already strong. You can simply embed Scheme-code inside Wisp, and you can turn any Wisp-code into Scheme. In Racket part of this embedding is provided by #lang, but not in the same detail: You can copy-paste arbitrary tree-parts of Scheme-code and implant them in Wisp. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de

[mflatt]

The documented part is in the "enforestation" document, particularly the API section for the Racket side. On the Rhombus side, the relevant compile-time function is the parsed constructor from rhombus/macro; unfortunately, the Rhombus prototype documentation is not fully filled out in that direction, yet. (After getting the documentation to a point, I shifted gears for a while to the problem of error messages.)

For the rest of the message: We have really tried to create an open an cooperative project here. (In the specific case of Wraith, I personally invited one of the Wraith authors to create a proposal here, and it didn't happen. I am aware of Wraith, but I have not seen a presentation at any Rhombus meeting.)

[cwebber]

By the way, I have actually written some presentations in Wraith. I like it enough that I enjoy using it for some things.

But the real question is best asked as: has the final decision been made to signal that the primary advocated (if not only supported) language direction future of Racket is something Shrubbery-derived?

[cwebber]

I see there's a meeting in 50 minutes. If it's welcome I'll show up, and if there's space for conversation, make my case. If not, can do so at a future meeting.

[rfindler]

I can't make the meeting but my own $0.02 is that, while Rhombus seems to have gone in a specific direction, I think that's more because people wanted to try it out and see how well it worked! I also think that you'd be welcome at the meeting.

[cwebber]

Great! Look forward to it.

[cwebber]

Per the meeting, I've made a PR about Wraith and it's proposed to be on the agenda for the next meeting (January 6th).

[rocketnia]

My primary concern about Rhombus from the start, and the one I still hold, is that moving away from s-expressions unleashes a lot of idiosyncratic design issues that both the language designers and the individual macro designers can otherwise blissfully avoid. It's just a lot of extra work, not only upfront, but ongoing as people write more macros and more Racket implementations.

A secondary concern I have is that it usually seems like the new syntax isn't being pursued because it has particular advantages, but merely because "something really is wrong with S-expressions." If we're going to take on all these idiosyncrasies, I'd much prefer to see somewhere particularly compelling Racket is headed toward, not somewhere it's running from.

When it comes to running away from s-expressions, we don't even have a very clear direction to run in: We're a bunch of s-expression users, and if we trust our own sensibilities, we might pick an endpoint that has the exact same downsides as the place we started from. And yet if we don't trust our sensibilities, we might end up with a poser of a language that's too preoccupied with external approval to bring out its own best qualities.

I personally suspect the "problem" with s-expressions to be that users are asked to edit programs via editing semanticsless trees via editing semanticsless text. I suspect that makes for a more indirect experience, especially when the tooling around the second "via" falls short because it doesn't have enough semantic context to be well tailored to the code's domain. As long as we have an intermediate tree representation that matters to programmers, especially a semanticsless one, the "problem" with s-expressions will likely follow us. But if we want to have macros play a part in determining the code's semantics, what are we supposed to pass them if not something a little bit semanticsless? The problem is the macros.

When it comes to "somewhere to go," I think the closest thing we have to a destination is expressed in the current draft of the State of Rhombus document (permalink, latest): Rhombus can show more people that Racket's language construction techniques can work for languages they actually want to use.

Based on what I was just saying, this is a catch-22. How do we show people they can use macros without, on some level, bringing the s-expression "problem" with us?

And wouldn't we have a better foothold in this outreach if we instead promoted the existing work in Groovy's AST transformations, Rust's macros, Idris's elaborator tactics language, Template Haskell, or any of the examples in prior art, rather than building something all new? If none of those is quite right to make the desired point, what about developing a framework that is? Why must it be a language at the center of the Racket ecosystem?

I guess my own best answer to that is "it needs to be at the center of the ecosystem so outsiders stand a chance of being able to read the macro system docs, and so the docs can keep up with the latest developments rather than being an afterthought." But... will the macro system docs really be more readable this way, or will they be twice as big as before and full of boring minutiae due to all the new syntactic idiosyncrasies the system has to model?

Anyhow, syntax changes aren't the only thing to look forward to about Rhombus. I may have serious doubts about the syntax direction and how it relates to the ostensible goals, but I still feel like I can add to other discussions (including the more shallow syntax-related ones), and I believe a slightly better Rhombus is still going to be better than a worse one. :)

[cwebber]

Really well said!

[cwebber]

I think the meeting clarified that Rhombus is pretty committed to the "something seems wrong with s-expressions so we're trying something else" path. It's true that Rhombus is a research project at this stage; it does sound like the key Racket leadership is intent on making it the main future though. I guess I don't find this personally very satisfying because I'm deciding what the future focus of my projects are (whether the primary implementation is in Guile or Racket; both will continue to be supported but which one is where most of the work happens?). It's good to hear that the s-expression stuff isn't going away, but I do still get the sense that there's a good chance that the sexp stuff will be considered the "legacy" part of Racket in the future, and I have bad experiences with building things on what end up being "legacy" foundations, so that does affect my decisionmaking right now. Maybe I'm wrong.

At any rate, I don't mean to kill everyone else's excitement; good luck with the project.

[cwebber]

I've expressed my reservations but I fear they may have come across as too negative and draining for the people who are volunteering on Racket + Rhombus. I definitely don't want to do that; I've spent the time and energy arguing my case because I do care a lot about Racket, and because of my own need to make a decision about focus at this point (not about whether or not spritely continues to have a presence in racket-land... part of the goal of porting to both is to demonstrate that both systems can talk to each other). But I love Racket and the Racket community a lot, and I know the maintainers do mean it when they say the s-expression stuff won't go away... and I don't plan on going away either or my own work leaving this space.

It's unclear to me what the mapping is between Racket and Rombus in the future, and I'm guessing that's still to be defined, but I'm sure it will be taken with significant care. I hope the best for both projects individually and together.

[countvajhula]

Speaking from the sidelines here, I just want to say that I appreciate your vigilance and championing of the cause of S-expressions (or symexes as I like to call them). My hope is that the Rhombus effort will succeed to the betterment of Racket and the language ecosystem (including, I hope, symex-oriented languages such as Wisp and Wraith) rather than at the cost of it. The comments that I've seen from Matthew and others seem to support this, and I remain hopeful that this will pan out 🙂

[benhsz]

This could be of interest to the Rhombus project, or other folks that agree that )))))) can be problematic but otherwise like S-expressions. Not sure if it needs a new thread so I'll share here instead.

https://github.com/benhsz/parless

I've been furiously documenting it the past few days, so hopefully it's clear/detailed enough.

[spdegabrielle]

Thanks @benhsz
Can I ask you to

• Put on package catalog with tags DrRacket, plugin
• Post in show and tell on discourse
• Post in #show-and-tell on Racket Discord

[benhsz]

Well I think it's too early for that since there's literally zero code at the moment, so the plugin doesn't exist yet technically. It's also unlikely that I'll get it working anytime soon, I'm on part 2 of How to Design Programs and I'd rather finish the exercises in that book before trying to implement it.

[spdegabrielle]

Well I think it's too early for that since there's literally zero code at the moment, so the plugin doesn't exist yet technically.

LOL
I read the first paragraph and got too excited!